medkit.text.metrics.classification

This module needs extra-dependencies not installed as core dependencies of medkit. To install them, use pip install medkit-lib[metrics-text-classification].

Classes:

TextClassificationEvaluator(attr_label)

An evaluator for attributes of TextDocuments

class TextClassificationEvaluator(attr_label)

An evaluator for attributes of TextDocuments

Initialize the text classification evaluator

Parameters

attr_label (str) – Label of the attribute to evaluate.

Methods:

compute_classification_report(true_docs, ...)	Compute classification metrics of document attributes giving annotated documents.
compute_cohen_kappa(docs_annotator_1, ...)	Compute the cohen's kappa score, an inter-rated agreement score between two annotators.
compute_krippendorff_alpha(docs_annotators)	Compute the Krippendorff alpha score, an inter-rated agreement score between multiple annotators.

compute_classification_report(true_docs, predicted_docs, metrics_by_attr_value=True, average='macro')

Compute classification metrics of document attributes giving annotated documents. This method uses sklearn.metrics.classification_report to compute precision, recall and F1-score for value of the attribute.

Warning

The set of true and predicted documents must be sorted to calculate the metric

Parameters

• true_docs (List[TextDocument]) – Text documents containing attributes of reference

• predicted_docs (List[TextDocument]) – Text documents containing predicted attributes

• metrics_by_attr_value (bool) – Whether return metrics by attribute value. If False, only global metrics are returned

• average (Literal['macro', 'weighted']) – Type of average to be performed in metrics. - macro, unweighted mean (default) - weighted, weighted average by support (number of true instances by attr value)

Return type

Dict[str, Union[float, int]]

Returns

Dict[str,Union[float,int]] – A dictionary with the computed metrics

compute_cohen_kappa(docs_annotator_1, docs_annotator_2)

Compute the cohen’s kappa score, an inter-rated agreement score between two annotators. This method uses ‘sklearn’ as backend to compute the level of agreement.

Warning

The set of documents must be sorted to calculate the metric

Parameters

• docs_annotator_1 (List[TextDocument]) – Text documents containing attributes annotated by the first annotator

• docs_annotator_2 (List[TextDocument]) – Text documents to compare, these documents contain attributes annotated by the other annotator

Return type

Dict[str, Union[float, int]]

Returns

Dict[str, Union[float, int]] – A dictionary with cohen’s kappa score and support (number of annotated docs). The value is a number between -1 and 1, where 1 indicates perfect agreement; zero or lower indicates chance agreement.

compute_krippendorff_alpha(docs_annotators)

Compute the Krippendorff alpha score, an inter-rated agreement score between multiple annotators.

Warning

Documents must be sorted to calculate the metric.

Note

See medkit.text.metrics.irr_utils.krippendorff_alpha for more information about the score

Parameters

docs_annotators (List[List[TextDocument]]) – A list of list of Text documents containing attributes. The size of the list is the number of annotators to compare.

Return type

Dict[str, Union[float, int]]

Returns

Dict[str, Union[float,int]] – A dictionary with the krippendorff alpha score, number of annotators and support (number of documents). A value of 1 indicates perfect reliability between annotators; zero or lower indicates absence of reliability.