Skip to content

CER / WER metrics

The atr-ner-eval cer command can be used to compute the Character and Word Error Rates of your automatic workflow globally, or for each semantic category.

Metric description

The Character Error Rate and Word Error Rate are well known metrics for Automatic Text Recognition (ATR) and Automatic Speech Recognition (ASR), based on the Levenshtein edit distance.

  • The Character Error Rate is the percentage of characters that have been transcribed incorrectly by the automatic workflow. A 5% CER means that the 95 characters out of 100 are correctly transcribed, while 5 are misread (insertion, deletion, substitution).
  • The Word Error Rate is the percentage of words that have been transcribed incorrectly by the automatic workflow. A 5% WER means that the 95 words out of 100 are correctly transcribed, while 5 are misread (insertion, deletion, substitution).

Parameters

Here are the available parameters for this metric:

Parameter Description Type Default
--label-dir Path to the directory containing BIO label files. pathlib.Path
--prediction-dir Path to the directory containing BIO prediction files. pathlib.Path
--by-category Whether to display CER and WER for each category. bool False

The parameters are also described when running atr-ner-eval cer --help.

Examples

Global evaluation

Use the following command to compute the overall CER and WER: 

atr-ner-eval cer --label-dir Simara/labels/ \
                 --prediction-dir Simara/predictions/

It will output the results in Markdown format: 

2023-12-13 16:35:05,598 INFO/atr_ner_eval.utils: The dataset is complete.
2023-12-13 16:35:05,828 INFO/atr_ner_eval.cer_wer: CER / WER evaluation table:

| Category | CER (%) | WER (%) | Support |
|:---------|:-------:|:-------:|--------:|
| total    |   6.37  |  17.26  |     804 |

Evaluation for each category

Use the following command to compute CER and WER for each semantic category: 

atr-ner-eval cer --label-dir Simara/labels/ \
                 --prediction-dir Simara/predictions/ \
                 --by-category

It will output the results in Markdown format: 

2023-12-13 16:35:05,598 INFO/atr_ner_eval.utils: The dataset is complete.
2023-12-13 16:35:05,828 INFO/atr_ner_eval.cer_wer: CER / WER evaluation table:

| Category        | CER (%) | WER (%) | Support |
|:----------------|:-------:|:-------:|--------:|
| analysis        |   9.42  |  23.36  |     780 |
| arrangement     |   8.32  |  15.82  |      79 |
| article_number  |   3.14  |   4.2   |     676 |
| date            |   2.18  |   3.58  |     757 |
| series          |   3.07  |   3.22  |     676 |
| title           |   7.67  |  21.04  |     804 |
| volume_number   |  12.38  |   14.1  |     675 |
| total           |   6.37  |  17.26  |     804 |