Permutations of the sentences give different results

[jitsi]

Permutations of the sentences give different results

[nikvaessen]

You're right - permutations of the input list do give different results due to the concatenation of all elements of the list into a single string over which the edit distance is computed.

I initially designed this library for evaluating meeting transcriptions. This has a baked-in assumption that multiple sentences are ordered, and share context. I ran into the problem that the transcription of the meeting (as a list of sentences) and the hypothesis of a speech-to-text engine (as a list of sentences) did not overlap nicely, e.g the lists had different lengths. However, for this use-case, the implementation as-is is desired, as you would never permute the lists.

I agree that when you're evaluating, say librispeech, you would want invariance when permuting the (gt, hypothesis) tuples.

Do you have any suggestions for improving the current API?

[gabeur]

Thanks for you reply. I understand the original purpose of the library, and that this library is not suitable for evaluating WER over a set of utterances like librispeech.
I am therefore surprised that several implementations (here, here, here, ...) use the wer function of this library for their evaluation.

It might be important to make it clear that the wer function of this library is not suitable for that purpose.
When evaluating on a set of utterances like librispeech, the edit distance should be computed independently for each utterance. In the case of a test set of short utterances, the performance difference is significant and unfair when comparing against approaches that use the correct implementation.

The following function computes the edit distance independently for each utterance:

def wer(
  truth: List[str],
  hypothesis: List[str],
  truth_transform: Union[tr.Compose, tr.AbstractTransform] = _default_transform,
  hypothesis_transform: Union[tr.Compose, tr.AbstractTransform] = _default_transform,
  **kwargs
) -> float:
  """
  Calculate word error rate (WER) between a set of ground-truth sentences and
  a set of hypothesis sentences.
  :return: WER as a floating point number
  """
  # raise an error if the number of ground-truth sentences and the number of 
  # hypothesis sentences differ.
  if len(truth) != len(hypothesis):
    raise ValueError("the number of ground-truth sentences and the number "
                     "of hypothesis sentences differ")
  hits, substitutions, deletions, insertions = 0, 0, 0, 0
  for (truth_sample, hypothesis_sample) in zip(truth, hypothesis):
    m = compute_measures(
        truth_sample, hypothesis_sample, truth_transform, hypothesis_transform, **kwargs
    )
    hits += m["hits"]
    substitutions += m["substitutions"]
    deletions += m["deletions"]
    insertions += m["insertions"]
            
  error = substitutions + deletions + insertions
  total = substitutions + deletions + hits

  return error / total

[nikvaessen]

This issue should be fixed from version 2.3.0 onwards.