SentencesToListOfWords is removed after 2.2.0

[jitsi]

SentencesToListOfWords is removed after 2.2.0

[andrewrfreed]

We used the following pipeline (passing in a word_list):

        pipeline = []
        pipeline.append(jiwer.ToLowerCase())
        pipeline.append(jiwer.RemoveMultipleSpaces())
        pipeline.append(jiwer.RemoveWhiteSpace(replace_by_space=True))
        pipeline.append(jiwer.SentencesToListOfWords(word_delimiter=" "))
        pipeline.append(jiwer.RemoveSpecificWords(word_list.split(",")))
        pipeline.append(jiwer.RemovePunctuation())
        pipeline.append(jiwer.Strip())
        pipeline.append(jiwer.RemoveEmptyStrings())

        return jiwer.Compose(pipeline)

1. Unclear whether we can just swap in ReduceToListOfListOfWords
2. Our team must use an exact (older) version of this very nice library.

[nikvaessen]

I intended 2.2 to 2.3 to a be a major change, and also didn't think hard enough about depreciation policies, my bad.

In order to fix the bug mentioned in #46, I had to change the data structure used to compute the WER. Your pipeline now needs to end with the ReduceToListOfListOfWords transform.

Do you have some unit tests over your pipeline? I think the following should lead to the same WER:

        pipeline = []
        pipeline.append(jiwer.ToLowerCase())
        pipeline.append(jiwer.RemoveWhiteSpace(replace_by_space=True))
        pipeline.append(jiwer.RemoveMultipleSpaces()) # note, I believe this must come after RemoveWhiteSpace(True)
        pipeline.append(jiwer.RemoveSpecificWords(word_list.split(",")))
        pipeline.append(jiwer.RemovePunctuation())
        pipeline.append(jiwer.Strip())
        pipeline.append(jiwer.RemoveEmptyStrings())
        pipeline.append(jiwer.ReduceToListOfListOfWords(word_delimiter=" "))

        return jiwer.Compose(pipeline)

[andrewrfreed]

Thanks for the suggestion, I will try it out.
As for deprecation, I understand, I will likely do a forced dependency upgrade on our users.

[andrewrfreed]

This pipeline is close but not quite correct. I build a pipeline as you suggest, then ultimately fail in the compute_measures function.

cleaned_ref = pipeline(reference)
cleaned_hyp = pipeline(hypothesis)
measures = jiwer.compute_measures(cleaned_ref, cleaned_hyp)  'line 211 in stack track below

Traceback (most recent call last):
  File "analyze.py", line 252, in <module>
    main()
  File "analyze.py", line 246, in main
    results = analyzer.analyze()
  File "analyze.py", line 211, in analyze
    measures = jiwer.compute_measures(cleaned_ref, cleaned_hyp)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jiwer/measures.py", line 210, in compute_measures
    truth, hypothesis, truth_transform, hypothesis_transform
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jiwer/measures.py", line 321, in _preprocess
    transformed_truth = truth_transform(truth)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jiwer/transforms.py", line 76, in __call__
    text = tr(text)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jiwer/transforms.py", line 55, in __call__
    return self.process_list(sentences)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jiwer/transforms.py", line 202, in process_list
    return [self.process_string(s) for s in inp]
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jiwer/transforms.py", line 202, in <listcomp>
    return [self.process_string(s) for s in inp]
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jiwer/transforms.py", line 199, in process_string
    return re.sub(r"\s\s+", " ", s)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py", line 192, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

I use compute_measures for two reasons:

• Convenience of computing all measures at once
• So that I can later output the original & transformed strings (useful for analysis)

[andrewrfreed]

I used a list-comprehension to optionally flatten my lists before sending to compute_measures

This fails in case of insertions in the hypothesis:

  File "analyze.py", line 216, in analyze
    measures = jiwer.compute_measures(cleaned_ref, cleaned_hyp)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jiwer/measures.py", line 210, in compute_measures
    truth, hypothesis, truth_transform, hypothesis_transform
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jiwer/measures.py", line 329, in _preprocess
    len(transformed_truth), len(transformed_hypothesis)
ValueError: number of ground truth inputs (8) and hypothesis inputs (9) must match.

My line 216 is:

measures = jiwer.compute_measures(cleaned_ref, cleaned_hyp)

[nikvaessen]

cleaned_ref = pipeline(reference)
cleaned_hyp = pipeline(hypothesis)
measures = jiwer.compute_measures(cleaned_ref, cleaned_hyp)  'line 211 in stack track below

In the above code, you apply the pipeline before compute_measures, while you should give the pipeline as a keyword argument:

measures = compute_measures(
    reference,
    hypothesis,
    truth_transform=pipeline,
    hypothesis_transform=pipeline,
)

Also, does the length between reference and hypothesis differ? If that's the case, you must use ReduceToSingleSentence before ReduceToListOfListOfWords. It would be easiest to debug if you could share a complete, running example, including reference and hypothesis.

[andrewrfreed]

Aha, the pair of ReduceToSingleSentence and ReduceToListOfListOfWords were critical. Now my pipeline runs to completion, with accurate results.

To summarize, I used to run:

        pipeline = []
        pipeline.append(jiwer.ToLowerCase())
        pipeline.append(jiwer.RemoveMultipleSpaces())
        pipeline.append(jiwer.RemoveWhiteSpace(replace_by_space=True))
        pipeline.append(jiwer.SentencesToListOfWords(word_delimiter=" "))
        pipeline.append(jiwer.RemoveSpecificWords(word_list.split(",")))
        pipeline.append(jiwer.RemovePunctuation())
        pipeline.append(jiwer.Strip())
        pipeline.append(jiwer.RemoveEmptyStrings())

        return jiwer.Compose(pipeline)

I have updated to run

        pipeline = []
        pipeline.append(jiwer.ToLowerCase())
        pipeline.append(jiwer.RemoveWhiteSpace(replace_by_space=True))
        pipeline.append(jiwer.RemoveMultipleSpaces())
        pipeline.append(jiwer.RemoveSpecificWords(word_list.split(",")))
        pipeline.append(jiwer.RemovePunctuation())
        pipeline.append(jiwer.Strip())
        pipeline.append(jiwer.RemoveEmptyStrings())
        pipeline.append(jiwer.ReduceToSingleSentence())
        pipeline.append(jiwer.ReduceToListOfListOfWords(word_delimiter=" "))

        return jiwer.Compose(pipeline)

I'm going to run a few more test cases but I think it's working. Thanks @nikvaessen for all the help!

[xzwworkplace]

`def chunked_cer(targets, predictions, chunk_size=None):
_predictions = [char for seq in predictions for char in list(seq)]
_targets = [char for seq in targets for char in list(seq)]

if chunk_size is None: return jiwer.wer(_targets, _predictions)

start = 0
end = chunk_size
H, S, D, I = 0, 0, 0, 0

while start < len(targets):
    _predictions = [char for seq in predictions[start:end] for char in list(seq)]
    _targets = [char for seq in targets[start:end] for char in list(seq)]
    chunk_metrics = jiwer.compute_measures(_targets, _predictions)
    H = H + chunk_metrics["hits"]
    S = S + chunk_metrics["substitutions"]
    D = D + chunk_metrics["deletions"]
    I = I + chunk_metrics["insertions"]
    start += chunk_size
    end += chunk_size

return float(S + D + I) / float(H + S + D)`

I face a error when use jiwer
ValueError: number of ground truth inputs (15787) and hypothesis inputs (14746) must match.