fgnt/meeteval

MeetEval - A meeting transcription evaluation toolkit

meeteval's Issues

CTM files with comments can't be parsed

If you try to use CTM files with comments in them (as defined in https://github.com/usnistgov/SCTK/blob/f48376a203ab17f0d479995d87275db6772dcb4a/doc/infmts.htm#L285 ) meeteval throws a error.

Reason: When reading lines from CTM files, meeteval ignores empty lines, but lines with comments (starting with ";;") also need to be ignored.

Computing time-constrained WER

I am thinking of a metric for long-form ASR and segmentation. Consider the following scenario:

• The input is a long recording (either single speaker or multi speaker).
• References may be with word-level timestamps (CTM file) or segment-level (STM).
• Hypothesis may be word-level or segment-level (CTM or STM).

If reference is STM and hypothesis is CTM, this may correspond to computing the asclite aWER metric, but we also want to support (i) other kinds of systems that may not provide word-level timestamps, and (ii) tighter penalty on segmentation by providing reference CTM.

Additionally, we also want to be able to include multiple possible references (e.g., references may be orthographic or normalized in some way), although I understand that this may be beyond the scope of this toolkit.

I am looking for suggestions about what would be a good metric (if one exists) for this scenario.

(cc @MartinKocour since we were having related discussions.)

Missing `LICENSE` file

Hi folks,

Thanks for your work here - much appreciated!

Would you please consider adding a LICENSE text file to the repository?

Thanks in advance!

Getting ORC-WER breakdown

Is there a simple way to get the WER break-down into ins/del/sub when computing the ORC-WER?

Does mdeval computes DER taking overlapped speech into account?

Thank you very much for sharing this repository. It is very useful to have a single repo with many audio metrics :)

Tools like pyannote allows us to choose the collar and if we want to compute DER on overlapped speech regions or not.

With mdeval, we can specify the collar but it seems like there is no option for including overlapped speech in the metric or not.
Does that mean that by default it computes over overlapped regions? Or are they excluded for the calculations?

Thank you for your answer !

circular import error

Hi,

Thank you for open sourcing these multispeaker ASR metrics :).

Just reporting some issue I encountered:

Installing as follows works fine

pip install cython
git clone [email protected]:fgnt/meeteval.git
pip install -e ./meeteval[cli]

However installing like:

pip install cython
pip install https://github.com/fgnt/meeteval/archive/refs/heads/main.zip

Causes circular import issue:

import meeteval

Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/lib/python3.7/site-packages/meeteval/init.py", line 1, in
from . import io
ImportError: cannot import name 'io' from 'meeteval' (/opt/conda/lib/python3.7/site-packages/meeteval/init.py)

Very strange, wondering if you have any ideas for workarounds for this? The build software we are using doesn't support installation via the first method.

ToDo: fix mimo_matching_v4 for python 3.9

Python 3.9.17 (main, Jul 28 2023, 05:54:52) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from meeteval.wer.matching import orc_matching, mimo_matching
   ...: ref=[['a']]
   ...: hyps=['aaabb']
   ...: mimo_matching.mimo_matching_v4(ref, hyps)
   ...: 
Out[1]: (2, [(0, 0)])

Python 3.11.4 (main, Jul  5 2023, 13:45:01) [GCC 11.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from meeteval.wer.matching import orc_mat
ching, mimo_matching
   ...: ref=[['a']]
   ...: hyps=['aaabb']
   ...: mimo_matching.mimo_matching_v4(ref, hyps)
   ...: 
Out[1]: (4, [(0, 0)])

See GitHub Action log: logs_144.zip

ImportError: ...cy_levenshtein.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZSt28__throw_bad_array_new_lengthv

This issue is for documentation: See conda/conda#10757

ImportError: .../meeteval/wer/matching/cy_levenshtein.cpython-38-x86_[64-linux-gnu.so](http://64-linux-gnu.so/): undefined symbol: _ZSt28__throw_bad_array_new_lengthv

According to conda/conda#10757 this error can happen, when you use different gcc versions, e.g. conda and system gcc, for details, see that issue.