kwanwaichung/mt-eval

Code and data for "MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models"

mt-eval's Issues

Is there a Chinese version of the test set?

Thanks for such a wonderful work. Is there any plan to open the test set in Chinese?

can't load the dataset

from datasets import load_dataset

tasks = [
  "refinement_single",
  "refinement_multi",
  "expansion_single",
  "expansion_multi",
  "follow-up_single",
  "follow-up_multi",
  "recollection_single_cls",
  "recollection_multiple_cls",
  "recollection_single_global-inst",
  "recollection_multi_global-inst",
]

data = load_dataset("wckwan/MT-Eval", tasks, split="test").to_list()

and got the error below:

Downloading data files: 100%|█████████████████████████████████████| 1/1 [00:00<00:00, 200.08it/s]

Extracting data files: 100%|███████████████████████████████████████| 1/1 [00:00<00:00, 84.66it/s]

Traceback (most recent call last):
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\builder.py", line 1875, in _prepare_split_single
    writer.write_table(table)
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\arrow_writer.py", line 568, in write_table
    pa_table = table_cast(pa_table, self._schema)
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\table.py", line 2290, in table_cast
    return cast_table_to_schema(table, schema)
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\table.py", line 2249, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]  
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\table.py", line 2249, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]  
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\table.py", line 1817, in wrapper   
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\table.py", line 1817, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\table.py", line 2048, in cast_array_to_feature
    casted_values = _c(array.values, feature[0])
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\table.py", line 1819, in wrapper   
    return func(array, *args, **kwargs)
  File "D:\coding\Anaconda3-2021.11\lib\site-packages\datasets\table.py", line 2110, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
struct<user: string, sys: string, id: string, do_inference: bool>
to
{'user': Value(dtype='string', id=None), 'sys': Value(dtype='string', id=None), 'id': Value(dtype='string', id=None), 'do_inference': Value(dtype='bool', id=None), 'inst': Value(dtype='string', id=None)}

where's the jsonl files when calculate the scores

code\MT-Eval\calculate_score.py", line 209, in <module>
    for row in open(
FileNotFoundError: [Errno 2] No such file or directory: '/kfdata05/kf_grp/wckwan/Multi-Turn-Benchmark/our_data/global-inst_conv_v1.0.jsonl'

Is the corresponding relationship right?
/kfdata05/kf_grp/wckwan/Multi-Turn-Benchmark/our_data/global-inst_conv_v1.0.jsonl
to
recollection_multi_global-inst.jsonl

/kfdata05/kf_grp/wckwan/Multi-Turn-Benchmark/our_data/global-inst-single_conv_v1.0.jsonl
to
recollection_single_global-inst.jsonl