OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
Home Page: https://opencompass.org.cn/
License: Apache License 2.0
的
的
No response
解码会偶现失败 最后的评估会因为数量不一致无法评估 请问如何回复其中一条 或者剔除这一条正常进行评估
{'CUDA available': True,
'CUDA_HOME': '/home/work/cuda-11.3',
'GCC': 'gcc (GCC) 8.2.0',
'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-SXM4-40GB',
'MMEngine': '0.8.2',
'NVCC': 'Cuda compilation tools, release 11.3, V11.3.58',
'OpenCV': '4.8.0',
'PyTorch': '2.0.1',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.8\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
' - CuDNN 8.7\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
'CUDNN_VERSION=8.7.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_QNNPACK '
'-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
'-Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0]',
'TorchVision': '0.15.2',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+4b0aa80',
'sys.platform': 'linux'}
No response
自己试了下，报错蛮多的，想问下是否有官方文档支持清华chatglm系列
ceval是单选题，每个科目题目数目不一样，模型推测出结果之后，如果做对一题就加分吗？然后再归一化为百分制？
nq之类问答题，是如何计算分数的呢？
在discussion区提问，响应比较慢，提到issue区试试。多谢。
python3.10
No response
我遇到了类似问题，在ceval,mmlu, gaokao 上都速度异常（3个任务2卡22h还未结束），单个样本速度正常。
outputs 输出发现评估重复，比如profession_law只有170个，但输出文件中有 15 个 xxxx_mmlu_profession_law_[0-15]，每个95个样本，且样本有重复。请问是评估方式不对吗，还是 sizepartition 这里有 bug？
python run.py ./configs/eval_xxx.py --mode eval -w outputs/ --slurm --partition xxx --debug
Originally posted by @Desein-Yang in #30 (comment)
The result I get is listed below.
dataset version metric mode internlm-chat-7b-hf
--------------------------- --------- ------------- ------ ---------------------
--------- 考试 Exam --------- - - - -
mmlu - naive_average gen 50.52
But according to the leaderboard, it should be 50.8
...
No response
使用llama_7B_hf 这个模型进行解码评测 观测nvidia-smi 里面显卡的使用会不断有显存开辟释放 八块a100 两个小时只解了300条 速度很慢 求助是否有设置上的问题
下图是模型设置
显卡观测
{'CUDA available': True,
'CUDA_HOME': '/home/work/cuda-11.3',
'GCC': 'gcc (GCC) 8.2.0',
'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-SXM4-40GB',
'MMEngine': '0.8.2',
'NVCC': 'Cuda compilation tools, release 11.3, V11.3.58',
'OpenCV': '4.8.0',
'PyTorch': '2.0.1',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.8\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
' - CuDNN 8.7\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
'CUDNN_VERSION=8.7.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_QNNPACK '
'-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
'-Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0]',
'TorchVision': '0.15.2',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+4b0aa80',
'sys.platform': 'linux'}
No response
As proposed in #80 (comment)
https://opencompass.readthedocs.io/en/latest/advanced_guides/new_model.html
这个文档中已经有关于如何新增model支持的描述，但是对于如何完成评测一个新增model，这部分文档还比较分散，比如需要：
无
No response
test
07/28 16:33:32 - OpenCompass - INFO - Task [opencompass.models.MyModel_opencompass_customize model test/agieval-gaokao-physics]
/opt/conda/envs/opencompass/lib/python3.10/site-packages/mmengine/utils/manager.py:113: UserWarning: <class 'mmengine.logging.logger.MMLogger'> instance named of OpenCompass has been created, the method get_instance should not accept any other arguments
warnings.warn(
Traceback (most recent call last):
File "/opt/conda/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 122, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/mnt/workspace/opencompass/opencompass/datasets/base.py", line 12, in init
self.dataset = self.load(**kwargs)
File "/mnt/workspace/opencompass/opencompass/datasets/agieval/agieval.py", line 52, in load
dataset = Dataset.from_list(data)
File "/opt/conda/envs/opencompass/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 951, in from_list
return cls.from_dict(mapping, features, info, split)
File "/opt/conda/envs/opencompass/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 911, in from_dict
pa_table = InMemoryTable.from_pydict(mapping=mapping)
File "/opt/conda/envs/opencompass/lib/python3.10/site-packages/datasets/table.py", line 799, in from_pydict
return cls(pa.Table.from_pydict(*args, **kwargs))
File "pyarrow/table.pxi", line 3849, in pyarrow.lib.Table.from_pydict
File "pyarrow/table.pxi", line 5401, in pyarrow.lib._from_pydict
File "pyarrow/array.pxi", line 357, in pyarrow.lib.asarray
File "pyarrow/array.pxi", line 243, in pyarrow.lib.array
File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
File "/opt/conda/envs/opencompass/lib/python3.10/site-packages/datasets/arrow_writer.py", line 189, in arrow_array
out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
File "pyarrow/array.pxi", line 327, in pyarrow.lib.array
File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 123, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: Expected bytes, got a 'list' object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/workspace/opencompass/opencompass/tasks/openicl_infer.py", line 147, in
inferencer.run()
File "/mnt/workspace/opencompass/opencompass/tasks/openicl_infer.py", line 66, in run
self.dataset = build_dataset_from_cfg(self.dataset_cfg)
File "/mnt/workspace/opencompass/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
return LOAD_DATASET.build(dataset_cfg)
File "/opt/conda/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/opt/conda/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 144, in build_from_cfg
raise type(e)(
pyarrow.lib.ArrowTypeError: class AGIEvalDataset_v2 in opencompass/datasets/agieval/agieval.py: Expected bytes, got a 'list' object
The 'agieval-gaokao-physics' dataset loads failed
{'CUDA available': True,
'CUDA_HOME': '/usr/local/cuda',
'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
'GPU 0,1,2,3': 'Tesla V100-SXM2-32GB',
'MMEngine': '0.8.2',
'NVCC': 'Cuda compilation tools, release 11.3, V11.3.109',
'OpenCV': '4.8.0',
'PyTorch': '1.13.1+cu116',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201402\n'
' - Intel(R) Math Kernel Library Version '
'2020.0.0 Product Build 20191122 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.6.0 (Git Hash '
'52b5f107dd9cf10910aaa19cb47f3abf9b349815)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.6\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86\n'
' - CuDNN 8.3.2 (built against CUDA 11.5)\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.6, '
'CUDNN_VERSION=8.3.2, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM '
'-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
'-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
'-DEDGE_PROFILER_USE_KINETO -O2 -fPIC '
'-Wno-narrowing -Wall -Wextra '
'-Werror=return-type -Werror=non-virtual-dtor '
'-Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, '
'USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, '
'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, '
'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, '
'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0]',
'TorchVision': '0.14.1+cu116',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+2cce271',
'sys.platform': 'linux'}
No response
控制台打印：
outputs/internlm_7b/20230726_232201/logs/eval/internlm-7b-hf/openai_humaneval.out文件的输入为：
完整log为：
openai_humaneval.out.txt
帮忙看下是怎么回事？多谢
python3.10
cuda11.8
No response
按照第一个例子执行：
python run.py configs/eval_demo.py -w outputs/demo
上报如下错误：
Traceback (most recent call last):
File "/root/project/opencompass-0.1.0/opencompass/tasks/openicl_eval.py", line 10, in
from opencompass.registry import (ICL_EVALUATORS, MODELS, TASKS,
ModuleNotFoundError: No module named 'opencompass'
看了下opencompass-0.1.0/opencompass/tasks/openicl_eval.py的代码：
from opencompass.registry import (ICL_EVALUATORS, MODELS, TASKS,TEXT_POSTPROCESSORS)
感觉文件夹的层次不对？
这块需要怎么配置，能让openicl_eval.py直接看到两层之外的opencompass目录呢？
python小白，请大侠帮忙，感谢
python3.10
No response
无法指定gpu，无论在shell中使用CUDA_VISIBLE_DEVICES=1，还是在python中使用os.environ["CUDA_VISIBLE_DEVICES"]=1，都无法让程序只占用1卡，程序永远都会优先占用0卡。
shell中
CUDA_VISIBLE_DEVICES=1 python run.py configs/eval_llama_7b.py -w outputs/llama_7b --gpu_idx=1
python中
parser = argparse.ArgumentParser(description='Run an evaluation task')
parser.add_argument('--gpu_idx', type=str, default=None)
args = parser.parse_args()
os.environ["CUDA_VISIBLE_DEVICES"] = args.gpu_idx
No response
MDLRetriever uses the gpt2-xl model to process the commonsenseqa dataset.
In def topk_search(self): ,it took me nearly 2 hours to process to get rtr_idx_list on V100. If I don't care about the random seed, can I pickle the result of the first processing into a binary file, which is convenient for testing and calling quickly later?
Hello, I encountered the following problems when evaluating a custom model on the Alibaba Cloud server, how to solve it?
FileNotFoundError: Couldn't find a module script at /mnt/workspace/xxx/evaluation/opencompass/accuracy/accuracy.py. Module 'accuracy' doesn't exist on the Hugging Face Hub either.
python run.py ./config/eval_my_model.py --debug
No response
1、data/ceval/包含formal_ceval和release_ceval两个文件夹，分别都是什么用途？评测internLM-7b时，使用的是formal_ceval下的数据。使用release_ceval的数据在数据集合配置glm文件夹下，是评测GLM的模型的时候使用release_ceval数据吗？
2、data/ceval/formal_ceval下分别有dev，test，val三个文件夹，CEvalDataset类将这些数据load到了下面变量：
return DatasetDict({ 'val': val_dataset, 'dev': dev_dataset, 'test': test_dataset })
这块后续是如何评测的呢？测试集、开发集，验证集这三类数据如何使用？
按照ceval官方的说法，测试机没有答案，需要将测试结果提交他们网站才能拿到得分，opencompass是如何操作的呢？
3、opencompass是否支持问答题的评测，如果支持，哪个数据集是问答题的例子？
建议opencompass开个论坛，我们可以到论坛里去交流相关技术问题。
感谢~
无。
No response
Hi OpenCompass team, thank you for this excellent project. I see the results for InternLM-Chat-7B reported in the OpenCompass leaderboard, but I don't see any results for the base model InternLM-7B. Would you be willing to share those numbers as well?
I think it would be very useful to the community to understand how much of the performance comes from the base model vs. the chat finetuning.
In https://opencompass.readthedocs.io/en/latest/MMBench.html:
if data_sample['context'] is None:
prompt = data_sample['context'] + ' ' + data_sample['question'] + ' ' + data_sample['options']
else:
prompt = data_sample['question'] + ' ' + data_sample['options']
It should be
if data_sample['context'] is not None:
No anomaly
Also, the evaluation pipeline of MMbench is not complete. Can you release the code of Circular Evaluation Strategy and ChatGPT-involved Choice Extraction? How should we submit the results to the leaderboard? The hyperlink of instruction https://opencompass.org.cn/mmbench seems not correct.
评测humaneval数据，推理完成，eval阶段出现下面异常，可能是哪里出了问题？
Traceback (most recent call last):
File "/root/project/code/opencompass-3715be6/opencompass/tasks/openicl_eval.py", line 180, in
inferencer.run()
File "/root/project/code/opencompass-3715be6/opencompass/tasks/openicl_eval.py", line 54, in run
self._score()
File "/root/project/code/opencompass-3715be6/opencompass/tasks/openicl_eval.py", line 121, in _score
result = icl_evaluator.score(
File "/root/project/code/opencompass-3715be6/opencompass/datasets/humaneval.py", line 36, in score
score = self.eval(out_dir,
File "/root/project/code/opencompass-3715be6/human-eval-master/human_eval/evaluation.py", line 51, in evaluate_functional_correctness
problems = read_problems(problem_file)
File "/root/project/code/opencompass-3715be6/human-eval-master/human_eval/data.py", line 12, in read_problems
return {task["task_id"]: task for task in stream_jsonl(evalset_file)}
File "/root/project/code/opencompass-3715be6/human-eval-master/human_eval/data.py", line 12, in
return {task["task_id"]: task for task in stream_jsonl(evalset_file)}
File "/root/project/code/opencompass-3715be6/human-eval-master/human_eval/data.py", line 21, in stream_jsonl
with open(filename, "rb") as gzfp:
FileNotFoundError: [Errno 2] No such file or directory: '/root/project/code/opencompass-3715be6/human-eval-master/human_eval/../data/HumanEval.jsonl.gz'
python3.10
humanEval使用master分之
No response
I am currently using LLaMa to evaluate PPL's ACC on MMLU dataset and I have a question about opencompass toolkit. When I use opencompass to divide the evaluation task into 40 sub-tasks, does this mean that the toolkit loads the same model 40 times for evaluation? From my observation, the time it takes to load the model is equivalent to the inference time. Therefore, I am concerned that the evaluation may take a long time due to the repeated loading of the model.
Additionally, I am wondering if it is possible to configure the opencompass config to support batch inference? This could potentially improve the efficiency of the evaluation process.
I would greatly appreciate it if you could provide detailed guidance on these issues.
test
test
No response
运行评测脚本时，出现以下错误
Traceback (most recent call last):
File "/d1/pub/wyzh/evaluation/opencompass/run.py", line 339, in
main()
File "/d1/pub/wyzh/evaluation/opencompass/run.py", line 210, in main
tasks = partitioner(cfg)
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/partitioners/base.py", line 48, in call
tasks = self.partition(models, datasets, work_dir, self.out_dir)
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/partitioners/size.py", line 67, in partition
datasets = sorted(datasets,
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/partitioners/size.py", line 68, in
key=lambda x: self.get_cost(x),
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/partitioners/size.py", line 188, in get_cost
dataset = build_dataset_from_cfg(dataset)
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
return LOAD_DATASET.build(dataset_cfg)
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 98, in build_from_cfg
obj_cls = registry.get(obj_type)
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/mmengine/registry/registry.py", line 451, in get
self.import_from_location()
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/mmengine/registry/registry.py", line 376, in import_from_location
import_module(loc)
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/datasets/init.py", line 1, in
from .afqmcd import * # noqa: F401, F403
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/datasets/afqmcd.py", line 7, in
from .base import BaseDataset
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/datasets/base.py", line 6, in
from opencompass.openicl import DatasetReader
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/openicl/init.py", line 2, in
from .icl_evaluator import * # noqa
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/openicl/icl_evaluator/init.py", line 4, in
from .icl_hf_evaluator import * # noqa
File "/d1/pub/wyzh/evaluation/opencompass/opencompass/openicl/icl_evaluator/icl_hf_evaluator.py", line 4, in
import evaluate
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/evaluate/init.py", line 29, in
from .evaluation_suite import EvaluationSuite
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/evaluate/evaluation_suite/init.py", line 10, in
from ..evaluator import evaluator
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/evaluate/evaluator/init.py", line 17, in
from transformers.pipelines import SUPPORTED_TASKS as SUPPORTED_PIPELINE_TASKS
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/pipelines/init.py", line 44, in
from .audio_classification import AudioClassificationPipeline
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/pipelines/audio_classification.py", line 21, in
from .base import PIPELINE_INIT_ARGS, Pipeline
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/pipelines/base.py", line 36, in
from ..modelcard import ModelCard
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/modelcard.py", line 48, in
from .training_args import ParallelMode
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/training_args.py", line 67, in
from accelerate import PartialState
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/accelerate/init.py", line 3, in
from .accelerator import Accelerator
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/accelerate/accelerator.py", line 33, in
from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in
from .utils import (
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/accelerate/utils/init.py", line 109, in
from .launch import (
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/accelerate/utils/launch.py", line 23, in
from ..commands.config.config_args import SageMakerConfig
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/accelerate/commands/config/init.py", line 19, in
from .config import config_command_parser
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/accelerate/commands/config/config.py", line 25, in
from .sagemaker import get_sagemaker_input
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/accelerate/commands/config/sagemaker.py", line 35, in
import boto3 # noqa: F401
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/boto3/init.py", line 17, in
from boto3.session import Session
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/boto3/session.py", line 17, in
import botocore.session
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/session.py", line 26, in
import botocore.client
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/client.py", line 15, in
from botocore import waiter, xform_name
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/waiter.py", line 18, in
from botocore.docs.docstring import WaiterDocstring
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/docs/init.py", line 15, in
from botocore.docs.service import ServiceDocumenter
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/docs/service.py", line 14, in
from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/docs/client.py", line 17, in
from botocore.docs.example import ResponseExampleDocumenter
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/docs/example.py", line 13, in
from botocore.docs.shape import ShapeDocumenter
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/docs/shape.py", line 19, in
from botocore.utils import is_json_value_header
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/utils.py", line 37, in
import botocore.httpsession
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/botocore/httpsession.py", line 45, in
from urllib3.contrib.pyopenssl import (
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/urllib3/contrib/pyopenssl.py", line 50, in
import OpenSSL.crypto
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/OpenSSL/init.py", line 8, in
from OpenSSL import crypto, SSL
File "/home/llm/anaconda3/envs/safe-rlhf/lib/python3.10/site-packages/OpenSSL/crypto.py", line 3268, in
_lib.OpenSSL_add_all_algorithms()
AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'
{'CUDA available': True,
'CUDA_HOME': '/home/llm/anaconda3/envs/safe-rlhf',
'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-SXM4-40GB',
'MMEngine': '0.8.3',
'NVCC': 'Cuda compilation tools, release 11.7, V11.7.99',
'OpenCV': '4.8.0',
'PyTorch': '2.0.1',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.7\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37\n'
' - CuDNN 8.5\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.7, '
'CUDNN_VERSION=8.5.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_QNNPACK '
'-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
'-Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.11 | packaged by conda-forge | (main, May 10 2023, 18:58:44) '
'[GCC 11.3.0]',
'TorchVision': '0.15.2+cu117',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+e9b7b8a',
'sys.platform': 'linux'}
No response
test
test
No response
Command:
python3 run.py configs/eval_internlm_7b.pyDataprep steps:
cd ./data/AGIEval
git clone https://github.com/microsoft/AGIEval.git
cp -R AGIEval/data dataLogs:
pyarrow.lib.ArrowTypeError: Expected bytes, got a 'list' object{'CUDA available': True,
'CUDA_HOME': '/usr/local/cuda',
'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-SXM4-80GB',
'MMEngine': '0.8.1',
'NVCC': 'Cuda compilation tools, release 12.1, V12.1.66',
'OpenCV': '4.8.0',
'PyTorch': '2.0.1',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.8\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
' - CuDNN 8.7\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
'CUDNN_VERSION=8.7.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_QNNPACK '
'-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
'-Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0]',
'TorchVision': '0.15.2',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+18efcb0',
'sys.platform': 'linux'}No response
测试运行报错，log里感觉像缺少文件：
opencompass-main/accuracy/accuracy.py. Module 'accuracy' doesn't exist on the Hugging Face Hub either.
{'CUDA available': True,
'CUDA_HOME': '/usr/local/cuda',
'GCC': 'x86_64-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0',
'GPU 0,1,2,3,4,5,6,7': 'Tesla V100-SXM2-32GB',
'MMEngine': '0.8.2',
'NVCC': 'Cuda compilation tools, release 11.6, V11.6.112',
'OpenCV': '4.8.0',
'PyTorch': '2.0.1+cu117',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2022.2-Product Build 20220804 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.7\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86\n'
' - CuDNN 8.5\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.7, '
'CUDNN_VERSION=8.5.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_QNNPACK '
'-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
'-Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.11 (main, Apr 5 2023, 14:15:30) [GCC 7.5.0]',
'TorchVision': '0.15.2+cu117',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+',
'sys.platform': 'linux'}
No response
Like title.
运行配置：
python run.py configs/eval_internlm_7b.py -w outputs/internlm_7b
代码几乎都是库里默认的，没做大的改动。数据集配置：.datasets.collections.base_medium
运行环境：
P100 GPU 16G 两块；
下面是目前生成的log：
20230725_222436.tar.gz
GPU使用情况：
目前在推理，速度如下：
请问这个速度正常吗？这么慢吗？
{'CUDA available': True,
'CUDA_HOME': '/usr/local/cuda',
'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
'GPU 0,1': 'Tesla P100-PCIE-16GB',
'MMEngine': '0.8.2',
'NVCC': 'Cuda compilation tools, release 11.8, V11.8.89',
'OpenCV': '4.8.0',
'PyTorch': '2.0.0+cu118',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2022.2-Product Build 20220804 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.8\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
' - CuDNN 8.7\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
'CUDNN_VERSION=8.7.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_QNNPACK '
'-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
'-Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.10 (main, Jul 24 2023, 16:03:07) [GCC 9.4.0]',
'TorchVision': '0.15.1+cu118',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+',
'sys.platform': 'linux'}
对代码几乎没做改动，唯一动的地方是 base_medium.py注释掉了下面数据集，因为没有找到相关数据；
from ..bbh.bbh_gen_5b92b0 import bbh_datasets
from ..agieval.agieval_mixed_2f14ad import agieval_datasets
from ..Xsum.Xsum_gen_31397e import Xsum_datasets
from ..TheoremQA.TheoremQA_gen_ef26ca import TheoremQA_datasets
from ..triviaqa.triviaqa_gen_2121ce import triviaqa_datasets
为感觉不应该这么慢。
We are curious how far can 13B models go compared with 7B models. Do you have plans to list them in the leader-boards? Some recommendations for the models: LLAMA-13B, Baichuan-13B, WizardLM-13B, Vicuna-13B and etc.
根据Docs中的demo示例，测试在 SIQA 和 Winograd 这两个数据，其中Winograd数据集与本地dataset代码不兼容，跑不通，注释掉即可正常使用，希望进行排查
即使把这里的名字改为‘winogrande_xl’，又会报其他错，把这数据注释掉，就没有问题
{'CUDA available': True,
'CUDA_HOME': '/usr/local/cuda-11.1',
'GCC': 'gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)',
'GPU 0,1,2,3': 'NVIDIA GeForce RTX 3090',
'MMEngine': '0.8.2',
'NVCC': 'Cuda compilation tools, release 11.1, V11.1.74',
'OpenCV': '4.8.0',
'PyTorch': '2.0.1',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.8\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
' - CuDNN 8.7\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
'CUDNN_VERSION=8.7.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_QNNPACK '
'-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
'-Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0]',
'TorchVision': '0.15.2',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+',
'sys.platform': 'linux'}
No response
描述
修改文件minigpt_4_7b_mmbench.py的minigpt4_model的llama_model参数和minigpt_4_load_from参数，执行python run.py ./configs/multimodal/minigpt_4/minigpt_4_7b_mmbench.py --mm-eval --debug -w outputs/minigpt_4命令，报错如下。
Traceback (most recent call last):
File "/home/taas/lidan6/opencompass/run.py", line 372, in
main()
File "/home/taas/lidan6/opencompass/run.py", line 213, in main
tasks = partitioner(cfg)
File "/home/taas/lidan6/opencompass/opencompass/partitioners/mm_naive.py", line 99, in call
models = cfg['models']
File "/home/taas/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 1483, in getitem
return self._cfg_dict.getitem(name)
File "/home/taas/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 131, in getitem
return self.build_lazy(super().getitem(key))
File "/home/taas/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 98, in missing
raise KeyError(name)
KeyError: 'models'
定位问题：错误原因是没有读到模型，感觉框架内部参数传递中间格式错了
环境配置无误，LLM模型评测可以跑通
No response
了解下， 如何处理 TheoremQA, TriviaQA and Xsum 这些数据集用于本框架的预测呢？
无
无
中文描述：
opencompass-3715be6/opencompass/models/huggingface.py中的
def _get_ppl(self, inputs: List[str], mask_length: Optional[List[int]] = None) -> List[float]: outputs, inputs = self.get_logits(inputs) .......
这里给你一个input列表，就可以计算出PPL的值；
首先调用get_logits，返回模型预测的token和input token
这个input 就是按照prompt模板组成的prompt，以ceval为例，就是five-shot的例子加上新的题目以及答案是一个label “A”
我的问题是：为啥将这个input给模型，调用模型的前向传播方法，获取output[0]就是模型的预测呢？
“fiveshot + 新的选择题目+answer ：A“ 这个给模型是想让模型预测啥呢？
大模型小白，希望高手赐教。
English description：
In the opencompass-3715be6/opencompass/models/huggingface.py file, there is a function _get_ppl that takes a list of inputs and calculates the value of Perplexity (PPL). It first calls the get_logits function to get the model's predicted tokens and input tokens.
The input list provided to the function is constructed based on a prompt template, such as the "ceval" template. In this case, it consists of a five-shot example, a new question, and the answer labeled as "A".
The question is why this input is given to the model and why the output[0] of the model's forward pass is considered as the model's prediction. What is the intention behind providing the input "fiveshot + new question + answer: A" to the model?
As a novice in large models, I would appreciate guidance from experts.
python3.10
No response
首页readme文档：
下面展示了快速安装的步骤。有部分第三方功能可能需要额外步骤才能正常运行，详细步骤请参考安装指南。
该链接not found了
None
No response
When I test my LLM on the commonsenseqa dataset, I get array out of bounds. But I don't know how to fix it.
from mmengine.config import read_base
from opencompass.partitioners import SizePartitioner, NaivePartitioner
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask, OpenICLEvalTask
with read_base():
from .datasets.commonsenseqa.commonsenseqa_ppl_5545e2 import commonsenseqa_datasets
Traceback (most recent call last):
File "/home/opencompass/opencompass/tasks/openicl_infer.py", line 147, in <module>
inferencer.run()
File "/home/opencompass/opencompass/tasks/openicl_infer.py", line 76, in run
self._inference()
File "/home/opencompass/opencompass/tasks/openicl_infer.py", line 124, in _inference
inferencer.inference(retriever,
File "/home/opencompass/opencompass/openicl/icl_inferencer/icl_ppl_inferencer.py", line 108, in inference
prompt = retriever.generate_label_prompt(
File "/home/opencompass/opencompass/openicl/icl_retriever/icl_base_retriever.py", line 146, in generate_label_prompt
self.test_ds[idx], ice, label, remain_sep)
File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2803, in __getitem__
return self._getitem(key)
File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2787, in _getitem
pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 583, in query_table
_check_valid_index_key(key, size)
File "/root/miniconda3/envs/opencompass/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 611 is out of bounds for size 611
I output partial intermediate results.
In BaseRetriever lines 40-41:
len of self.index_ds in commonsenseqa : 9741
len of self.test_ds in commonsenseqa : 611
but the len of ice_idx_list is 1221. It is too long for self.test_ds.
``{'CUDA available': True,
'CUDA_HOME': '/usr/local/cuda',
'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
'GPU 0': 'Tesla V100-SXM2-32GB',
'MMEngine': '0.8.2',
'NVCC': 'Cuda compilation tools, release 12.1, V12.1.66',
'OpenCV': '4.8.0',
'PyTorch': '2.0.1',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.8\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
' - CuDNN 8.7\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
'CUDNN_VERSION=8.7.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_QNNPACK '
'-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
'-Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0]',
'TorchVision': '0.15.2',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+',
'sys.platform': 'linux'}
No response
在加载huggingface上的数据集时，需要服务器联网去huggingface上下载；
而我的服务器不能上网，所以在执行：
from datasets import load_dataset
dataset = load_dataset(**kwargs)
会发生ConnectTimeout错误；
能否提供一种离线的数据加载方式？
或者如何修改代码更加容易从本地disk加载数据集？
请给点建议，非常感谢。
Support TydiQA?
运行评估测试
python run.py configs/test_internlm.py -w outputs/001
cat configs/test_internlm.py
from mmengine.config import read_base
with read_base():
from .datasets.ceval.ceval_gen import ceval_datasets
from .datasets.mmlu.mmlu_ppl import mmlu_datasets
datasets = [*ceval_datasets, *mmlu_datasets]
from opencompass.models import HuggingFaceCausalLM
_meta_template = dict(
round=[
dict(role='HUMAN', begin='<|User|>:', end='\n'),
dict(role='BOT', begin='<|Bot|>:', end='\n', generate=True),
],
)
models = [
dict(
type=HuggingFaceCausalLM,
abbr='internlm-chat-7b-8k-hf',
path="/media/root/sdc/data/model/internlm-chat-7b-8k",
tokenizer_path='/media/root/sdc/data/model/internlm-chat-7b-8k',
tokenizer_kwargs=dict(
padding_side='left',
truncation_side='left',
use_fast=False,
trust_remote_code=True,
),
max_out_len=100,
max_seq_len=2048,
batch_size=8,
meta_template=_meta_template,
model_kwargs=dict(trust_remote_code=True, device_map='auto'),
run_cfg=dict(num_gpus=1, num_procs=1),
)
]
然后获得评测结果
测试结果里面看不到ceval或者mmlu的测试平均值，需要如何设置，还是需要自己实现代码
之前给的 pjeval/add_grpc_support 现在没法跑了。需要
File "/workspace/GitProjects/pjeval/opencompass/models/grpc_api.py", line 108, in _generate
for status, result, tokens in self.grpcapi.generate(
TypeError: cannot unpack non-iterable StatusCode object现在 lmdeploy 里面就有 llama_service，干脆适配一次 lmdeploy，废弃掉内部的吧。
需要用这个来测非对称量化版本精度。
部分数据集下有 ‘ppl’ 和 ‘gen’ 两种评估方式，请问在leaderboard中是哪种评估方式？比如mmlu、ceval、gsm8k、hellaswag...
一直卡在evaluating，持续好几个小时。
None
No response
比较慢
比较慢
No response
无
a
之前评测了好几个数据集然后进程中断了。
文件夹下有这三个文件：
prediction里为每个任务的输出，我该怎么评测结果，尝试加上--mode eval -r 20230724_021658参数后生成的csv文件都是‘--’值
Running opencompass on my own SFT model, we saw some suspicious results.
dataset version metric mode model-hf
------------------------------------ --------- ---------------- ------ -----------------
--------- 语言 Language --------- - - - -
WiC d06864 accuracy gen 0.00
WSC 6dc406 accuracy gen 0.00
winogrande a9ede5 accuracy gen 0.00
--------- 推理 Reasoning --------- - - - -
AX_g 68aac7 accuracy gen 0.00
RTE 68aac7 accuracy gen 0.00
openai_humaneval 8e312c humaneval_pass@1 gen 0.00
mbpp 1e1056 score gen 0.00
{'CUDA available': True,
'CUDA_HOME': '/usr/local/cuda',
'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-SXM4-80GB',
'MMEngine': '0.8.1',
'NVCC': 'Cuda compilation tools, release 12.1, V12.1.66',
'OpenCV': '4.8.0',
'PyTorch': '2.0.1',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.8\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
' - CuDNN 8.7\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
'CUDNN_VERSION=8.7.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -Wno-deprecated '
'-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
'-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_QNNPACK '
'-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
'-Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wunused-local-typedefs '
'-Wno-unused-parameter -Wno-unused-function '
'-Wno-unused-result -Wno-strict-overflow '
'-Wno-strict-aliasing '
'-Wno-error=deprecated-declarations '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=redundant-decls '
'-Wno-error=old-style-cast '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0]',
'TorchVision': '0.15.2',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.0+18efcb0',
'sys.platform': 'linux'}Take WiC as example, the prediction of the model is either Yes or No.
{
"origin_prompt": "Human: Sentence 1: As he called the role he put a check mark by each student's name.\nSentence 2: A check on its dependability under stress.\nAre 'check' in the above two sentenses the same?\nA. Yes\nB. No\nAnswer: \n\nAssistant:",
"prediction": "Yes"
}But the labels are true or false, as can be seen below.
{"word": "brush", "sentence1": "She gave her hair a quick brush.", "sentence2": "The dentist recommended two brushes a day.", "idx": 3, "label": "true", "start1": 26, "start2": 28, "end1": 31, "end2": 35, "version": 1.1}如果是子集的话是哪些？
按照文档中的快速上手，数据及使用的是ceval，模型为OPT-125M；
运行python run.py configs/eval_demo.py -w outputs/demo时，特别慢，没有在GPU上运行，不知道是我哪里的问题；
log打印：
GPU情况：
一直卡着，没有进度。
python3.10
No response
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.