[Bug]: Running llama2-7b on H20, Floating point exception (core dumped) appears on float16

[yk1012664593]

[Bug]: Running llama2-7b on H20, Floating point exception (core dumped) appears on float16

[chk4991]

After testing, the --enforce-eager parameter does not work, but setting dtype to float32 works. However, this means that the quantized model cannot be deployed on h20.

[yk1012664593]

llama model init start
INFO 04-26 17:03:13 llm_engine.py:98] Initializing an LLM engine (v0.4.1) with config: model='/mnt/deep_learning_test/testsuite/dataset/llms_inference_llama7b-v2_accelerate/checkpoint/7B-V2/', speculative_config=None, tokenizer='/mnt/deep_learning_test/testsuite/dataset/llms_inference_llama7b-v2_accelerate/checkpoint/7B-V2/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=Falsequantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0)
INFO 04-26 17:03:13 utils.py:613] Found nccl from library /root/.config/vllm/nccl/cu12/libnccl.so.2.18.1
INFO 04-26 17:03:16 selector.py:77] Cannot use FlashAttention-2 backend because the flash_attn package is not found. Please install it for better performance.
INFO 04-26 17:03:16 selector.py:33] Using XFormers backend.
INFO 04-26 17:03:25 model_runner.py:173] Loading model weights took 12.5523 GB
INFO 04-26 17:03:25 gpu_executor.py:119] # GPU blocks: 9217, # CPU blocks: 512
INFO 04-26 17:03:26 model_runner.py:977] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 04-26 17:03:26 model_runner.py:981] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.
Floating point exception (core dumped)

[youkaichao]

Does it occur everytime? Or only for certain prompt?

In addition:

If you experienced crashes or hangs, it would be helpful to run vllm with export VLLM_TRACE_FUNCTION=1 . All the function calls in vllm will be recorded. Inspect these log files, and tell which function crashes or hangs.

[yk1012664593]

它每次都会发生吗？还是只针对某些提示？

另外：

如果您遇到崩溃或挂起，使用 .vllm 中的所有函数调用都将被记录下来。检查这些日志文件，并判断哪个函数崩溃或挂起。export VLLM_TRACE_FUNCTION=1

Yes, this issue is inevitable. On the H20 model, all vllm versions with float16 accuracy will experience this error

[pipecat]

Met the same bug, and it's useful to add --enforce-eager to avoid it.

Additionally, some models (for example facebook/opt-125m) with float16 won't meet this bug.

[ElvisWai]

Met the same bug with Qwen1.5-14B-Chat on the H20, and I was able to solve it with float32. But using float16 and add --enforce-eager does not solve the problem.

[tianbiai]

Met the same bug with almost all LLM on the H20, but --enforce-eager does not solve the problem.

[caddfa31434]

Try to build vllm from source using the nvcr.io/nvidia/pytorch:24.04-py3 container to avoid the bug related to cuBLAS on specific shapes (on H20).

[mir-of]

I am using nvcr.io/nvidia/pytorch:23.10-py3，it's all right when I use the float32 and float16, but floating point exception when I use bfloat16.
I tried:

• build vllm from source using the nvcr.io/nvidia/pytorch:24.04-py3 container on H20 (still , float16 good, bfloat16 floating point exception)
• nvcr.io/nvidia/pytorch:23.10-py3 on H800 (all good)