[Bug]: Qwen1.5-72B L20x8 latest vLLM TPOT slower than v0.4.0.post, 48ms vs 39ms, why?

[DefTruth]

[Bug]: Qwen1.5-72B L20x8 latest vLLM TPOT slower than v0.4.0.post, 48ms vs 39ms, why?

[DefTruth]

@youkaichao @rkooo567

[DefTruth]

command：

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
python3 -m vllm.entrypoints.openai.api_server \
    --model Qwen1.5-72B-Chat \
    --tensor-parallel-size 8 \
    --max-model-len 8192 \
    --trust-remote-code \
    --disable-custom-all-reduce \
    --enable-prefix-caching \
    --tokenizer-mode slow \
    --enforce-eager \
    --gpu-memory-utilization 0.9 \
    --port 8861

[youkaichao]

If you can biset to find out the commit that leads to the degradation, that would be helpful. Otherwise, it is very difficult to answer a generic report of a performance regression.

[DefTruth]

If you can biset to find out the commit that leads to the degradation, that would be helpful. Otherwise, it is very difficult to answer a generic report of a performance regression.

vLLM seems to have been under reconstruction recently. I'm not quite sure what's causing TPOT to be slow

[youkaichao]

most commits should be runnable since they pass the ci tests. it's not related with the refactoring. just bisect to find out the potentially commit that leads to this regression.

[DefTruth]

I have test on L20, not sure the device is same as in CI.

[DefTruth]

with cuda graph: 35.7ms -> 36.7ms, without cuda graph: 39ms -> 45ms