BUG: Failed to launch qwe1.5-14b-GGUF using Xinferrence but works fine in pure llama_cpp python

[onionknightdd]

BUG: Failed to launch qwe1.5-14b-GGUF using Xinferrence but works fine in pure llama_cpp python

[qinxuye]

Is this issue related to #1038 ?

[onionknightdd]

Is this issue related to #1038 ?

seems not

qwen1.5_chat_gguf_q8 only takes 16GB , but there are 40GB for each GPU

the GPUs has enough memory when launch the model:
befor :

Then launch model with llama_cpp in 1 GPU:

or launch model with llama_cpp in 4 GPUs:

[zly9844]

Could you please show me the model.json file when register gguf-format model？ Thank you in advance!