success | load in 8 bit. it runs on one-3090ti (24G)

[openlmlab]

success | load in 8 bit. it runs on one-3090ti (24G)

[littlestone0806]

老哥，我用你的代码，加载模型时不报错，提问题时报这个错
Traceback (most recent call last):
File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\routes.py", line 401, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1302, in process_api
result = await self.call_function(
File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1025, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "C:\Users\stone\Desktop\MOSS\moss_gui_demo(8bit).py", line 153, in predict
outputs = model.generate(
File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 1485, in generate
return self.sample(
File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 2560, in sample
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either inf, nan or element < 0

显卡M40 24G cuda11.6 pytorch 1.13.1+cu116

[stevezhang88]

老哥，我用你的代码，加载模型时不报错，提问题时报这个错 Traceback (most recent call last): File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\routes.py", line 401, in run_predict output = await app.get_blocks().process_api( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1302, in process_api result = await self.call_function( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1025, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 867, in run result = context.run(func, *args) File "C:\Users\stone\Desktop\MOSS\moss_gui_demo(8bit).py", line 153, in predict outputs = model.generate( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 1485, in generate return self.sample( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 2560, in sample next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either inf, nan or element < 0

显卡M40 24G cuda11.6 pytorch 1.13.1+cu116

generate函数我使用的是MOSS原文中的方式。目前来看，可以运行。你先试试FastChat项目，能否正常运行？看看是不是transformers的库或者tokenizer库需要升级了？

[956237586]

老哥，我用你的代码，加载模型时不报错，提问题时报这个错 Traceback (most recent call last): File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\routes.py", line 401, in run_predict output = await app.get_blocks().process_api( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1302, in process_api result = await self.call_function( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1025, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 867, in run result = context.run(func, *args) File "C:\Users\stone\Desktop\MOSS\moss_gui_demo(8bit).py", line 153, in predict outputs = model.generate( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 1485, in generate return self.sample( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 2560, in sample next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either inf, nan or element < 0

显卡M40 24G cuda11.6 pytorch 1.13.1+cu116

remove do_sample=True can pass the error
refer: THUDM/ChatGLM-6B#31 (comment)

[littlestone0806]

老哥，我用你的代码，加载模型时不报错，提问题时报这个错 Traceback (most recent call last): File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\routes.py", line 401, in run_predict output = await app.get_blocks().process_api( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1302, in process_api result = await self.call_function( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1025, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 867, in run result = context.run(func, *args) File "C:\Users\stone\Desktop\MOSS\moss_gui_demo(8bit).py", line 153, in predict outputs = model.generate( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 1485, in generate return self.sample( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 2560, in sample next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either inf, nan or element < 0
显卡M40 24G cuda11.6 pytorch 1.13.1+cu116

remove do_sample=True can pass the error refer: THUDM/ChatGLM-6B#31 (comment)

感谢您的回答，这样虽然不报错了，但是会卡很久没有回复

[littlestone0806]

老哥，我用你的代码，加载模型时不报错，提问题时报这个错 Traceback (most recent call last): File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\routes.py", line 401, in run_predict output = await app.get_blocks().process_api( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1302, in process_api result = await self.call_function( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\gradio\blocks.py", line 1025, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\anyio_backends_asyncio.py", line 867, in run result = context.run(func, *args) File "C:\Users\stone\Desktop\MOSS\moss_gui_demo(8bit).py", line 153, in predict outputs = model.generate( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 1485, in generate return self.sample( File "C:\Users\stone\Desktop\MOSS\env\lib\site-packages\transformers\generation\utils.py", line 2560, in sample next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either inf, nan or element < 0
显卡M40 24G cuda11.6 pytorch 1.13.1+cu116

remove do_sample=True can pass the error refer: THUDM/ChatGLM-6B#31 (comment)

为了验证是不是只有MOSS有这个问题，我特意在ChatGLM-6B上用了相同的办法，同样也是卡住很久没有回复