python fastchat.serve.openai_api_server

[herexk]

python fastchat.serve.openai_api_server

[ScarletPan]

Hi. Can you check whether you have enough GPU memory to run the model?

[herexk]

Thanks for your reply，I am using a local model, and the baichuan2-13B-mat model loads successfully. When running this command, it can provide answers. The command is as follows: curl http://0.0.0.0:6006/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "kagentlms_baichuan2_13b_mat", "messages": [{"role": "user", "content": "刘德华是谁？"}]}'

[ScarletPan]

Can you put more tokens on content to check what it happens?, like >2k. If the same error happens, please set a small number of --max_tokens_num on kagentsys commands

[herexk]

我是在，使用“kagentsys --query="刘德华是谁？" --llm_name="kagentlms_baichuan2_13b_mat" --use_local_llm --local_llm_host="0.0.0.0" --local_llm_port=6006 --lang="zh" ”时，触发的，在运行fastchat.serve.openai_api_server 的端口 报的“| INFO | stdout | INFO: 127.0.0.1:59622 - "POST /v1/completions/ HTTP/1.1" 307 Temporary Redirect”这个错误

[ScarletPan]

嗯嗯，我怀疑是KAgentSys的prompt太长导致你的显存不够报错了，重启下模型，加下--max_tokens_num=512试一试，如果可以的话，把log也放一下

[herexk]

在运行时的日志，python -m fastchat.serve.vllm_worker --model-path /root/autodl-tmp/kagentlms_baichuan2_13b_mat --trust-remote-code --dtype half --host 0.0.0.0
INFO 01-03 09:42:07 async_llm_engine.py:371] Received request c9e4a747627e462e9ba9cd77c16adbf4: prompt: '<reserved_106>你是AI助手，你能帮助人类解决他们的问题\n\n当前阶段是任务规划阶段，你将给定目标或问题，你的决策将独立执行而不依赖于人类的帮助，请发挥LLM的优势并且追求高效的策略进行任务规划。\n1.你有~4000字的短期记忆\n2.不需要用户的帮助\n3.规划的时候可以用参考工具中提到的工具\n4.互联网搜索、信息聚合和鉴别真伪的能力\n5.保持谦逊，对自己没把握的问题，尽可能调用command，但尽量少调用，不能重复调用\n6.当你从自身知识或者历史记忆中能得出结论，请聪明且高效，完成任务并得出结论\n7.经常建设性地自我批评整个行为大局，反思过去的决策和策略，以改进你的方法\n8.你最多只能进行1步思考，规划1个任务，所以尽可能高效规划任务\n9.你有反思能力，如果已完成的任务和结果暂不能得到回答问题所需信息或尚不能完成目标，应继续规划，但不能跟之前任务重复\n\nCommands:\n1:{"name": "web_search", "description": "Perform an internet search.", "parameters": {"type": "object", "properties": {"text": {"type": "str", "description": "Search query."}}}, "returns": {"description": "", "type": "str"}, "required": ["text"]}\n2:{"name": "browse_website", "description": "Browse a specific website using the provided URL link. ", "parameters": {"type": "object", "properties": {"url": {"type": "str", "description": "The website's URL link."}, "question": {"type": "str", "description": "The specific content or topic sought on the website."}}}, "returns": {"description": "", "type": "str"}, "required": ["url", "question"]}\n3:{"name": "get_weather_info", "description": "Retrieve weather information for specified locations and dates.", "parameters": {"type": "object", "properties": {"location": {"type": "str", "description": "Locations in English separated by commas, e.g., \"Beijing,Vancouver,...,Chicago\"."}, "start_date": {"type": "str", "description": "Start date in format \"yyyy-MM-dd\"."}, "end_date": {"type": "str", "description": "End date in format \"yyyy-MM-dd\"."}, "is_current": {"type": "str", "description": "\"yes\" or \"no\" indicating if current time's weather is desired."}}}, "returns": {"description": "", "type": "str"}, "required": ["location", "start_date", "end_date", "is_current"]}\n4:{"name": "get_calendar_info", "description": "Retrieve calendar details between specified dates.", "parameters": {"type": "object", "properties": {"start_date": {"type": "str", "description": "Start date in the format \"yyyy-MM-dd\"."}, "end_date": {"type": "str", "description": "End date in the format \"yyyy-MM-dd\"."}}}, "returns": {"description": "", "type": "str"}, "required": ["start_date", "end_date"]}\n5:{"name": "time_delta", "description": "Calculate the time interval between two timestamps.", "parameters": {"type": "object", "properties": {"start_time": {"type": "str", "description": "format of \"yyyy-MM-dd HH:mm:ss\"."}, "end_time": {"type": "str", "description": "format of \"yyyy-MM-dd HH:mm:ss\"."}}}, "returns": {"description": "", "type": "str"}, "required": ["start_time", "end_time"]}\n6:{"name": "get_solar_terms_info", "description": "Retrieve solar terms in Chinese for a given year. ", "parameters": {"type": "object", "properties": {"year": {"type": "int", "description": "Target year for query."}}}, "returns": {"description": "", "type": "str"}, "required": ["year"]}\n7:{"name": "do_nothing", "description": "Do nothing. Other functions are unsuitable due to inability to determine correct parameters from the query or no matching function exists.", "parameters": {"type": "object", "properties": {}}, "returns": {"description": "", "type": ""}, "required": []}\n8:{"name": "task_complete", "description": "Indicate task completion without the need for further functions. ", "parameters": {"type": "object", "properties": {}}, "returns": {"description": "", "type": ""}, "required": []}\n\n\n当前阳历日期和时间: 2024-01-03 09:42:06.002120\n当前星期: 星期三\n当前农历日期: 二〇二三年冬月廿二\n当前时辰: 巳时\n\n\n\nGOAL:你好\n\n\n根据目标和已有任务，规划一个新Task(不能重复)，你只能以以下json列表的格式生成Task\n{\n "task_name": "任务描述",\n "command":{\n "name":"command name",\n "args":{\n "arg name":"value"\n }\n }\n}\n确保Task可以被Python的json.loads解析\n\n当已完成的Tasks已经能够帮助回答这个目标，则尽可能生成任务完成Task，否则生成一个其他Task。一个新Task:<reserved_107>', sampling params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, temperature=0.1, top_p=0.75, top_k=40, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=['', '###'], ignore_eos=False, max_tokens=512, logprobs=None, skip_special_tokens=True), prompt token ids: None.

在运行python -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 6006 的日志

2024-01-03 09:42:07 | INFO | stdout | INFO: 127.0.0.1:50622 - "POST /v1/completions/ HTTP/1.1" 307 Temporary Redirect

[herexk]

而且 我也没找到 --max_tokens_num=512 这个参数可以在哪里设置，加载模型的时候，并不能设置这个参数。麻烦大佬看一下。十分感谢！

[ScarletPan]

--max_tokens_num=512这个参数是在运行系统命令kagentsys的时候添加 ；如果这个能运行的话，说明是超显存了。你可以再给我看看你的nvidia-smi么？

[herexk]

是的，就是prmpt过长的原因，我在设置max_tokens_num 之后就正常能使用了，十分感谢！