没有function调用了吗？im_start_id 和 im_end_id 这些都没有了？

[zhiguoxu]

没有function调用了吗？im_start_id 和 im_end_id 这些都没有了？

[JianxinMa]

如 README 所说：

NOTE: Neither the vLLM nor SGLang APIs currently offer built-in support for function calling. If you require function calling capabilities, please refer to the Qwen-Agent project, which provides a wrapper around these APIs to support function calling.

在 Qwen-Agent 项目有提供一层 wrapper 来提供函数调用

[zhiguoxu]

那我应该如何防止特殊token注入呢？

[jklj077]

From what I understand, <|im_start|> and <|im_end|> does not prevent the misuse of function calls (injection of function calls that should not be generated) nor does it protect the running of arbitrary code (in terms of defintion of functions). In that sense, you should always run the model in a secure, isolated environment.

To prevent the injection of special tokens from user inputs, you could try:

>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained(path_to_model, split_special_tokens=True, use_fast=False)

>>> tokenizer("<|im_start|>This is a test.<|im_end|><|endoftext|>") # safe tokenization
{'input_ids': [27, 91, 318, 4906, 91, 29, 1986, 374, 264, 1273, 15757, 91, 318, 6213, 91, 1784, 91, 8691, 723, 427, 91, 29], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

>>> tokenizer.convert_tokens_to_ids("<|im_start|>") # get special token ids manually
151644

>>> tokenizer.convert_tokens_to_ids("<|im_end|>") # get special token ids manually
151645

You could then manually construct the model inputs with the chatml template.