关于few shots的问题

[baai-agents]

关于few shots的问题

[DVampire]

Hi, a message that we call GPT-4V API is divided into 4 parts:

1. system message: Provide a comprehensive introduction to GPT-4V, mainly describing the games GPT-4V is currently playing and its role positioning.
2. user message part1: This is the text description part before the few-shot examples that include images, such as the current task definition and description. Clearly, it cannot be used as an instruction for the image. Therefore, following logical order, we place this preceding description of the few-shot example within a user message.
3. image introduction message：This includes some few-shot examples, images, and their instructions. Because few-shot examples may contain replies from GPT-4V as assistant messages, we combine the few shots and the new images and their prompts into this part of the message.
4. user message part2: This part is the specific prompt, for example, some observable information, historical information about reflection, including the constraints and formats of the output.

Finally, all the message items will be combined into a complete message to call the GPT-4V API to get response.

You can refer to the template of the prompts: https://github.com/BAAI-Agents/Cradle/tree/main/res/prompts

A minor note: To ensure everyone can participate in the discussion, it would be better if communication could be in English. Thank you very much for your attention and support.

[qinxiangyujiayou]

Hi, a message that we call GPT-4V API is divided into 4 parts:

1. system message: Provide a comprehensive introduction to GPT-4V, mainly describing the games GPT-4V is currently playing and its role positioning.
2. user message part1: This is the text description part before the few-shot examples that include images, such as the current task definition and description. Clearly, it cannot be used as an instruction for the image. Therefore, following logical order, we place this preceding description of the few-shot example within a user message.
3. image introduction message：This includes some few-shot examples, images, and their instructions. Because few-shot examples may contain replies from GPT-4V as assistant messages, we combine the few shots and the new images and their prompts into this part of the message.
4. user message part2: This part is the specific prompt, for example, some observable information, historical information about reflection, including the constraints and formats of the output.

Finally, all the message items will be combined into a complete message to call the GPT-4V API to get response.

You can refer to the template of the prompts: https://github.com/BAAI-Agents/Cradle/tree/main/res/prompts

A minor note: To ensure everyone can participate in the discussion, it would be better if communication could be in English. Thank you very much for your attention and support.

Thank you very much for your timely reply and patient explanation. Fortunately, your explanation of the prompt handling process closely aligns with my previous understanding. However, there is still one issue that perplexes me:

In the section of your code related to the image introduction message:

paragraph_input = params.get(constants.IMAGES_INPUT_TAG_NAME, None)

if I understand correctly, it exclusively extracts the content tagged as IMAGES_INPUT_TAG_NAME (i.e., 'image_introduction') from params. Yet, I have not found the code responsible for extracting the "few shot" sections, such as those within decision_making.json:

"few_shots": [
{
"introduction": "Here are some examples of the positions of the bounding box shown in the image.",
"path": "",
"assistant": ""
},…]

[DVampire]

The few_shots field is reserved for updating extensions. In the current version, this field is not used.

In the current version few shot examples are included in the image_introduction. So only the image_introduction field is parsed.

Thank you.