"You uploaded an unsupported image" when trying to streamText with image data

[lucasishuman]

"You uploaded an unsupported image" when trying to streamText with image data

[lgrammel]

The error comes from the OpenAI API. Can you test whether you also receive it when using the OpenAI fetch API directly?

[lucasishuman]

When I use the OpenAI SDK directly (as shown below) - I get a valid response from OpenAI.

Note - I need to change the message > content > image props to match OpenAI's message schema - maybe there is an issue mapping AI SDK image props to OpenAI image props in the AI SDK?

Using
"openai": "4.44.0"

Sent message (using OpenAI message schema)

    const messages = [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'What is in this image',
        },
        {
          type: 'image_url',
          image_url: {
            url: 'https://static01.nyt.com/images/2024/05/12/.....',
          },
        },
      ],
    }
];

Route handler

import OpenAI from "openai";

export async function POST(req: Request) {
  const { messages } = await req.json();

  const openai = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  });

  const response = await openai.chat.completions.create({
    model = 'gpt-4-turbo',
    messages,
  });

  console.log(response.choices[0]);
}

I feel like I must be missing something very obvious if no one else sees this behavior - but I also don't see an example in the AI SDK for messages with image content.

My OPENAI_API_KEY is in my env file and is valid.

[lucasishuman]

Maybe worth also noting is that client-side I am using the useChat hook, manually constructing the user message with image data/URL shown in original post, and then using append to send it to my app API.

[lucasishuman]

This is related to #1555 because when I remove the data:image/jpeg;base64, from the beginning of my base64 encoded image data string, I get a valid response.

However, when I pass a URL I still get an error.

So part of the fix may be just a matter of providing better documentation for messages with image content, but there's still something funky when sending an image URL.

[lgrammel]

@lucasishuman can you share the img?

[lucasishuman]

You can use any valid image URL to test, such as...

  const result = await streamText({
    messages: [
      {
        role: 'user',
        content: [
          {
            type: 'text',
            text: 'What’s in this image?',
          },
          {
            type: 'image',
            image:
              'https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg',
          },
        ],
      },
    ],
    model: openai('gpt-4-turbo'),
  });

[lgrammel]

@lucasishuman try using a URL

image: new URL('https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg')

See examples/ai-core/src/generate-text/openai-multimodal-url.ts for a full example

[lucasishuman]

@lgrammel - Great, thanks. That was the missing info I needed.

I feel like adding the image data and image URL examples to the AI SDK docs would be helpful.

You can close this issue.