web-llm integration

[louis030195]

web-llm integration

[lgrammel]

AI SDK core could also be used on the client side. That said, you are right, useChat etc require a server connection. What you could do is use e.g. AI SDK Core streamObject or streamText client side and then operate directly on those results (without useChat et al).

[lgrammel]

You could implement a custom provider (see https://sdk.vercel.ai/providers/community-providers/custom-providers ) that directly calls web llm. The llamacpp community provider does something similar (in process communication) I think.

[louis030195]

@lgrammel i thought about this but isn't provider called on server side?

basically wants to leverage the hooks like useChat, generative UI stuff etc. but with client side LLM

[louis030195]

Is this sdk going to have better support for client side LLMs? Clients have more and more powerful AI accelerators, future apps will use both client and server side LLMs

Got it working but too much effort to make it work with all the generative UIs features atm which assume a lot of server side stuff.
Depending update of this issue will implement my own lib for client side generative UI