Stream never closes with LangChainStream using postman

[vercel]

Stream never closes with LangChainStream using postman

[e-roy]

Here is another approach to trigger onCompletion with no wrapper

import { StreamingTextResponse, LangChainStream, Message } from 'ai'
import { CallbackManager } from 'langchain/callbacks'
import { ChatOpenAI } from 'langchain/chat_models/openai'
import { AIChatMessage, HumanChatMessage } from 'langchain/schema'

export const runtime = 'edge'

export async function POST(req: Request) {
  const { messages } = await req.json()

  const { stream, handlers } = LangChainStream({
    onStart: async () => console.log('start'),
    // onToken: async token => console.log(token),
    onCompletion: async () => console.log('end')
  })

  const llm = new ChatOpenAI({
    streaming: true,
    callbackManager: CallbackManager.fromHandlers(handlers)
  })

  llm
    .call(
      (messages as Message[]).map(m =>
        m.role == 'user'
          ? new HumanChatMessage(m.content)
          : new AIChatMessage(m.content)
      )
    )
    .catch(console.error)
    .finally(() => {
      // Call handleStreamEnd when the chat or stream ends
      handlers.handleChainEnd()
    })

  return new StreamingTextResponse(stream)
}

[nfcampos]

@aranlucas the issue is actually that you're passing handlers for chain events to the constructor of a chat model, therefore they will never be called. You should pass the handlers either to the constructor of the chain you're using or to the .call() method

[nfcampos]

I can confirm Langchain does call handleLLMEnd, see the screenshot attached, so it must be an issue with something in this vercel library

[MichaelReaNZ]

Have had the same issue which makes onCompletion not work but also the onLoading in the UI.

[MichaelReaNZ]

Thanks e-roy.

.finally(() => {
      // Call handleStreamEnd when the chat or stream ends
      handlers.handleChainEnd()
    })

Is a good workaround 🙂

[feliche93]

@e-roy thanks for the snippet, also solved my problem 👏

[aranlucas]

This also causes isLoading from useChat() to never go back to false when using LangChainStream.

However, e-roy's fix seems to work.

[Itsnotaka]

Had the same issue and the workaround I used was to just overwrite the callback after llm.call

[Otoris]

@Itsnotaka Could I trouble you to share an example?

[e-roy]

I noticed the same issue with the example.

One workaround to get to work is add an wrappedCall

  const { messages } = await req.json()

  const { stream, handlers } = LangChainStream({
    onStart: async () => console.log('start'),
    onToken: async token => console.log(token),
    onCompletion: async () => console.log('end')
  })

  const llm = new ChatOpenAI({
    streaming: true,
    callbackManager: CallbackManager.fromHandlers(handlers)
  })

  async function wrappedCall(messages: Message[], onCompletion: () => void) {
    try {
      await llm.call(
        messages.map(m =>
          m.role == 'user'
            ? new HumanChatMessage(m.content)
            : new AIChatMessage(m.content)
        )
      )
    } catch (error) {
      console.error(error)
    } finally {
      onCompletion()
    }
  }

  wrappedCall(messages as Message[], () => {
    console.log('end')
  })

  return new StreamingTextResponse(stream)

The issue might be within the models in the langchain library to expose a completion event or callback.

[jasan-s]

another solution:

new ChatOpenAI({
	temperature: 0,
	modelName: 'gpt-3.5-turbo',
	maxTokens: 512, // Choose the max allowed tokens in completion
	streaming: true,
	callbackManager: CallbackManager.fromHandlers({
		async handleLLMNewToken(token: string) {
		// console.log({ token })
		handlers.handleLLMNewToken(token)
	        },
		async handleLLMEnd(output) {
		console.log('End of stream...', output)
		handlers.handleChainEnd(output)
		},
		async handleLLMError(e) {
		console.log('Error in stream...', e)
		handlers.handleLLMError(e)
		},
	}),
})

[aranlucas]

I think the issue is caused by be90740

When you do what @nfcampos suggested overwriting the returned handlers

  const { stream, handlers } = LangChainStream();

  const model = new ChatOpenAI({
    temperature: 0,
    streaming: true,
    callbacks: CallbackManager.fromHandlers({
      handleLLMNewToken: handlers.handleLLMNewToken,
      handleChainEnd: async () => {
        console.log("handleChainEnd");
        await handlers.handleChainEnd();
      },
      handleLLMEnd: async () => {
        console.log("handleLLMEnd");
        await handlers.handleChainEnd();
      },
      handleLLMError: handlers.handleLLMError,
    }),
  });

handleChainEnd never ends up in showing in the logs. (For my use case, using ConversationalRetrievalQAChain)

It seems that depending on which chain/model you use, handleChainEnd is not guaranteed to be called.

It looks like adding handleLLMEnd (keeping handleChainEnd) should fix this issue, but I'm not sure what issue @jaredpalmer might have been fixing when switching handleLLMEnd -> handleChainEnd

[aranlucas]

Yeah took me a while to get to that conclusion in #205 (comment).