Conversation with RAG enabled Claude V2 is ending abruptly

[aws-solutions]

Conversation with RAG enabled Claude V2 is ending abruptly

[wzr1337]

I just checked the Lambda that I pulled the log from and its timeout is set to 15 min, so it should not be the issue

[wzr1337]

I changed the deployment settings to NOT use streaming and I get the same result: Corrupted response ending in the middle of the response.

I also tried Claude V2 and V2.1, both have the same issue

[wzr1337]

I seem to have found the issue.. It is a too tighly set max_tokens_to_sample based on DEFAULT_MAX_TOKENS_TO_SAMPLE by changing it to 4096 the response does not get canceled anymore

[ihmaws]

Hi @wzr1337, thanks for the report and for taking the time to work through a detailed analysis of the issue.

As you've identified, many LLMs come with the ability to truncate/cap the number of output tokens generated. One of the primary things this helps to do is control costs by providing an upper bound on the number of output tokens generated by the model (output tokens are substantially more costly than input tokens).

For the anthropic family of models this can be controlled by a model parameter called max_tokens_to_sample (for their text completion API).

At the time this integration was first built, the default value for this parameter used by Anthropic's API was 256, so we chose to match this. This may have changed since the first release, so we will have a look and reassess for our next release.

That being said, the solution allows for customers to override any of the default model parameters and provide their own. You can do this at deployment time by setting an advanced model parameter at the model selection step (see example screenshot below). The parameters supported depends on the specific model used, so consult the model documentation.

Adding Advanced Parameters: