Custom Plugin in Chat-Copilot Fetching Outdated Data Instead of Up-to-Date Results

[microsoft]

Custom Plugin in Chat-Copilot Fetching Outdated Data Instead of Up-to-Date Results

[crickman]

Thank you for reporting this. I'm happy to review chat-copilot's interaction with custom-plug-ins for result handling and if additional logging might be helpful.

I'm wondering if the model is cosolidating test, Test, and Test5 as a single logical item: Test. Is it possible to provide the prompt metrics (click the "information" icon in the chat bubble) and the short/long term memory values from the persona tab. This may help determine if actual values are provided and if the model then consolidates.

Another test might be to add distinct items such as Cheese, Celery, Catfood, etc...

I look forward to hearing back from you and will also investigate on this end.

[PBitukunti]

@crickman - Thank you for your prompt response. I appreciate your suggestion to explore whether the model is consolidating similar terms and to examine the prompt metrics and memory values.

I have conducted the tests as you've suggested. I created distinct items such as 'Cheese,' 'Celery,' and other 'Test' to ensure the items are unique. Despite this, the issue persists, and the plugin still fetches outdated data when interacting with Chat-Copilot.

I have attached the screenshots showing the error. The first screenshot demonstrates the outdated results from Chat-Copilot, while the second screenshot displays the prompt metrics and expected up-to-date results from Swagger for comparison. The third and fourth screenshots showcases the additional tests.

In addition, I've captured the short/long term memory values from the persona tab. I hope these details will be helpful in your investigation.

Please let me know if there's any other information or steps you'd like me to take to assist in resolving this issue. Your expertise is greatly valued, and I'm looking forward to your insights.

Thank you once again for your assistance.

[crickman]

Wonderful, thank you for these additional data-points. I aim to complete my analysis today.

[crickman]

I was able to reproduce your report with a function that adds a new guid to the list upon each invocation:

In my case, the "cached" value is being derived from the chat history. I suspect something similiar is occuring for you while the function result is being captured as expected. I am a little uncertain as to which prompt details you've shared...is that the planner prompt? I'd expect the prompt related to the plan/function/plug-in execution to contain the function results.

In anycase, let me raise this to the team for ideas.

[crickman]

I think we may have a work-around to try...unclear how effective it may be.