Input tokens somewhat surprising (471k for 89 requests)

[MarkHelsinki]

Input tokens somewhat surprising (471k for 89 requests)

[MarkHelsinki]

well, I changed strategy and consolidated my API calls, since this isn't a 'chat' bot inside the game but just leveraging AI to produce game data. But I think more than anything, switching back to 3.5 seems to have dropped the count. Even the previous method when I tested the token count it was only incrementing by 100 each time, so 100 bigger calls each time because it was accumulating the chat history when it didn't really need to. So, I created bigger batch calls and inputs, modified the parsing for the outputs and that solved my particular use case.

OpenAI really should enable single use calls that don't generate a history. I guess simply reinitializing everything does the same thing, but doing this at runtime adds some memory overhead, no?

Anyhow, problem solved for me, though I'm still scratching my head about what turned out to be 580k of input tokens for less than 100 simple calls.