Large files -- Can this project work with large files?

[hchaudhary1]

Large files -- Can this project work with large files?

[freuk]

Hi @hchaudhary1!

• You can't use this demo tool on such a large file, because of Groq's current choice of deployed models.

  5000 lines assuming 10 tokens/line goes beyond 32k tokens, which is the maximum context window offered by any model Groq currently hosts (mixtral-8x7b-32768).

  I think we can expect improvements there. And then the answer will be yes.

• (b) It expects the LLM to provide the entire source.

  I have used zero tricks to compress the LLM bandwidth, because Groq inference is so fast anyways, it's just easier to have it generate the full source every time. For "improving on an ongoing diff", I do provide the original source and the changes in diff form though. IIRC, that was mostly done because models understood that better in my experiments.

  I have just added #3, so if you run the tool, you can set log-llm in the config file (look in the demo folder for an example) and see the full LLM interaction logs there. You can also browse the source if you are comfortable with Haskell, the prompts are hardcoded in app/Main.hs.