[Feature Requests] Do not limit suggestions

[esauser]

[Feature Requests] Do not limit suggestions

[mrT23]

There is no such thing as "all suggestions"

• Ask GPT-4 for 100 suggestions, you might get 100 suggestions. But most of them will probably be bad\repetative\simplistic. Without a cap, again you might get bad\repetative\simplistic suggestions

• In addition, there is a limit on the number of output tokens, so even without a cap, you won't get "all suggestions".

• Limiting the number of output suggestions is the right way to go, as we try to encourage the model to focus on the most important ones. We can still use a relatively high cap (~7-8). Reviewing each suggestion still takes time from a human.

I recommend reviewing the prompt. Try to remove lines 8 and 24, and you will see you still don't get "all suggestions", as there is no such thing.

[esauser]

How about all the most important ones? Setting an arbitrary limit doesn't make sense either. The two options seem to be ditch to ditch. How do we find the happy medium?

[mrT23]

the prompt says " up to {{ num_code_suggestions }} ". It can provide less

More generally, It's always a challenge to prompt LLM, especially when the input is large and complicated like we have in PRs code. No prompt will give you 100% success, 100% of the time.

I recommend not relying on PR-Agent suggestions (or any other AI tool) to find every possible problem. The code suggestions are ... suggestions.
Use them as an extra reviewer to inspect and improve the code, but not as a replacement for baseline human inspection.

[esauser]

We are not relying on them for anything, we are using them as an extra tool. The point I'm trying to make is that for a very large PR it makes sense to get more suggestions, and for a very small PR it makes sense to get less suggestions. There ought to be a way to navigate this.

[mrT23]

see #221 (comment), i am thinking about a tool to address this

[zmeir]

I agree with @esauser, and also wanted to add that I tested num_code_suggestions configuration with large values (50) just to see what it does, and in moderate PRs (100-ish lines) it doesn't "explode" - it just gives me a few code suggestions, but it's usually capped at around 10 and doesn't actually reach 50 (as you would expect in a PR of that size). That is it say - it looks like the "up to" part of the prompt is really taken into account (at least in these cases).

[mrT23]

@esauser
this tool ("/improve --extend"):
#229

Should address all the issues you raised.

It allows for:

• no compression of diff files
• generates code suggestions for each group of diff files, and aggregates these suggestions. (hence number of code suggestions will increase for larger PRs)
• rank and sort suggestions based on their importance
• clip the least important suggestions.