Constrained beam search in chat

[esaks1]

Constrained beam search in chat

[esaks1]

I have not tested it enough to trust that it is a safe solution but what I ended up doing to fix this issue was to go to beam_search.py and slice the token_ids list in all calls to reset(), so that it does not include the prompt.

[gante]

Hi @esaks1 👋 Thank you for opening this issue!

Constrained generation is indeed not being actively maintained due to lack of bandwidth -- it deserves a proper dive at some point in the future.

Meanwhile, I'd like to suggest the following workarounds:

1. If your constraints are structured, i.e. you know the shape of your desired LLM output, have a look at this library: https://github.com/outlines-dev/outlines
2. If you simply want to fit certain words in the answer, you may consider passing sequence_bias to make those words more likely. If they become too repetitive with sequence_bias, you should be able to avoid repetitions with repetition_penalty (full docs)