Implement kv cache sparsity like H2O with attention score

[HarryWu99]

Implement kv cache sparsity like H2O with attention score

[amyeroberts]

cc @ArthurZucker @gante

[gante]

Hey @HarryWu99 👋

Techniques that improve memory utilization with LLMs are always exciting! At first glance, it seems like a good candidate to be added to transformers with the API you showcased in your example. Two additional points for consideration:

1. Benchmarks need to be run before merging, to confirm the implementation is working as expected;
2. You mentioned changes in LlamaAttention.forward, to use the attention scores. We may need a new function for that, like Cache.post_process(), we may need to iterate on the design throughout the PR.

If you're happy with these two points, we'd be happy to take your PR and guide you in the process 🤗

(P.S. your first link is to the DeepSeek-V2 paper, I'm assuming you meant the H2O paper :) )