How is it possible to do non-prefix KV cache? I was under the impression that th... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		dist-epoch 8 months ago \| parent \| context \| favorite \| on: Lossless LLM 3x Throughput Increase by LMCache How is it possible to do non-prefix KV cache? I was under the impression that the V for one token potentially depends on the V of all previous ones.

da-x 8 months ago [–]

Yes, there's KV cache 'Blending' see [1].

Future versions of LMCache are aiming to support this.

[1] CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion- https://arxiv.org/abs/2405.16444

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact