@midribbon_action@lemmy.blahaj.zone
2026-10-07 01:46 UTC
KV caching doesn’t persist between prompts, it is an internal optimization used while generating text. I don’t see how it’s relevant.
Edit: it kinda sounds like you’re technobabbling me, like “but it uses a recursive algorithm, it must be alive, check mate!” Your question is a non sequitur.
Replies (1)
-
@morrowind@lemmy.ml 2026-10-07 02:12
I’m not techno babbling you. I specifically said “if you consider the model + kv cache together,” e.g preserved. A kv cache is not an internal optimization, a model cannot generate text without a kv cache. There’s a separate idea of caching the kv cache between requests as an optimization. That’s not the one I’m talking about. Or if it helps, consider it just within one generation. The model + kv cache. Does that not match your definition?