Research2h ago
KVBoost Reduces LLM Prefill Latency
A dual-hash keying scheme allows KVBoost to reuse key-value cache chunks regardless of their position in a prompt. This removes the requirement for shared content to be a contiguous leading prefix. The system uses deviation-guided recomputation to fix attention boundary errors. Developers can now optimize inference for prompts with non-linear shared content.