FlashMemory-DeepSeek-V4: Cutting KV Cache to 13.5% for 500K Context

Quick answer

FlashMemory-DeepSeek-V4 trims the decoding KV cache to 13.5% of a full-context baseline while keeping downstream accuracy flat or up by about 0.6 points on average. It does this with Lookahead Sparse Attention (LSA): a small neural indexer predicts which key-value chunks the next tokens will actually attend to, and only those chunks stay resident in GPU memory. At 500K-token context, the method cuts the KV cache footprint by over 90%.

The KV cache problem it attacks

Long-context decoding does not blow up on compute first — it blows up on memory. Every token a transformer has already seen leaves a key and value vector in cache, and that cache grows linearly with context length. At 500K tokens the KV cache, not the model weights, becomes the thing that won’t fit on the GPU. Existing fixes either evict tokens by past attention scores (which is reactive and throws away chunks that matter later) or compress everything uniformly (which blurs the few chunks that carry the answer). FlashMemory-DeepSeek-V4’s bet is that you can predict, before decoding the next span, which historical chunks it will query — and keep only those hot.

How Lookahead Sparse Attention works

LSA splits the KV history into chunks and puts a learned Neural Memory Indexer in front of attention. The indexer scores chunks by how relevant they are to the upcoming query and surfaces the top ones into GPU memory; the rest stay offloaded. The “lookahead” part is the key design choice: instead of deciding what to keep from what was attended to in the past, the indexer anticipates the next span’s needs, so a chunk that goes quiet for thousands of tokens and then becomes critical can be pulled back before it is needed.

The indexer is a dual-encoder — one encoder embeds the query state, one embeds the candidate chunks, and relevance is a dot product, the same shape as a dense retrieval system. That framing is what makes the training cheap.

Why decoupled training is the real trick

The training story matters more than the inference one. Normally, training a component that lives inside attention means loading the full backbone model into GPU memory while you train it — for a DeepSeek-V4-scale model that is brutal. FlashMemory-DeepSeek-V4 formulates the indexer as a standard dual-encoder and trains it independently with off-the-shelf retrieval frameworks, never loading the massive backbone. That decoupling is the practical contribution: it turns “modify the attention of a frontier model” into “train a retriever,” which a far smaller team can actually run.

Key results

• KV cache: 13.5% of full-context baseline on average across the evaluated tasks — a roughly 7x reduction in resident cache.
• Accuracy: +0.6% absolute on average, so the sparsification is effectively free on the benchmarks tested rather than a quality-for-memory trade.
• At 500K context: over 90% KV cache overhead suppressed, which is where dense decoding simply runs out of memory.
• Evaluated on LongBench-v2, LongMemEval, and RULER — the standard long-context and long-memory suites, not a single bespoke benchmark.
• Code and weights are released on GitHub and Hugging Face under the libertywing/FlashMemory-Deepseek-V4 repos.

Limits and open questions

The honest caveat is that “+0.6% on average” hides the variance: an indexer that mispredicts which chunk matters can drop the exact span holding the answer, and aggregate accuracy won’t show a single catastrophic miss on a needle-in-haystack query. The paper reports averages on retrieval-friendly benchmarks; reasoning that must integrate many scattered chunks at once is the harder case and is where lookahead prediction is most likely to leak. The indexer also adds its own latency and a second model to serve, so the 90% memory win is not a 90% cost win. And as an 11-page technical report tied to a specific DeepSeek-V4 backbone, how cleanly LSA ports to other architectures and how the dual-encoder holds up beyond 500K are still open.

FAQ

What is Lookahead Sparse Attention in FlashMemory-DeepSeek-V4?

It is an attention scheme that keeps only the key-value chunks a neural indexer predicts the next tokens will query, instead of holding the entire KV cache. By anticipating future demand rather than reacting to past attention, it keeps memory bounded while preserving the chunks that carry the answer.

How much does FlashMemory-DeepSeek-V4 reduce KV cache?

It cuts physical KV cache to 13.5% of a full-context baseline on average — about a 7x reduction — and suppresses over 90% of KV cache overhead at 500K-token context, with average downstream accuracy flat to roughly +0.6%.

Why is FlashMemory-DeepSeek-V4’s training cheaper than other sparse-attention methods?

Because the indexer is framed as a standard dual-encoder and trained with ordinary retrieval frameworks, the backbone model is never loaded into GPU memory during indexer training. That decoupling turns a frontier-model attention change into a retrieval training job that small teams can run.

What benchmarks does FlashMemory-DeepSeek-V4 use?

LongBench-v2, LongMemEval, and RULER — established long-context and long-memory benchmarks — rather than a single custom test, which makes the memory-vs-accuracy claim more credible.

One line: predict which KV chunks the next tokens need, keep only those, and 500K-token decoding fits with no accuracy hit. Read the original paper on arXiv.