Paper Review
DuoAttention: Efficient Long-Context Inference with Retrieval and Streaming Heads
Learns which attention heads actually retrieve from long context and gives only those a full KV cache, cutting long-context memory up to 2.55x with retrieval accuracy intact.
어떤 attention head가 실제로 long context에서 retrieval을 수행하는지 gate 최적화로 학습해 그 head에만 full KV cache를 주고, 나머지 streaming head는 상수 크기 cache로 줄여 retrieval 성능 손실 없이 최대 2.55배 메모리를 절감한다.
Background Long-context LLM serving is dominated by the KV cache, which grows linearly with context length across every attention head and layer and quickly exceeds GPU memory at million-token scales. Because decoding reads the whole cache at every step, this growth inflates not just memory but also per-token latency. Architecture-level mitigations like GQA shrink the constant factor but leave the linear growth in context length untouched.
배경 Long-context LLM 서빙에서는 모든 attention head와 layer의 KV cache가 context 길이에 비례해 커져 million-token 규모에서는 GPU 메모리를 금방 초과한다. decoding은 매 step마다 cache 전체를 읽기 때문에 이 증가는 메모리 용량뿐 아니라 token당 지연시간도 함께 키운다. GQA 같은 아키텍처 수준 완화책은 상수 계수를 줄일 뿐, context 길이에 대한 선형 증가 자체는 그대로 남는다.
Problem Caching full Key/Value states for all heads makes both pre-filling and decoding memory- and latency-bound, yet naively dropping cache entries destroys the model's long-context retrieval ability. The bind is that a token that looks unimportant now may be exactly what some head needs thousands of steps later — like a library discarding books based on this week's checkouts and only discovering the loss when a researcher asks for the rare volume. Any compression scheme therefore has to know, in advance, which parts of the model can genuinely afford to forget.
문제 모든 head에 full Key/Value state를 캐싱하면 pre-filling과 decoding 모두 메모리·지연시간에 묶이지만, cache entry를 단순히 버리면 모델의 long-context retrieval 능력이 무너진다. 문제의 핵심은 지금 안 중요해 보이는 token이 수천 step 뒤 어떤 head에는 꼭 필요할 수 있다는 점이다 — 이번 주 대출 순위만 보고 도서관 장서를 폐기했다가 희귀본을 찾는 연구자가 나타나서야 손실을 깨닫는 것과 같다. 따라서 압축 기법은 모델의 어느 부분이 정말로 잊어도 되는지를 미리 알아야 한다.
Prior limitations Token-level eviction methods (H2O, TOVA, StreamingLLM) make one shared keep/drop decision for entries that all heads must live with, so retrieval from deep context collapses once the needed tokens are gone. H2O and TOVA additionally rely on materialized attention scores, making them incompatible with FlashAttention during pre-filling, and FastGen's attention profiling has quadratic memory cost — the paper's experiments could only run it up to 24K (Llama-2) / 32K (Llama-3) tokens. The paper further argues that attention-score profiling is itself an unreliable instrument: it ignores the end-to-end output impact of dropping a token, the role of value states, and how attention distributions vary across inputs.
기존 한계 Token 단위 eviction(H2O, TOVA, StreamingLLM)은 모든 head가 공유해야 하는 entry에 대해 하나의 keep/drop 결정을 내리므로, 필요한 token이 사라지는 순간 깊은 context의 retrieval이 붕괴한다. H2O와 TOVA는 materialized attention score에 의존해 pre-filling에서 FlashAttention과 호환되지 않고, FastGen의 attention profiling은 메모리가 제곱으로 늘어 논문 실험에서도 24K(Llama-2)/32K(Llama-3) token이 상한이었다. 나아가 논문은 attention-score profiling 자체가 부정확한 측정 도구라고 지적한다 — token을 버렸을 때의 end-to-end 출력 영향, value state의 역할, 입력마다 달라지는 attention 분포를 모두 놓치기 때문이다.
Goal Cut long-context KV memory and latency structurally — with a fixed, input-independent cache layout — while preserving full-attention retrieval accuracy. Equally important, the design must stay compatible with the rest of the serving stack: FlashAttention pre-filling, GQA models, and weight/KV quantization, so the savings compose instead of conflicting.
목표 입력과 무관한 고정 cache 레이아웃으로 long-context KV 메모리와 지연시간을 구조적으로 줄이되 full attention 수준의 retrieval 정확도를 유지하는 것이 목표다. 마찬가지로 중요한 요구는 서빙 스택의 나머지와 호환되는 것 — FlashAttention pre-filling, GQA 모델, weight/KV quantization과 충돌하지 않고 절감이 결합(compose)되어야 한다.
Method Each KV head gets a gate α∈[0,1] that blends full attention with streaming (sink+recent) attention, and only these gates are trained — 2,000 steps with all model weights frozen — using an L2 distillation loss against the full-cache model's last hidden states (computed only on passkey tokens) plus an L1 sparsity penalty (λ=0.05). The training data is a synthetic passkey-retrieval set (ten 32-token passkeys embedded at random depths in long text), so a gate can shrink only if dropping that head's deep-context KV provably doesn't break retrieval — like turning down a dimmer on each head and leaving it dim only when the final answer stays intact. Gates are then thresholded (τ from a sparsity quantile) into a binary layout: at deployment retrieval heads keep full KV while streaming heads keep a constant cache (64 sink + 256 recent tokens on long-context benchmarks). Chunked pre-filling with FlashAttention drops the streaming heads' pre-fill cost from O(L²) to O(LK) time and O(L) to O(K) memory for chunk size K.
방법 KV head마다 full attention과 streaming(sink+recent) attention을 보간하는 gate α∈[0,1]를 두고, 모델 weight는 전부 얼린 채 gate만 2,000 step 학습한다 — full-cache 모델의 마지막 hidden state에 대한 L2 distillation loss(passkey token에서만 계산)에 L1 sparsity 페널티(λ=0.05)를 더해서. 학습 데이터는 synthetic passkey-retrieval set(32-token passkey 10개를 긴 텍스트의 임의 깊이에 삽입)이라, 어떤 head의 deep-context KV를 버려도 retrieval이 안 깨질 때만 gate가 줄어든다 — head마다 조광기(dimmer)를 서서히 내려 보고 최종 답이 멀쩡할 때만 어둡게 놔두는 식이다. 이후 gate를 threshold τ(sparsity quantile)로 이진화해, 배포 시 retrieval head는 full KV를, streaming head는 상수 cache(long-context 벤치마크 기준 sink 64 + recent 256 token)를 갖는 두 정적 레이아웃으로 돌린다. FlashAttention 기반 chunked pre-filling은 streaming head의 pre-fill 비용을 시간 O(L²)→O(LK), 메모리 O(L)→O(K)로 낮춘다(K=chunk 크기).
Key idea The key idea is a head-level functional dichotomy: only a minority of retrieval heads need the whole context, while the majority of streaming heads attend mostly to attention sinks and recent tokens — like a newsroom where a few archivists need keys to the full archive while everyone else works off today's desk pile. What makes the paper distinctive is how the split is measured: an output-level optimization that directly checks whether removing a head's deep-context KV changes the model's answers, rather than inferring importance from attention-score heuristics. Figure 1's ablation makes the dichotomy vivid — converting retrieval heads to streaming first crashes accuracy almost immediately, while converting streaming heads first keeps it near 100%.
핵심 아이디어 핵심은 head 수준의 기능적 이분법이다 — 소수의 retrieval head만 전체 context가 필요하고, 다수의 streaming head는 attention sink와 최근 token 위주로만 본다. 소수의 아키비스트만 전체 서고 열쇠가 필요하고 나머지 기자들은 오늘 책상 위 서류만으로 일하는 뉴스룸과 같다. 이 논문의 차별점은 그 분할을 재는 방식이다: attention-score 휴리스틱으로 중요도를 추정하는 대신, head의 deep-context KV를 제거했을 때 모델 출력이 실제로 변하는지를 직접 확인하는 출력 수준 최적화를 쓴다. Figure 1의 ablation이 이 이분법을 극명하게 보여준다 — retrieval head부터 streaming으로 바꾸면 정확도가 즉시 붕괴하지만, streaming head부터 바꾸면 거의 100%를 유지한다.
Validation Validated on Needle-in-a-Haystack up to 1,048K tokens and 21 LongBench tasks using Llama-2-7B (MHA), Llama-3-8B-Instruct-Gradient-1048k, Llama-3-70B, and Mistral-7B-v0.2 (GQA), plus MMLU/MBPP/MT-Bench checks showing near-lossless short-context ability (e.g., Llama-3-70B at a 50% budget scores MMLU 79.35 vs 79.38 for full attention). Efficiency is measured end to end — decoding latency and memory, chunked pre-filling latency, and a combined-with-quantization capacity test. The identification phase itself is cheap: 2,000 gate-training steps on 8×A100 GPUs, run once per model.
검증 최대 1,048K token의 Needle-in-a-Haystack과 21개 LongBench task에서 Llama-2-7B(MHA), Llama-3-8B-Instruct-Gradient-1048k, Llama-3-70B, Mistral-7B-v0.2(GQA)로 검증했고, MMLU/MBPP/MT-Bench로 short-context 능력이 거의 무손실임을 확인했다(예: Llama-3-70B, 50% budget에서 MMLU 79.35 vs full 79.38). 효율은 end-to-end로 잰다 — decoding 지연·메모리, chunked pre-filling 지연, quantization 결합 capacity 테스트까지. identification 단계 자체는 싸다: 모델당 한 번, 8×A100에서 gate 학습 2,000 step이면 끝난다.
Results Delivers up to 2.55x KV-memory and 2.18x decoding-latency reduction on MHA (1.67x/1.50x on GQA) and up to 1.73x (MHA) / 1.63x (GQA) faster pre-filling, while matching full attention on LongBench at a 25% (MHA) / 50% (GQA) KV budget. The compounding result is the headline demo: combined with QServe's 8-bit-weight/4-bit-KV quantization, Llama-3-8B serves a 3.3M-token context on a single A100-80G — a 6.4x capacity increase over FP16 full attention. Crucially these gains are structural: the memory footprint is fixed per model, not a function of the input.
결과 MHA에서 최대 2.55배 KV 메모리 절감과 2.18배 decoding 지연 감소(GQA는 1.67배/1.50배), pre-filling은 최대 1.73배(MHA)/1.63배(GQA) 가속을 달성하면서, KV budget 25%(MHA)/50%(GQA)로 LongBench에서 full attention과 대등하다. 결합 효과가 헤드라인 데모다: QServe의 8-bit weight/4-bit KV quantization과 결합하면 Llama-3-8B가 단일 A100-80G에서 3.3M token context를 서빙한다 — FP16 full attention 대비 6.4배 capacity. 이 이득이 구조적이라는 점이 중요하다: 메모리 footprint가 입력이 아니라 모델별로 고정된다.
Comparison Compared against H2O, TOVA, and StreamingLLM at the same KV cache budget, DuoAttention is the only method that holds NIAH accuracy across all context depths — the token-eviction baselines collapse precisely at deep-context positions where the passkey has already been evicted. Half of the comparison is also about feasibility: FastGen's quadratic profiling memory caps it at 24K-32K contexts in these experiments, while DuoAttention runs at million-token scale.
비교 H2O, TOVA, StreamingLLM과 같은 KV cache budget으로 비교하면 DuoAttention만 유일하게 NIAH 정확도를 전 depth에서 유지한다 — token-eviction 계열은 passkey가 이미 evict된 깊은 위치에서 정확히 붕괴한다. 비교의 절반은 실행 가능성 문제이기도 하다: FastGen은 profiling 메모리가 제곱으로 늘어 이 실험에서 24K~32K context가 상한이었지만, DuoAttention은 million-token 규모에서 돌아간다.
Significance It shifts KV compression from input-dependent token selection to a static head-level structure that a serving system can pre-allocate — the difference between rearranging desks every morning based on who shows up and drawing a floor plan once. The finding that most heads are streaming also gives mechanistic support to hybrid full+linear attention architectures: if a head never uses deep context, replacing it with a cheap sliding or linear layer is principled rather than hopeful. And because the method is orthogonal to quantization and GQA, it composes with the existing efficiency stack instead of competing with it.
의의 KV 압축을 입력 의존적 token 선택에서 서빙 시스템이 미리 할당할 수 있는 정적 head 구조로 옮겼다 — 매일 아침 출근자를 보고 책상을 재배치하는 것과 도면을 한 번 그려두는 것의 차이다. 대부분의 head가 streaming이라는 발견은 hybrid full+linear attention 아키텍처의 기계론적 근거가 된다: 어떤 head가 deep context를 쓰지 않는다면 그것을 값싼 sliding·linear layer로 대체하는 것은 희망사항이 아니라 원리에 기반한 설계다. 또한 quantization·GQA와 직교하므로 기존 효율화 스택과 경쟁하는 대신 결합된다.
Limitations The clearest cost visible in the paper's own numbers is that GQA models need a 50% retrieval-head ratio versus 25% for MHA, so the gains shrink exactly where modern architectures have already compressed the KV cache — plausibly because several query heads sharing one KV head couple their needs (mechanism not explained in the paper; reviewer judgment). The method also requires a per-model gate-training identification phase before deployment. And since calibration relies on synthetic passkey retrieval, whether the resulting head split is equally right for reasoning, code, and multi-hop use is not fully settled (reviewer judgment); the paper itself has no explicit limitations section.
한계 논문 수치에서 가장 뚜렷한 비용은 GQA 모델이 MHA의 25%와 달리 retrieval head 비율 50%를 요구한다는 것으로, 현대 아키텍처가 이미 KV를 압축해 둔 지점에서 정확히 이득이 줄어든다 — 여러 query head가 하나의 KV head를 공유하며 요구가 결합되기 때문으로 보인다(메커니즘은 논문 내 설명 없음, 리뷰어 판단). 또한 배포 전에 모델별 gate 학습 identification 단계가 필요하다. calibration이 synthetic passkey retrieval에 의존하므로 이 head 분할이 reasoning·code·multi-hop task에도 똑같이 옳은지는 완전히 검증되지 않았고(리뷰어 판단), 논문 자체에 명시적 limitations 섹션은 없다.
Future work Not stated in the paper. Natural extensions are budgeted non-binary head allocations instead of a hard retrieval/streaming split, calibration objectives that reflect task diversity beyond passkey retrieval, and co-designing GQA/MLA or hybrid architectures around the retrieval/streaming structure from the start (reviewer judgment). The head-identification recipe itself — cheap gate distillation against the full model — also looks reusable for other structural sparsity decisions (reviewer judgment).
향후 과제 논문 내 명시 없음. 자연스러운 다음 단계는 이진 분할 대신 budget 기반 연속 head 할당, passkey retrieval을 넘어 task 다양성을 반영한 calibration 목적함수, 그리고 retrieval/streaming 구조를 처음부터 반영한 GQA/MLA·hybrid 아키텍처 co-design이다(리뷰어 판단). full 모델에 대한 값싼 gate distillation이라는 head-identification 레시피 자체도 다른 구조적 sparsity 결정에 재사용될 여지가 있다(리뷰어 판단).
Released resources The official implementation is open-sourced at the MIT-HAN Lab GitHub repository (mit-han-lab/duo-attention). The paper was published at ICLR 2025, and the arXiv version (2410.10819) includes full per-task result tables for all 21 LongBench tasks in its appendix.
자원 공개 공식 구현이 MIT-HAN Lab GitHub 저장소(mit-han-lab/duo-attention)에 공개되어 있다. 논문은 ICLR 2025에 게재되었고, arXiv 버전(2410.10819) 부록에 21개 LongBench task별 전체 결과 표가 실려 있다.
The research thread
Where this paper sits 연구의 흐름
DuoAttention sits at the junction of two lines that had been running in parallel. The first is the KV-cache compression line: StreamingLLM (2023) — from the same first author, Guangxuan Xiao — showed that an attention-sink-plus-recent window keeps a model fluent indefinitely but blind to deep context, H2O (2023) and TOVA (2024) tried to cure that blindness by scoring tokens and evicting the unimportant ones, and FastGen (2024) profiled per-head attention habits to pick eviction policies. The second is an interpretability line: Wu et al.'s retrieval-heads work (2024) found that only a small set of attention heads mechanistically carries long-range copy-and-recall. DuoAttention is what happens when the efficiency line takes the interpretability result seriously as an engineering blueprint.
The shift is a change of unit and a change of instrument. Prior compression asked which tokens to drop — an input-dependent question re-answered at every step. DuoAttention asks which heads ever need deep context at all — a question with a static, per-model answer that a serving system can bake into its memory plan before the first request arrives. Just as important is how the answer is measured: instead of trusting attention scores as a proxy for importance, it trains a per-head gate against the full model's outputs, so a head is declared streaming only if deleting its deep-context KV verifiably leaves the answer intact. Importance becomes a matter of consequence, not attention mass — a quiet methodological correction aimed at the whole profiling tradition.
What this opens is the bridge from post-hoc compression to architecture design. If most heads never read deep context, then hybrid stacks that interleave a few full-attention layers with cheap linear or sliding-window ones — MiniMax-01's lightning-attention hybrid, Gated DeltaNet-style recurrent layers — stop being empirical bets and become applications of a measured dichotomy. The field is already moving that way: sparsity is migrating from inference-time patches into pretraining itself, with natively trainable sparse attention (DeepSeek's NSA, Moonshot's MoBA) making the sparsity pattern something the model learns rather than something a calibration pass discovers afterwards.
The tension that remains is that the win shrinks exactly where modern architectures already live: GQA models need roughly twice the retrieval-head ratio of MHA models, suggesting that KV-head sharing entangles the very roles the method wants to separate. And the binary retrieval/streaming split, calibrated on synthetic passkey retrieval, is likely the coarsest version of a more continuous, more task-dependent truth. The natural next round — visible in the paper's own loose ends — is co-designing the head layout with the architecture instead of discovering it after the fact, and calibrating against tasks that stress reasoning rather than pure recall.
DuoAttention은 나란히 달리던 두 연구 계보의 교차점에 서 있다. 첫째는 KV-cache 압축 계보다: 같은 1저자 Guangxuan Xiao의 StreamingLLM(2023)이 attention sink + recent window만으로 모델이 무한히 유창하되 deep context에는 눈멀게 됨을 보였고, H2O(2023)와 TOVA(2024)는 token에 점수를 매겨 안 중요한 것을 evict하는 방식으로 그 실명을 고치려 했으며, FastGen(2024)은 head별 attention 습관을 profiling해 eviction 정책을 골랐다. 둘째는 interpretability 계보다: Wu et al.(2024)의 retrieval-heads 연구는 소수의 attention head만이 long-range copy-and-recall을 기계론적으로 수행함을 밝혔다. DuoAttention은 효율화 계보가 이 interpretability 결과를 엔지니어링 설계도로 진지하게 받아들일 때 나오는 논문이다.
전환의 핵심은 단위의 변경과 측정 도구의 변경이다. 기존 압축은 어떤 token을 버릴지 물었다 — 매 step 다시 답해야 하는 입력 의존적 질문이다. DuoAttention은 어떤 head가 애초에 deep context를 필요로 하는지를 묻는다 — 모델마다 한 번 정해지는 정적인 답이라, 서빙 시스템이 첫 요청이 오기 전에 메모리 플랜에 박아 넣을 수 있다. 답을 재는 방식도 그만큼 중요하다: attention score를 중요도의 proxy로 믿는 대신 head별 gate를 full 모델의 출력에 대해 학습시켜, 그 head의 deep-context KV를 지워도 답이 멀쩡함이 확인될 때만 streaming으로 판정한다. 중요도가 attention 질량이 아니라 결과(consequence)의 문제가 된다 — profiling 전통 전체를 겨눈 조용한 방법론적 교정이다.
이 논문이 여는 것은 사후(post-hoc) 압축에서 아키텍처 설계로 가는 다리다. 대부분의 head가 deep context를 읽지 않는다면, 소수의 full-attention layer 사이에 값싼 linear·sliding-window layer를 끼워 넣는 hybrid 스택 — MiniMax-01의 lightning-attention hybrid, Gated DeltaNet류 recurrent layer — 은 경험적 도박이 아니라 측정된 이분법의 응용이 된다. 필드도 이미 그 방향으로 움직이고 있다: sparsity가 inference-time 패치에서 pretraining 자체로 이주하는 중이고, natively trainable sparse attention(DeepSeek의 NSA, Moonshot의 MoBA)은 sparsity 패턴을 calibration이 사후에 발견하는 것이 아니라 모델이 직접 학습하는 것으로 만든다.
남은 긴장은 이득이 현대 아키텍처가 이미 서 있는 지점에서 정확히 줄어든다는 것이다: GQA 모델은 MHA의 대략 두 배에 달하는 retrieval-head 비율을 요구하며, 이는 KV-head 공유가 이 방법이 분리하려는 역할들을 얽어버림을 시사한다. 그리고 합성 passkey retrieval로 calibration한 이진 retrieval/streaming 분할은, 더 연속적이고 더 task 의존적인 진실의 가장 거친 버전일 가능성이 크다. 논문의 느슨한 끝단에서 이미 보이는 자연스러운 다음 라운드는 head 레이아웃을 사후에 발견하는 대신 아키텍처와 함께 co-design하는 것, 그리고 순수 recall이 아니라 reasoning을 압박하는 task로 calibration하는 것이다.
Sparks — where one could take this 아이디어의 씨앗
-
One could rerun the gate identification with a calibration set spanning reasoning, code, and multi-hop QA instead of synthetic passkeys, and directly measure how much the retrieval/streaming head split shifts across task families. 합성 passkey 대신 reasoning·code·multi-hop QA를 아우르는 calibration set으로 gate identification을 다시 돌려, retrieval/streaming head 분할이 task family에 따라 얼마나 이동하는지 직접 측정해볼 수 있다.
-
One could relax the binary threshold into a budgeted allocation — giving each head a cache length proportional to its learned gate value — and trace whether the accuracy-memory Pareto curve beats the hard split. 이진 threshold를 budget 기반 할당으로 완화해 — head마다 학습된 gate 값에 비례한 cache 길이를 주고 — accuracy-memory Pareto curve가 hard split을 이기는지 추적해볼 수 있다.
-
One could probe why GQA halves the benefit by measuring retrieval behavior separately for each query head inside a shared KV group, testing whether role entanglement — not compression itself — forces the higher retrieval-head ratio. GQA에서 이득이 절반으로 주는 이유를 파볼 수 있다: 공유 KV group 안의 query head별로 retrieval 행동을 따로 측정해, 압축 자체가 아니라 역할 entanglement가 높은 retrieval-head 비율을 강제하는지 검증하는 것이다.
Seeds grounded in the paper's own stated limitations and open questions. 논문이 스스로 밝힌 limitations와 open questions에 뿌리를 둔 씨앗입니다.
Notes
Notes
DuoAttention reframed KV-cache compression: instead of asking which tokens to keep, it asks which heads need long context at all. Its optimization-based identification — gates distilled against the full-cache model’s outputs — is a more reliable measurement instrument than attention-score inspection, and the binary retrieval/streaming split gives serving systems a predictable, input-independent memory footprint. The paper is also frequently cited as mechanistic evidence for hybrid designs: if most heads are streaming anyway, replacing them with cheap sliding or linear layers (as in MiniMax-01-style architectures) has principled headroom.
Token 단위 eviction이 아니라 head 단위 구조 분해라는 점에서, KV 압축 연구와 hybrid attention 설계를 잇는 다리 같은 논문이다.
Linked reviews
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs
A small set of "heavy-hitter" tokens dominates attention mass, so greedily evicting low-accumulated-attention entries keeps only ~20% of the KV cache while matching full-cache accuracy and raising serving throughput up to 29x.
quantizationKIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache
Asymmetric KV quantization derived from element-distribution analysis — per-channel keys, per-token values — pushes the KV cache to tuning-free 2 bits, cutting peak memory 2.6x and lifting serving throughput up to 3.47x.
hybrid-architectureMiniMax-01: Scaling Foundation Models with Lightning Attention
One softmax-attention block per seven lightning (linear) attention blocks, scaled to a 456B-parameter MoE, matches GPT-4o-class quality while training at 1M-token context and extrapolating to 4M.
linear-attentionGated Delta Networks: Improving Mamba2 with Delta Rule
Unifies Mamba2-style gated decay with the delta-rule targeted memory update into a single hardware-efficient linear-attention layer that beats both parents and anchors strong hybrid stacks with sliding-window attention.
attentionAVQ-Attention: Adaptive Vector-Quantized Attention
Vector-quantized attention cuts quadratic attention cost by clustering keys into a fixed codebook, but spends the same resolution everywhere regardless of where attention mass actually concentrates; AVQ-Attention adaptively refines codebook resolution toward high-attention regions using a hierarchical parent-child codebook, implemented as custom Triton kernels fused directly into Flash-Attention-style tiling, reaching up to 127x over Flash Attention at 64k tokens.
sparse-attentionCOBS: Cumulant Order Block Sparse Attention
COBS shows that every existing cacheable block-sparse-attention selector (including DeepSeek's NSA) scores blocks using only the first-order (mean-key) term of an exact cumulant expansion of attention mass, and that adding the discarded second-order term, a compressed within-block key covariance, closes 86 percent of the accuracy gap to dense attention at 32k context while using 15x less KV-cache read traffic than dense and only 1.21x more than the NSA baseline.
sparse-attentionLearning What Matters: Supervising Global Context Pruning with Causal Evidence Sets
Sparse-attention routers are usually trained to imitate a dense teacher's attention pattern; this paper builds tasks with exactly known causal evidence and shows attention and causal dependence disagree, with a router trained on masking-derived causal labels reaching 0.99 routed accuracy on a 2-hop task versus 0.41 for an attention-distilled router, plus a pretrained-model result where larger models get answers right well before their attention becomes trustworthy.
long-contextJet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
A tuning-free method extends an LLM's context window beyond its pretrained length by combining a local RoPE-faithful attention window with a long-range window whose rescaling factor adapts dynamically to the current sequence length, fused into a single GPU kernel.
kv-cacheKVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Pre-RoPE per-channel key and per-token value quantization with sensitivity-weighted non-uniform datatypes plus ~1% fp16 outliers pushes the KV cache to 3 bits at under 0.1 perplexity cost, reaching million-token contexts on a single A100.
kv-cacheLiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time
A block-causal video diffusion model with a retrieval-augmented sink-attention KV-cache policy sustains real-time, constant-memory streaming human animation for minutes, not seconds.
kv-cacheWhat to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents
A survey reframes KV-cache eviction, prompt compression, bounded architectural state, and agent memory consolidation as one rate-distortion decision on a shared bytes-per-token-of-history axis, showing all four communities' scoring signals fail identically (deciding what to keep before the query is known) and proposing a COMPACT-Bench benchmark.
ssmRaven: High-Recall Sequence Modeling with Sparse Memory Routing
Raven unifies SSMs and sliding-window attention under one framework (Routing Slot Memories) by giving a fixed set of memory slots a learned, input-dependent router that selects a Top-K subset to update each step and decays only those slots, matching or beating linear-time baselines on recall benchmarks and extrapolating to 16x its training context length.
video-diffusionPartition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models (SparsePR)
SparsePR makes training-free block-sparse attention for video diffusion transformers actually executable, by routing queries with key/value response geometry and correcting the sparse output with a few exact probe rows -- beating four prior sparse-attention methods on fidelity at similar or lower compute across four video/world models.
kv-cacheTriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Observes that pre-RoPE Q/K vectors concentrate around fixed centers, so attention can be predicted as a trigonometric series in token distance — a query-free KV eviction score that matches full-attention AIME25 accuracy at 2.5x throughput or 10.7x less KV memory.
quantizationTurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
A random rotation forces every coordinate into a known Beta distribution, so one precomputed Lloyd-Max codebook quantizes any vector stream online with zero calibration — plus a 1-bit QJL residual stage for unbiased inner products — provably within ≈2.7× of the information-theoretic distortion floor, yielding quality-neutral 3.5-bit KV caches and vector-search indexing in near-zero time.