Paper Review
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs
A small set of "heavy-hitter" tokens dominates attention mass, so greedily evicting low-accumulated-attention entries keeps only ~20% of the KV cache while matching full-cache accuracy and raising serving throughput up to 29x.
Attention mass의 대부분을 소수 "heavy-hitter" 토큰이 차지한다는 관측을 바탕으로, 누적 attention score가 낮은 엔트리를 greedy하게 evict해 KV cache의 약 20%만 유지하면서 full-cache 정확도를 유지하고 serving throughput을 최대 29배 높인 eviction 정책.
Background In LLM generative inference the KV cache grows linearly with sequence length and batch size, and for long-content generation (dialogue, story writing) it, rather than the model weights, becomes the deployment memory bottleneck. The paper's example: a 30B model with batch size 128 and sequence length 1024 already produces 180GB of KV cache. Capping the cache size, as in classical caching systems, is the natural response — but which entries to keep was an open question.
배경 LLM generative inference에서 KV cache는 sequence 길이와 batch size에 비례해 GPU 메모리를 선형으로 잠식하며, long-content 생성(dialogue, story writing)에서는 model weight보다 cache가 배포 비용의 병목이 된다. 논문의 예시로 30B 모델에 batch 128, sequence 1024만 되어도 KV cache가 180GB에 달한다. 고전 캐싱 시스템처럼 cache 크기에 상한을 두는 것이 자연스러운 대응이지만, 어떤 entry를 남겨야 하는지는 열려 있었다.
Problem Keeping the full KV cache ties batch size and context length to GPU memory and caps throughput, and there was no principled criterion for which tokens' K/V entries can be dropped without hurting generation quality. An ideal cache must satisfy three things at once — a small memory footprint, a low miss rate, and a low-cost eviction policy — yet in principle every decoding step may attend to any past K/V, and future attention is unknown at eviction time. It is like a stenographer who must keep every utterance of a meeting piled on the desk: the desk (GPU memory) fills up before the meeting (generation) ends.
문제 Full KV cache를 유지하면 batch size와 context 길이가 메모리에 묶여 throughput이 제한되는데, 어떤 토큰의 K/V를 버려도 생성 품질이 유지되는지에 대한 원칙적 eviction 기준이 없었다. 이상적인 cache는 작은 memory footprint, 낮은 miss rate, 저비용 eviction policy 세 가지를 동시에 만족해야 하지만, 원칙적으로 매 decoding step이 모든 과거 K/V를 참조할 수 있고 미래 토큰의 attention은 eviction 시점에 알 수 없다는 것이 난점이다. 회의의 모든 발언 기록을 책상 위에 쌓아둬야 하는 속기사처럼, 회의(생성)가 끝나기 전에 책상(GPU 메모리)이 먼저 차버리는 상황이다.
Prior limitations Fixed sparsity patterns (Sparse Transformer strided/fixed) and recency-only "local" caches also discard tokens that actually receive attention mass, so accuracy collapses at low budgets. Concretely, at a 20% budget the local policy scores 57.94 on PiQA versus 80.09 with the full cache. Learned compression such as gisting tokens exists, but its expensive eviction policies are hard to deploy during generation.
기존 한계 Sparse Transformer류의 고정 패턴(strided/fixed)이나 최근 토큰만 남기는 "local" 정책은 attention mass를 실제로 받는 토큰을 함께 버려서 낮은 budget에서 정확도가 급락한다. 실제로 20% budget에서 local 정책은 PiQA 57.94로 full cache(80.09) 대비 붕괴한다. Gisting token처럼 학습으로 cache를 압축하는 시도도 있었지만, eviction 비용이 커서 generation 중 배포가 어렵다.
Goal Build an online, training-free KV eviction policy that holds a fixed cache budget (e.g., 20%) while preserving full-cache accuracy and raising serving throughput. Since future attention is invisible and a decision must be made at every decoding step, the policy has to run on past-only information while keeping the miss rate low and the per-step cost cheap.
목표 재학습·fine-tuning 없이, 고정된 cache budget(예: 20%)만 유지하면서 full-cache 정확도를 보존하고 serving throughput을 높이는 online KV eviction 정책을 만드는 것. 미래 attention을 볼 수 없는 상태에서 매 decoding step마다 즉시 결정해야 하므로, 정책은 과거 정보만으로 동작하면서도 miss rate가 낮고 step당 비용이 싸야 한다.
Method At each decoding step it accumulates every past token's attention score, and once the budget k is exceeded it greedily evicts the entry with the lowest accumulated score. The cache budget is split evenly between heavy hitters and the most recent tokens, so the retained set = recent + heavy hitters. Eviction is formalized as dynamic submodular maximization, with a theorem showing the greedy policy attains a (1−1/e)-style near-optimality guarantee under mild assumptions. Structurally it works like an OS page cache mixing LFU (frequency) with LRU (recency) and evicting the coldest page — except "access frequency" here is accumulated attention mass.
방법 각 decoding step에서 past-token attention score를 토큰별로 누적하고, budget k 초과 시 누적 점수가 가장 낮은 entry를 greedy하게 evict한다. cache budget은 heavy hitter와 최근 토큰에 균등 배분되어 retained set = recent + heavy hitters로 유지된다. 이를 dynamic submodular maximization으로 정식화해, mild assumption 하에 greedy 정책이 (1−1/e)형 near-optimality 보장을 갖는다는 정리를 증명했다. OS page cache가 LFU(빈도)와 LRU(최근성)를 섞어 가장 차가운 페이지를 내보내는 것과 같은 구조인데, 여기서는 "접근 빈도"가 누적 attention mass다.
Key idea Even densely trained LLMs have attention matrices that are over 95% sparse at inference time, and accumulated attention scores follow a power-law distribution — a small set of heavy-hitter tokens carries most of the attention value. These heavy hitters correlate strongly with word co-occurrence frequency in the text, and masking them collapses accuracy, confirming their importance. Crucially, past-only local statistics are as effective as statistics that also see future attention, which is what makes online identification possible. Like library checkouts concentrating on a few popular books, a small front shelf (the 20% cache) can serve nearly every request (attention).
핵심 아이디어 Densely trained LLM도 attention matrix는 inference 시 95% 이상 sparse하고 누적 attention score는 power-law 분포를 따르므로, 소수 heavy-hitter 토큰이 attention value의 대부분을 차지한다. 이 heavy hitter들은 텍스트 내 단어 co-occurrence 빈도와 강하게 상관하며, 이들을 masking하면 정확도가 급락해 중요성이 확인된다. 결정적으로 과거 토큰만 합산한 local 통계가 미래 attention까지 아는 통계와 동등하게 효과적이어서 online 식별이 가능하다. 도서관 대출이 소수 인기 도서에 몰리는 것처럼, 작은 front shelf(20% cache)만으로 거의 모든 요청(attention)을 처리할 수 있다는 관측이다.
Validation Evaluated OPT (6.7B up to 175B), LLaMA-7B/13B, and GPT-NeoX-20B on eight lm-eval-harness/HELM tasks (COPA, MathQA, OpenBookQA, PiQA, RTE, Winogrande, XSUM, CNN/DailyMail) at cache budgets from 4% to 100%. Accuracy runs on an A100-80GB; throughput follows the FlexGen paper's settings on a T4-16GB, with latency measured on the A100 at sequence lengths of 4K-10K. XSUM/CNN-DailyMail use 1000 test samples at 4/10/20/60% budgets, and the appendix adds generation benchmarks such as AlpacaEval and MT-Bench.
검증 OPT(6.7B~175B), LLaMA-7B/13B, GPT-NeoX-20B를 lm-eval-harness/HELM의 8개 task(COPA, MathQA, OpenBookQA, PiQA, RTE, Winogrande, XSUM, CNN/DailyMail)에서 cache budget 4~100%로 평가했다. 정확도는 A100-80GB에서, throughput은 FlexGen 논문 세팅을 따라 T4-16GB에서 측정했고, A100에서는 sequence 4K~10K까지 latency를 쟀다. XSUM/CNN-DailyMail은 1000개 test sample에서 budget 4/10/20/60%로 평가했으며, appendix에서 AlpacaEval·MT-Bench 같은 generation benchmark도 다룬다.
Results At a 20% budget it matches full-cache accuracy (PiQA 79.22 vs 80.09) and improves throughput by up to 29x over DeepSpeed Zero-Inference and HF Accelerate and 3x over FlexGen, cutting latency by up to 1.9x at the same batch size. The freed memory allows larger batches, adding another 2.3x throughput on OPT-6.7B, and overall the paper reports a 5x KV cache memory reduction without accuracy loss across 6.7B-175B models. As bonuses, streaming up to four million tokens achieves lower perplexity than StreamingLLM on PG-19, and generated-text diversity increases.
결과 20% budget에서 full-cache 정확도와 대등(PiQA 79.22 vs 80.09)하고, throughput은 DeepSpeed Zero-Inference·HF Accelerate 대비 최대 29배, FlexGen 대비 3배, 같은 batch size에서 latency 최대 1.9배 감소를 달성했다. 절약된 메모리로 batch를 키우면 OPT-6.7B에서 추가로 2.3배 throughput이 나오며, 전체적으로 6.7B~175B에 걸쳐 정확도 손실 없는 5배 KV cache 메모리 절감을 보고한다. 보너스로 4백만 토큰까지의 streaming에서 StreamingLLM보다 낮은 perplexity(PG-19)를 보였고, 생성 텍스트의 diversity 증가도 관찰됐다.
Comparison Main baselines are the full KV cache, the recency-only "local" policy, and Sparse Transformer (strided/fixed), with appendix comparisons against SpAtten and StreamingLLM. Notably, simply adding heavy hitters on top of existing sparsification recovers performance — the local policy's COPA jumps 48.00 → 84.00 on OPT-30B at a 20% budget. The difference from SpAtten is granularity: SpAtten accumulates scores across heads and layers, while H2O evicts independently per head and per layer.
비교 Full KV cache, recency-only "local", Sparse Transformer(strided/fixed)를 주 baseline으로 비교하고 appendix에서 SpAtten·StreamingLLM과도 대조했다. 흥미롭게도 기존 sparsification 위에 heavy hitter를 얹기만 해도 성능이 복구된다 — OPT-30B 20% budget에서 local 정책의 COPA가 48.00→84.00으로 뛴다. SpAtten과의 차이는 granularity로, SpAtten은 head/layer에 걸쳐 score를 합산하지만 H2O는 head/layer별로 독립적으로 evict한다.
Significance It established the heavy-hitter / cache-budget / eviction-policy vocabulary and made training-free KV eviction a standalone axis of KV compression (reviewer judgment). Reducing the number of entries is orthogonal to reducing bits per entry via quantization (KIVI/KVQuant), and the paper's appendix directly shows throughput improves further when combined with quantization. In warehouse terms, eviction removes crates while quantization shrink-wraps each one — two levers that can be pulled independently.
의의 Heavy hitter·cache budget·eviction policy라는 용어 체계를 정립해 training-free KV eviction을 KV 압축의 독립 축으로 확립했다 (리뷰어 판단). entry 수를 줄이는 이 방향은 entry당 bit를 줄이는 quantization(KIVI/KVQuant)과 직교하며, 논문 appendix에서 quantization과의 결합으로 throughput이 추가 개선됨을 직접 보였다. 창고 정리에 비유하면 eviction은 상자 수를 줄이는 것이고 quantization은 상자 하나하나를 압축 포장하는 것이라, 두 lever를 독립적으로 당길 수 있다.
Limitations The theoretical guarantee holds only under the assumed submodularity, and the paper's own limitations discussion (Appendix B.2) is brief. The limitation the authors do name is that even with H2O's throughput gains, generative inference stays heavy due to the immense parameter count, much of it in MLP blocks. Separately, the backward-looking accumulated-attention statistic can evict tokens that only become important later (long-context retrieval), a failure mode documented by follow-up work (reviewer judgment) — like discarding a library book that has never been checked out right before its first request.
한계 이론 보장은 submodularity 가정 하에서만 성립하고, 논문 자체의 한계 논의(Appendix B.2)는 짧다. 저자들이 직접 언급한 한계는 H2O가 throughput을 높여도 방대한 parameter 수(상당 부분이 MLP block) 때문에 generative inference 자체는 여전히 무겁다는 점이다. 또한 누적 attention이 backward-looking 통계라서 나중에야 중요해지는 토큰(long-context retrieval)을 미리 evict하는 실패 모드는 후속 연구들이 문서화했다 (리뷰어 판단) — 한 번도 대출된 적 없는 책을 첫 요청 직전에 폐기하는 셈이다.
Future work The paper points to exploiting heavy hitters in MLP blocks (where they also emerge) as future work, and states the dynamic-submodular formulation could help guide follow-up research. Composition with quantization is already demonstrated in the appendix. Head/layer-adaptive budgets and retrieval-safe eviction are the natural next steps (reviewer judgment).
향후 과제 논문은 MLP block에서도 heavy hitter가 관찰된다는 점을 들어 MLP 쪽 활용을 후속 방향으로 제시하고, dynamic submodular 정식화가 후속 연구의 지침이 될 수 있다고 명시한다. quantization과의 결합은 appendix에서 이미 시연했다. head/layer별 이질성을 반영한 adaptive budget과 retrieval-safe eviction이 자연스러운 다음 단계다 (리뷰어 판단).
Released resources Official code is released on GitHub (FMInference/H2O) including the serving system implemented on top of FlexGen; no checkpoints are needed since the method is training-free. Accuracy experiments build on lm-eval-harness/HELM, so they are reproducible with standard harnesses.
자원 공개 공식 코드가 GitHub(FMInference/H2O)에 공개되어 있고 FlexGen 위에 구현된 serving 시스템을 포함하며, 별도 checkpoint는 없다(training-free). 정확도 실험은 lm-eval-harness/HELM 기반이라 표준 harness로 재현 가능하다.
The research thread
Where this paper sits 연구의 흐름
H2O stands where two older research lines intersect. The first is the attention-sparsity line: Sparse Transformer (Child et al., 2019) baked strided and fixed sparse patterns into the architecture at training time, and SpAtten (HPCA 2021) carried accumulated attention scores into cascade token pruning on the hardware side. The second is the LLM-serving line: FlexGen — the system H2O is literally built on — had just reframed generative inference as a memory-and-scheduling systems problem, with the KV cache as the fastest-growing item on the bill. H2O is the meeting point of these lineages: it takes sparsity, until then an architecture-design question answered before training, and re-asks it as a cache-management question answered online, on a frozen model.
The shift is easiest to see in the vocabulary the paper installs. The moment you call the retained set a cache budget and removal an eviction policy, the KV cache stops being a model internal and becomes a classical caching system — attention scores turn into access statistics, LFU and LRU intuitions become admissible, and heavy hitters, a term lifted from the streaming-algorithms literature, names the tokens that dominate the access distribution. Note what the core claim is not: that attention is sparse was already known. The real claim is that past-only local statistics identify the important tokens as well as oracle statistics that see the future — and that single empirical fact is what converts an offline observation into an online greedy algorithm, complete with a submodular near-optimality guarantee.
What it opened is by now a subfield. Concurrent work like Scissorhands reached a similar "importance persists" conclusion, and a wave of successors refined the policy H2O deliberately left simple — per-layer and per-head budget allocation, observation-window scoring in SnapKV, pyramid-shaped budgets across depth. The unresolved tension, though, is built into the founding statistic: accumulated attention is backward-looking, so a token whose importance arrives late — the needle in a long-context haystack — is evicted before it is ever requested once. Recency plus frequency is a good prior for language, but it is a prior, not a guarantee.
That tension sets where the line is heading. DuoAttention answers structurally rather than statistically — separating retrieval heads that must keep the full cache from streaming heads that never needed it, moving the decision from tokens to heads. Meanwhile the quantization branch (KIVI, KVQuant) shrinks bits per entry rather than the number of entries — an orthogonal lever the H2O appendix already showed composes. Expect the field to keep pushing from accumulated importance toward predicted importance, and from fixed budgets toward accuracy-aware adaptive ones. The eviction question H2O posed in caching vocabulary remains open in its hardest, retrieval-heavy regime.
H2O는 두 오래된 연구 계보가 만나는 지점에 서 있다. 하나는 attention sparsity 계보다 — Sparse Transformer(Child et al., 2019)는 strided/fixed sparse 패턴을 학습 시점에 아키텍처에 새겨 넣었고, SpAtten(HPCA 2021)은 누적 attention score를 hardware 쪽 cascade token pruning으로 가져갔다. 다른 하나는 LLM serving 계보다 — H2O가 그 위에 직접 구현된 FlexGen은 generative inference를 메모리와 스케줄링의 systems 문제로 재정의했고, 그 청구서에서 가장 빨리 불어나는 항목이 KV cache였다. H2O는 이 계보들의 합류점이다: 학습 전에 결정되는 아키텍처 설계 질문이던 sparsity를, frozen 모델 위에서 online으로 답해야 하는 cache 관리 질문으로 다시 묻는다.
전환은 논문이 심어놓은 어휘에서 가장 잘 보인다. 유지되는 집합을 cache budget, 제거를 eviction policy라고 부르는 순간 KV cache는 모델 내부 구조이길 멈추고 고전적 캐싱 시스템이 된다 — attention score는 access 통계가 되고, LFU/LRU 직관이 그대로 들어오며, streaming algorithms 문헌에서 빌려온 heavy hitter라는 이름이 access 분포를 지배하는 토큰들을 가리킨다. 핵심 주장이 무엇이 아닌지도 봐야 한다: attention이 sparse하다는 것은 이미 알려져 있었다. 진짜 주장은 과거만 본 local 통계가 미래까지 보는 oracle 통계만큼 중요한 토큰을 잘 식별한다는 것이고, 이 하나의 경험적 사실이 offline 관측을 submodular near-optimality 보장까지 갖춘 online greedy 알고리즘으로 바꿔놓는다.
이 논문이 연 것은 이제 하나의 subfield다. 동시대의 Scissorhands가 "중요도는 지속된다"는 비슷한 결론에 도달했고, H2O가 일부러 단순하게 남겨둔 정책을 후속 연구들이 다듬었다 — per-layer/per-head budget 배분, SnapKV의 observation-window scoring, 깊이에 따라 피라미드형으로 줄어드는 budget. 하지만 풀리지 않은 긴장은 창립 통계 자체에 내장돼 있다: 누적 attention은 backward-looking이라, 중요도가 뒤늦게 도착하는 토큰 — long-context 건초더미 속 바늘 — 은 한 번 요청되기도 전에 evict된다. recency + frequency는 언어에 좋은 prior지만, 어디까지나 prior지 보장이 아니다.
이 긴장이 이 계보가 향하는 방향을 정한다. DuoAttention의 답은 통계적이 아니라 구조적이다 — full cache를 유지해야 하는 retrieval head와 애초에 필요 없던 streaming head를 구분해, 결정을 토큰 단위에서 head 단위로 옮긴다. 한편 KIVI·KVQuant 같은 quantization 계열은 entry 수가 아니라 entry당 bit를 줄인다 — H2O appendix가 이미 결합 가능함을 보인 직교 lever다. 앞으로 field는 누적된 중요도에서 예측된 중요도로, 고정 budget에서 정확도를 인지하는 adaptive budget으로 계속 밀고 갈 것이다. H2O가 캐싱의 어휘로 제기한 eviction 질문은 가장 어려운 retrieval-heavy regime에서 여전히 열려 있다.
Sparks — where one could take this 아이디어의 씨앗
-
Following the pointer in the paper's own future-work section, one could try extending the heavy-hitter analysis from attention to MLP blocks, testing whether accumulated activation mass predicts which FFN neurons can be skipped online during generation. 논문의 future-work가 직접 가리키는 대로, heavy-hitter 분석을 attention에서 MLP block으로 확장해 누적 activation mass가 generation 중 online으로 skip해도 되는 FFN neuron을 예측하는지 시험해볼 수 있다.
-
One could stress-test eviction policies on long-context retrieval benchmarks where a token's importance emerges only late, quantifying how much of the remaining accuracy gap traces to prematurely evicted future-relevant entries. 토큰 중요도가 뒤늦게 드러나는 long-context retrieval benchmark에서 eviction policy들을 stress-test해, 남은 정확도 격차 중 얼마가 미리 evict된 future-relevant entry에서 오는지 정량화해볼 수 있다.
-
Taking up the dynamic-submodular formulation the authors offer as a guide, one could try characterizing which eviction policies — recency, frequency, hybrids, per-head budgets — admit near-optimality guarantees and exactly where the assumptions break. 저자들이 후속 연구의 지침으로 제시한 dynamic submodular 정식화를 받아, recency·frequency·hybrid·per-head budget 등 어떤 eviction policy가 near-optimality 보장을 갖고 정확히 어디서 가정이 깨지는지 특성화해볼 수 있다.
Seeds grounded in the paper's own stated limitations and open questions. 논문이 스스로 밝힌 limitations와 open questions에 뿌리를 둔 씨앗입니다.
Notes
Notes
H2O is one of the founding papers of KV-cache eviction: it turned the empirical observation of power-law attention skew into an online policy (recent + heavy hitters) with a submodular-greedy justification, and defined the vocabulary the whole subfield still uses. It is the sparsity-side complement to quantization work like KIVI and KVQuant — one drops cache entries, the others shrink them — and its known weakness (backward-looking scores evicting future-relevant tokens) is exactly what head-level approaches like DuoAttention later addressed.
H2O는 KV eviction 계열의 출발점으로, 이후 등장한 거의 모든 budget-based cache 압축 기법이 이 논문의 heavy-hitter 프레임 위에서 정의된다.
Linked reviews
KIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache
Asymmetric KV quantization derived from element-distribution analysis — per-channel keys, per-token values — pushes the KV cache to tuning-free 2 bits, cutting peak memory 2.6x and lifting serving throughput up to 3.47x.
kv-cacheKVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Pre-RoPE per-channel key and per-token value quantization with sensitivity-weighted non-uniform datatypes plus ~1% fp16 outliers pushes the KV cache to 3 bits at under 0.1 perplexity cost, reaching million-token contexts on a single A100.
kv-cacheDuoAttention: Efficient Long-Context Inference with Retrieval and Streaming Heads
Learns which attention heads actually retrieve from long context and gives only those a full KV cache, cutting long-context memory up to 2.55x with retrieval accuracy intact.
sparse-attentionCOBS: Cumulant Order Block Sparse Attention
COBS shows that every existing cacheable block-sparse-attention selector (including DeepSeek's NSA) scores blocks using only the first-order (mean-key) term of an exact cumulant expansion of attention mass, and that adding the discarded second-order term, a compressed within-block key covariance, closes 86 percent of the accuracy gap to dense attention at 32k context while using 15x less KV-cache read traffic than dense and only 1.21x more than the NSA baseline.
sparse-attentionLearning What Matters: Supervising Global Context Pruning with Causal Evidence Sets
Sparse-attention routers are usually trained to imitate a dense teacher's attention pattern; this paper builds tasks with exactly known causal evidence and shows attention and causal dependence disagree, with a router trained on masking-derived causal labels reaching 0.99 routed accuracy on a 2-hop task versus 0.41 for an attention-distilled router, plus a pretrained-model result where larger models get answers right well before their attention becomes trustworthy.
kv-cacheA JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation
Treating the KV cache as a single three-axis tensor rather than a stack of matrices, JoLT compresses only the token and feature axes with a partial Tucker decomposition, restores the discarded energy with a low-bit rotated residual, and lets one shared optimizer decide how to split the byte budget between rank and residual bits and between keys and values -- reaching a near-lossless 2-3x compression zone on both a grouped-query and a multi-head attention model.
kv-cacheError Certificates for KV-Cache Eviction via Randomized Design
This paper proves that deterministic top-k KV-cache eviction cannot certify its own damage -- evicted values can be altered so everything the serving system retains is unchanged while the true attention-output error grows unboundedly -- and shows that a cheap randomized retention scheme (Poisson sampling plus one logit correction) restores enough statistical structure to build a real per-step error certificate, though pre-registered testing finds that certificate is worth its 2x overhead only for attributing and scheduling recovery from failures, not predicting them.
kv-cacheWhat to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents
A survey reframes KV-cache eviction, prompt compression, bounded architectural state, and agent memory consolidation as one rate-distortion decision on a shared bytes-per-token-of-history axis, showing all four communities' scoring signals fail identically (deciding what to keep before the query is known) and proposing a COMPACT-Bench benchmark.
ssmRaven: High-Recall Sequence Modeling with Sparse Memory Routing
Raven unifies SSMs and sliding-window attention under one framework (Routing Slot Memories) by giving a fixed set of memory slots a learned, input-dependent router that selects a Top-K subset to update each step and decays only those slots, matching or beating linear-time baselines on recall benchmarks and extrapolating to 16x its training context length.
kv-cacheTriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Observes that pre-RoPE Q/K vectors concentrate around fixed centers, so attention can be predicted as a trigonometric series in token distance — a query-free KV eviction score that matches full-attention AIME25 accuracy at 2.5x throughput or 10.7x less KV memory.
quantizationTurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
A random rotation forces every coordinate into a known Beta distribution, so one precomputed Lloyd-Max codebook quantizes any vector stream online with zero calibration — plus a 1-bit QJL residual stage for unbiased inner products — provably within ≈2.7× of the information-theoretic distortion floor, yielding quality-neutral 3.5-bit KV caches and vector-search indexing in near-zero time.