Paper Review
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Pre-RoPE per-channel key and per-token value quantization with sensitivity-weighted non-uniform datatypes plus ~1% fp16 outliers pushes the KV cache to 3 bits at under 0.1 perplexity cost, reaching million-token contexts on a single A100.
Pre-RoPE per-channel key / per-token value 양자화에 sensitivity 가중 non-uniform datatype과 ~1% fp16 outlier 분리를 더해 KV cache를 3-bit(<0.1 PPL 손실)까지 압축, 단일 A100에서 1M-token context를 달성한 논문.
Background In long-context LLM inference the KV cache grows linearly with sequence length and batch size, so at long contexts it, not the weights, becomes the dominant GPU-memory bottleneck. By the paper's own measurement on LLaMA-7B at batch 1, weights take 98% of memory at 512 tokens but the KV cache takes 84% at 128K, inverting the relationship. It is like a book whose margin notes grow thicker than the book itself — any road to million-token inference must run through compressing the cache, not the weights.
배경 Long-context LLM inference에서 KV cache는 sequence length와 batch에 비례해 커져, 긴 context에서는 weight가 아니라 KV cache가 GPU memory의 지배적 병목이 된다. 논문의 측정으로는 LLaMA-7B(batch 1) 기준 512 token에서는 weight가 memory의 98%를 차지하지만, 128K token에서는 KV cache가 84%를 차지하며 관계가 역전된다. 책(weight)은 그대로인데 여백에 적는 메모(KV cache)가 책보다 두꺼워지는 셈이라, million-token 추론으로 가는 길은 weight 압축이 아니라 cache 압축을 통과할 수밖에 없다.
Problem Serving million-token contexts requires compressing the KV cache below 4 bits, and the problem is doing so without sacrificing accuracy. At 3 bits each value must be expressed with only 8 levels, and at 2 bits with 4 — like repainting a photo with an eight-color palette, where everything hinges on which eight colors (that is, which axis and which datatype) you choose. Existing uniform integer grids waste those scarce levels on empty regions of the distribution, which is why accuracy had collapsed below 4 bits.
문제 Million-token 급 context를 감당하려면 KV cache를 4-bit 이하 초저정밀도로 압축해야 하는데, 정확도 손실 없이 이를 달성하는 방법이 문제다. 3-bit에서는 각 값을 겨우 8개 level로, 2-bit에서는 4개로 표현해야 하므로 — 사진을 8색 팔레트로 다시 칠하는 것과 같아서 — 어떤 8색을 고르느냐(어느 축으로, 어떤 datatype으로 양자화하느냐)가 전부를 좌우한다. 기존 uniform integer grid는 이 희소한 level을 분포가 비어 있는 곳에 낭비해, 4-bit 미만에서 정확도가 무너져 있었다.
Prior limitations Prior approaches such as ATOM and FlexGen apply uniform per-token 4-bit quantization to the KV cache, ignoring the outlier-channel structure of keys and the channel-mixing effect of RoPE, so perplexity collapses below 4 bits. The collapse is large: at 3-bit, int3 with grouping pushes LLaMA-7B Wikitext-2 perplexity from the fp16 5.68 up to 10.87 (Figure 1). The root cause is a handful of high-magnitude key outlier channels — per-token uniform grids cannot see this channel structure, and after RoPE rotates and mixes channels the structure itself is smeared.
기존 한계 ATOM·FlexGen 같은 기존 방식은 uniform per-token 4-bit 양자화를 KV에 그대로 적용해, key의 outlier channel 구조와 RoPE가 채널을 섞는 효과를 무시하므로 4-bit 미만에서 perplexity가 크게 무너진다. 붕괴 폭은 커서, 3-bit의 int3+grouping은 LLaMA-7B Wikitext-2 perplexity를 fp16의 5.68에서 10.87까지 밀어올린다(Figure 1). 근본 원인은 key의 소수 high-magnitude outlier channel인데, per-token uniform grid는 이 채널 구조를 볼 수 없고, RoPE가 채널을 회전·혼합한 뒤에는 그 구조 자체가 흐려진다.
Goal Achieve 3-bit (and even 2-bit) KV cache quantization with <0.1 perplexity degradation, enabling 1M-token context on a single A100-80GB and 10M tokens on an 8-GPU system. The framing itself is distinctive: the headline metric is context length on fixed hardware rather than accuracy at a fixed budget, making KV memory — not compute — the explicit target. The 1M/10M figures are for LLaMA-7B under the most aggressive 2-bit (nuq2) setting.
목표 3-bit(나아가 2-bit) KV cache 양자화를 <0.1 perplexity 열화로 달성해, 단일 A100-80GB에서 1M, 8-GPU에서 10M token context를 가능하게 하는 것이 목표다. 헤드라인 지표를 "고정 budget에서의 정확도"가 아니라 "고정 하드웨어에서의 context 길이"로 잡은 framing 자체가 특징으로, compute가 아닌 KV memory를 명시적 타깃으로 만든다. 1M/10M 수치는 LLaMA-7B 기준이며 가장 공격적인 2-bit(nuq2) 설정에서의 용량 계산이다.
Method Keys are quantized per-channel before RoPE using non-uniform signposts (nuqX) derived offline via Fisher-information-weighted k-means on a 16-sample × 2K-token Wikitext-2 calibration set, while values are quantized per-token online. Values resist offline calibration because outlier value tokens appear at runtime, so scaling factors and outlier thresholds are computed online for each incoming token, with threshold computation offloaded to the CPU. On top of this, per-vector thresholds isolate ~1% of outliers into an fp16 sparse format — vacuum-pack the whole suitcase but hand-carry the few fragile items — and the attention-sink first token stays in fp16. Inference runs on lookup-table-based custom CUDA kernels that apply RoPE in a fused kernel after dequantization.
방법 Key는 RoPE 적용 전(pre-RoPE)에 per-channel로, Fisher information 가중 k-means로 offline calibration(Wikitext-2 16샘플×2K token)한 non-uniform signpost(nuqX)로 양자화하고, Value는 per-token으로 online 양자화한다. Value는 runtime에 나타나는 outlier token 때문에 offline calibration이 어려워, scaling factor와 outlier threshold를 token마다 online으로 계산하되 threshold 계산은 CPU로 offload한다. 여기에 per-vector threshold로 ~1% outlier를 fp16 sparse 포맷으로 분리하고 — 짐 전체는 진공 압축하되 깨지기 쉬운 소수 물건만 손에 들고 타는 식 — attention sink인 첫 token은 fp16으로 유지한다. 추론은 lookup-table 기반 custom CUDA kernel이 담당하며, dequantization 후에 RoPE를 fused kernel로 적용한다.
Key idea Align the quantization axis with the tensor distribution — key outliers are channel-aligned before RoPE, so quantize keys per-channel pre-RoPE with sensitivity-weighted non-uniform datatypes, values per-token, and push the few remaining outliers into a tiny sparse store. Like splitting wood along the grain, matching the quantization unit to the grain of the distribution (channels for keys, tokens for values) yields far less error at the same bit width. Since RoPE rotates channel pairs and smears the outlier structure, the key trick is an order swap: quantize before the rotation and apply RoPE after dequantization (3.82 and 0.82 perplexity improvements at 3-bit from per-channel and pre-RoPE respectively).
핵심 아이디어 양자화 축을 tensor의 분포 구조에 맞추는 것 — key의 outlier는 RoPE 이전에 channel 방향으로 정렬돼 있으므로 pre-RoPE per-channel + sensitivity 가중 non-uniform datatype으로 잡고, value는 per-token으로 잡으며, 남는 소수 outlier만 sparse로 뺀다. 나무를 결 따라 켜면 쉽게 쪼개지듯, 분포의 결(key는 channel, value는 token)을 따라 양자화 단위를 맞추면 같은 bit 수로 훨씬 적은 오차를 낸다. RoPE는 channel 쌍을 회전시켜 outlier 구조를 흐리므로, 회전 전에 양자화하고 dequantization 후 RoPE를 적용하는 순서 교환이 핵심 트릭이다(3-bit 기준 per-channel 3.82 + pre-RoPE 0.82 perplexity 개선).
Validation Evaluated via Wikitext-2 and C4 perplexity on LLaMA-7B/13B/30B/65B, Llama-2, Llama-3, and Mistral-7B, plus passkey-retrieval, LongBench, and RULER long-context experiments and A6000 kernel microbenchmarks. The component attribution is clean: for 3-bit LLaMA-7B the paper reports a ladder of perplexity improvements — per-channel 3.82 → pre-RoPE 0.82 → non-uniform 0.29 → 1% outliers 0.19. Robustness to calibration-data choice is also checked by cross-calibrating between Wikitext-2 and C4 with near-identical perplexity (Appendix Q).
검증 LLaMA-7B/13B/30B/65B, Llama-2, Llama-3, Mistral-7B에 대해 Wikitext-2·C4 perplexity로 평가하고, passkey retrieval·LongBench·RULER long-context 실험과 A6000 kernel microbenchmark로 검증했다. 구성요소별 기여 분리도 깔끔해서, 3-bit LLaMA-7B 기준 per-channel 3.82 → pre-RoPE 0.82 → non-uniform 0.29 → 1% outlier 0.19의 perplexity 개선 사다리로 보고된다. Wikitext-2와 C4를 교차 calibration해도 perplexity가 거의 변하지 않음을 보여(Appendix Q) calibration 데이터 선택에 대한 강건성도 점검했다.
Results Perplexity degradation of <0.02 at 4-bit, <0.1 at 3-bit, and <0.5 at 2-bit (3.7×/4.8×/6.9× compression) across all models, enabling 1M-token context for LLaMA-7B on one A100-80GB and 10M on 8 GPUs. Concretely, 3-bit with 1% outliers puts LLaMA-7B Wikitext-2 perplexity at 5.75 versus the fp16 5.68 — a 0.07 gap. The custom kernels deliver 1.2–1.6× (key) and 1.3–1.7× (value) latency savings over fp16 on an A6000, and on passkey retrieval even the average-2.33-bit nuq2-1% holds ~100% retrieval where the average-3.05-bit KIVI-2 drops to 0.68–0.76.
결과 4-bit <0.02, 3-bit <0.1, 2-bit <0.5 perplexity 열화(각각 3.7×/4.8×/6.9× 압축)를 전 모델에서 달성했고, LLaMA-7B 기준 단일 A100-80GB에서 1M token·8-GPU에서 10M token context를 가능하게 한다. 구체적으로 3-bit+1% outlier의 LLaMA-7B Wikitext-2 perplexity는 5.75로 fp16 5.68과 0.07 차이다. Custom kernel은 A6000에서 fp16 대비 key 1.2–1.6×, value 1.3–1.7× latency 절감을 보였고, passkey retrieval에서는 평균 2.33-bit(nuq2-1%)로도 ~100% 회수율을 유지해 평균 3.05-bit인 KIVI-2(0.68–0.76)를 앞선다.
Comparison Consistently beats uniform INT4/INT3/INT2, NormalFloat NF4/NF3/NF2, ATOM (group size 128), and FlexGen (group size 64), and shares the per-channel-key observation with the concurrent 2-bit method KIVI. Where KIVI takes the calibration-free route, KVQuant optimizes the datatype itself via offline calibration, and head-to-heads at similar average bit widths are reported in KVQuant's favor (LongBench average 31.21 vs KIVI's 30.04, with fp16 at 31.96, plus higher passkey retrieval). The paper notes that KIVI's public code lacked grouped-query-attention support, so 70B-scale comparisons were excluded.
비교 Uniform INT4/INT3/INT2, NormalFloat NF4/NF3/NF2, ATOM(group 128), FlexGen(group 64)을 일관되게 상회하며, 동시기 2-bit 방법인 KIVI와 같은 per-channel key 관찰을 공유한다. KIVI가 calibration-free 노선이라면 KVQuant는 offline calibration으로 datatype 자체를 최적화하는 노선이고, 비슷한 평균 bit에서의 직접 비교는 KVQuant 우위로 보고된다(LongBench 평균 31.21 vs KIVI 30.04, fp16 31.96; passkey도 우위). 단 KIVI 공개 코드가 grouped-query attention을 지원하지 않아 70B급 비교는 제외했다고 논문이 밝힌다.
Significance It codified the per-channel-key / per-token-value asymmetry plus outlier decomposition as the standard KV-quantization recipe, and showed calibration-based non-uniform datatypes stretch context length to millions of tokens on fixed hardware. Recasting the problem as "how many times longer a context on the same GPU" rather than "a few percent better accuracy" made KV memory an explicit optimization target, and (reviewer judgment) most later KV-quantization work is measured against this recipe. It also put pre-RoPE quantization on the map as a design axis.
의의 Per-channel key / per-token value 비대칭과 outlier 분리라는 KV 양자화의 표준 레시피를 calibration 기반 non-uniform datatype과 함께 정립해, 고정된 하드웨어에서 context 길이를 수백만 token 단위로 확장할 수 있음을 보였다. "정확도 몇 % 개선"이 아니라 "같은 GPU에서 context 몇 배"로 문제를 다시 씌운 framing이 KV memory를 명시적 최적화 대상으로 만들었고, (리뷰어 판단) 이후 KV 양자화 연구 대부분이 이 레시피를 기준 삼아 측정된다. Pre-RoPE 양자화라는 설계 축을 표준 논의에 올린 것도 이 논문이다.
Limitations The paper's own Limitations (Appendix S) name two things: latency benchmarking covers only memory-bound generation, not prompt processing (compressing many keys/values at once), and the current end-to-end implementation copies previous tokens' data when updating the sparse matrix — a memory-allocation inefficiency. It also states that training 100K+-context models remains open but is orthogonal to this inference-efficiency work. (Reviewer judgment) The reported speedups are A6000 kernel-level, so end-to-end throughput gains versus paged fp16 serving remain to be shown, and the cost of storing and updating the ~1% fp16 sparse outliers on a real serving path belongs in that audit.
한계 논문 자체의 Limitations(Appendix S)는 두 가지를 꼽는다: latency benchmark가 memory-bound generation에 국한되어 prompt 처리(여러 K/V 동시 압축)를 다루지 않는 점, 그리고 현재 end-to-end 구현이 sparse matrix 갱신 시 이전 token 데이터를 복사하는 memory allocation 비효율을 가진 점. 100K+ context 모델의 학습 자체도 남은 과제지만 추론 효율에 국한된 본 연구와는 직교한다고 명시한다. (리뷰어 판단) 보고된 speedup은 A6000 kernel 수준이라 paged fp16 serving 대비 end-to-end throughput 이득은 별도 검증이 필요하고, ~1% fp16 outlier의 sparse 저장·갱신이 실제 serving 경로에서 갖는 비용도 그 검증에 포함되어야 한다.
Future work There is no dedicated future-work section, but Appendix S states an explicit implementation plan: eliminating the sparse-matrix update overhead via blocked memory allocation. (Reviewer judgment) Beyond that, natural next steps are transfer to GQA/MLA architectures, combination with eviction and streaming methods, and integration into vLLM-style serving stacks.
향후 과제 논문에 별도 future work 절은 없지만, Appendix S에서 sparse matrix 갱신 오버헤드를 blocked allocation으로 없애겠다는 구현 개선 계획을 명시한다. (리뷰어 판단) 그 밖의 자연스러운 다음 단계는 GQA/MLA 구조로의 이식, eviction·streaming 기법과의 결합, vLLM류 serving 스택 통합이다.
Released resources The official implementation, including calibration code and custom CUDA kernels, is public on GitHub (SqueezeAILab/KVQuant). It spans the Fisher-weighted nuqX datatype derivation, the lookup-table kernels, and an end-to-end generation pipeline that compresses activations dynamically during inference, with calibration cost reported at 1.9–4.5 minutes per layer including k-means, plus 2.8 minutes for Fisher information.
자원 공개 Calibration 코드와 custom CUDA kernel을 포함한 공식 구현이 GitHub(SqueezeAILab/KVQuant)에 공개돼 있다. Fisher 가중 nuqX datatype 도출부터 lookup-table kernel, activation을 추론 중 동적으로 압축하는 end-to-end generation pipeline까지 포함하며, calibration 비용은 layer당 k-means 포함 1.9–4.5분 + Fisher information 계산 2.8분 수준으로 보고된다.
The research thread
Where this paper sits 연구의 흐름
KVQuant is the KV-cache descendant of the same Berkeley group's weight-quantization program. SqueezeLLM built the two tools — sensitivity-weighted non-uniform datatypes derived by Fisher-weighted k-means, and the dense-and-sparse outlier decomposition — and KVQuant transplants both, almost verbatim, from weights to cached activations. The problem it inherits comes from other lines: LLM.int8() showed that a transformer's trouble concentrates in a few outlier channels, while FlexGen and ATOM showed that a uniform per-token grid on the KV cache holds at 4 bits and collapses below. It also lands just as vLLM/PagedAttention made KV memory a first-class systems object and H2O attacked the same wall by evicting entries — KVQuant stands at that junction: keep every token, but bring the weight-quantization toolkit and first ask what the KV tensors actually look like.
The conceptual move is that the quantizer stops being a post-hoc numeric filter and becomes aware of where it lives inside the model's computation. Where prior work treated K and V as one homogeneous blob to compress after the fact, KVQuant reads each tensor's grain — keys are channel-structured, values are not — and, most strikingly, notices that the grain depends on when you look: RoPE's rotation smears the key outlier channels, so quantization moves before the rotation and RoPE is re-applied after dequantization. Quantization becomes an operator scheduled inside the graph, not a codec bolted on at the end. The second reframing is the headline metric itself: not accuracy at k bits but context length on a fixed GPU — quietly redefining what KV compression is for.
The recipe — per-channel keys, per-token values, an outlier escape hatch — became the reference grammar for later KV-quantization work, and the first tension it left sits on the calibration axis. The concurrent KIVI showed a similar accuracy region is reachable calibration-free, splitting the field into a datatype-optimizing branch and a plug-and-play branch. The deeper unresolved tension is systems-side: the sparse fp16 outliers and the pre-RoPE ordering mesh awkwardly with paged serving stacks, and the paper itself concedes its numbers are kernel-level, not end-to-end.
Two next moves are visible. One removes outliers instead of isolating them: the rotation line (QuaRot, SpinQuant) applies Hadamard-style rotations that erase the outliers outright, making the sparse path unnecessary and attacking KVQuant's messiest component head-on. The other is a collision with architecture: GQA and MLA shrink the KV cache at design time, so the open question is whether post-hoc quantization stays complementary or gets absorbed into cache-native designs. And unification with eviction (H2O, DuoAttention) still waits — precision and retention are two halves of one memory budget, and nobody yet allocates them jointly.
KVQuant는 같은 Berkeley 그룹의 weight quantization 계보를 KV cache로 옮겨온 논문이다. SqueezeLLM이 만든 두 도구 — Fisher 가중 k-means로 도출하는 sensitivity 가중 non-uniform datatype과 dense-and-sparse outlier 분해 — 를 weight에서 cached activation으로 거의 그대로 이식했다. 물려받은 문제는 다른 계보에서 온다. LLM.int8()이 transformer activation의 골칫거리가 소수 outlier channel에 몰려 있음을 보였고, FlexGen과 ATOM은 uniform per-token grid가 KV cache에서 4-bit까지는 버티지만 그 아래에서 무너진다는 사실을 남겼다. 마침 vLLM/PagedAttention이 KV memory를 일급 systems 객체로 만들고 H2O가 같은 벽을 eviction으로 공략하던 시점이라, KVQuant는 그 교차점에 서 있다: 모든 token을 유지하되, weight quantization의 도구상자를 들고 KV tensor가 실제로 어떻게 생겼는지부터 묻는다.
개념적 이동은 quantizer가 사후 numeric filter이기를 멈추고 자신이 모델 연산 그래프의 어디에 사는지를 인식하게 됐다는 점이다. 이전 연구가 K와 V를 하나의 균질한 덩어리로 취급해 나중에 압축했다면, KVQuant는 각 tensor의 결을 읽는다 — key는 channel 방향으로 결이 서 있고 value는 그렇지 않다 — 그리고 가장 인상적으로, 그 결이 "언제 보느냐"에 달려 있음을 알아챈다. RoPE의 회전이 key outlier channel을 뭉개므로 양자화를 회전 앞으로 옮기고 dequantization 뒤에 RoPE를 다시 적용하는 것이다. 양자화가 끝에 덧붙이는 codec이 아니라 그래프 안에 스케줄되는 연산자가 됐다. 두 번째 reframing은 헤드라인 지표 자체다: k-bit에서의 정확도가 아니라 고정 GPU에서의 context 길이 — KV 압축이 무엇을 위한 것인지를 조용히 재정의한다.
이 레시피 — per-channel key, per-token value, outlier 비상구 — 는 이후 KV 양자화 연구의 표준 문법이 됐고, 첫 번째 긴장은 calibration 축에 남았다. 동시기의 KIVI가 비슷한 정확도 영역에 calibration-free로 도달할 수 있음을 보이면서, 분야는 datatype을 최적화하는 노선과 plug-and-play 노선으로 갈라졌다. 더 깊은 미해결 긴장은 systems 쪽이다: sparse fp16 outlier와 pre-RoPE 순서는 paged serving 스택과 어색하게 맞물리고, 논문 스스로 수치가 kernel-level이지 end-to-end가 아님을 인정한다.
다음 수는 두 방향이 보인다. 하나는 outlier를 분리하는 대신 제거하는 것 — QuaRot, SpinQuant 같은 rotation 계열이 Hadamard류 회전으로 outlier 자체를 지워 sparse 경로를 불필요하게 만들며, KVQuant의 가장 지저분한 구성요소를 정면으로 공격한다. 다른 하나는 아키텍처와의 충돌이다. GQA와 MLA는 설계 단계에서 KV cache를 줄이므로, 사후 양자화가 보완재로 남을지 cache-native 아키텍처에 흡수될지가 열린 질문이다. 그리고 eviction 계열(H2O, DuoAttention)과의 통합이 남아 있다 — precision과 retention은 결국 하나의 memory budget의 두 반쪽인데, 이를 공동으로 배분하는 그림은 아직 비어 있다.
Sparks — where one could take this 아이디어의 씨앗
-
Since the latency benchmarks stop at memory-bound generation, one could design a prefill-side kernel that quantizes many keys and values at once and extend the speedup story into the compute-bound regime. 논문의 latency benchmark가 memory-bound generation에서 멈추는 만큼, 다수의 key/value를 한 번에 양자화하는 prefill 쪽 kernel을 설계해 compute-bound 구간까지 speedup 스토리를 확장해볼 수 있다.
-
One could replace the copy-on-update sparse outlier matrix with a blocked or paged allocation and run the audit the paper leaves open: end-to-end throughput against paged fp16 serving. copy-on-update 방식의 sparse outlier 행렬을 blocked/paged allocation으로 바꾸고 paged fp16 serving 대비 end-to-end throughput을 측정하는, 논문이 열어둔 바로 그 검증을 수행해볼 수 있다.
-
One could test whether the pre-RoPE per-channel recipe survives GQA and MLA, where head sharing or latent compression may erase the very channel structure the method leans on. pre-RoPE per-channel 레시피가 GQA/MLA처럼 KV를 구조적으로 줄인 아키텍처에서도 살아남는지 — head 공유나 latent 압축이 이 방법이 기대는 channel 구조 자체를 지우는지 — 시험해볼 수 있다.
Seeds grounded in the paper's own stated limitations and open questions. 논문이 스스로 밝힌 limitations와 open questions에 뿌리를 둔 씨앗입니다.
Notes
Notes
KVQuant and KIVI landed within days of each other and converged on the same core observation — key outliers are per-channel, values are not — but KVQuant goes further on the datatype side: pre-RoPE quantization, Fisher-weighted non-uniform signposts, and a dense-and-sparse outlier split, at the cost of an offline calibration pass that KIVI avoids. Its headline framing (context length on fixed hardware, rather than accuracy at a fixed budget) made KV memory, not compute, the explicit target, and most later KV-quantization work is measured against the recipe it standardized. The kernel-level 1.2–1.7× numbers are honest but narrow; the end-to-end serving story was left to later systems.
KIVI가 calibration-free 노선이라면 KVQuant는 offline calibration으로 datatype 자체를 최적화하는 노선으로, 두 논문이 KV 양자화 설계 공간의 양 극단을 정의한다.
Linked reviews
KIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache
Asymmetric KV quantization derived from element-distribution analysis — per-channel keys, per-token values — pushes the KV cache to tuning-free 2 bits, cutting peak memory 2.6x and lifting serving throughput up to 3.47x.
kv-cacheH2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs
A small set of "heavy-hitter" tokens dominates attention mass, so greedily evicting low-accumulated-attention entries keeps only ~20% of the KV cache while matching full-cache accuracy and raising serving throughput up to 29x.
kv-cacheDuoAttention: Efficient Long-Context Inference with Retrieval and Streaming Heads
Learns which attention heads actually retrieve from long context and gives only those a full KV cache, cutting long-context memory up to 2.55x with retrieval accuracy intact.
compressionBudget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
A systematic coupling study shows that quantization, pruning, and KV-cache compression do not simply add up when stacked -- and uses the resulting rules to fit a 70B model on a single 48GB GPU with a 10K-token context.
kv-cacheC²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
C2KV is a KV-cache reuse framework for long-context LLM inference that combines compression with non-prefix reuse via a lightweight sidecar Extractor and a structured attention flow, trained with a compression-concatenation co-training strategy, reaching up to 17x inference acceleration without modifying the base model.
kv-cacheFlashDrive: Algorithm-System Co-Design for Real-Time VLA Autonomous-Driving Inference
Algorithm-system co-design cuts real-time reasoning-VLA latency 4.7x by matching a distinct lightweight fix to each of four structurally different inference bottlenecks.
servingFreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
An edge-native MoE serving system that continuously repartitions GPU memory between KV cache and resident experts at runtime, letting consumer GPUs serve up to 753B-parameter models with 1.3-2.3x higher decode throughput than prior edge engines.
quantizationGaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
GaugeQuant learns the quantization-optimal rotation basis during training itself -- via a LogSumExp outlier-suppression proxy loss with a stop-gradient, constrained to SO(d) by a Cayley transform -- rather than searching for it post-hoc on a frozen model with calibration data, cutting LLaMA-2 7B W4A16 perplexity degradation by 51% with no calibration set and negligible training overhead.
kv-cacheA JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation
Treating the KV cache as a single three-axis tensor rather than a stack of matrices, JoLT compresses only the token and feature axes with a partial Tucker decomposition, restores the discarded energy with a low-bit rotated residual, and lets one shared optimizer decide how to split the byte budget between rank and residual bits and between keys and values -- reaching a near-lossless 2-3x compression zone on both a grouped-query and a multi-head attention model.
compressionKronQ: LLM Quantization via Kronecker-Factored Hessian
KronQ extends GPTQ-style post-training quantization with the output-side gradient covariance (Kronecker-factored Hessian), fixing a blind spot where prior compensation-based PTQ methods treat all output channels as equally important; on 2-bit LLaMA-3-70B it reaches 7.93 perplexity where GPTQ/GPTAQ diverge above 2000.
quantizationLiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection
A lift-then-project weight-quantization scheme that stores each weight block as a linear projection of a 1-bit lattice living in a higher-dimensional space, so the effective bit-width becomes a quasi-continuous dial (lifted dim / block dim) — e.g. Llama-3-70B at 2.4 bits exactly fills a 24GB GPU and clearly beats every 2-bit method forced onto the same card, with a lookup-table-free INT1-GEMM decode path.
kv-cacheWhat to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents
A survey reframes KV-cache eviction, prompt compression, bounded architectural state, and agent memory consolidation as one rate-distortion decision on a shared bytes-per-token-of-history axis, showing all four communities' scoring signals fail identically (deciding what to keep before the query is known) and proposing a COMPACT-Bench benchmark.
quantizationMXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
MXSens allocates 4/6/8-bit mantissa widths per column and layer based on Hessian-guided sensitivity, staying inside hardware-native MXINT microscaling formats (unlike rotation-based methods, which are incompatible with them), and reaches perplexities of 3.77 and 7.63 on LLaMA-2-70B and LLaMA-3-8B under W4A4KV4.
kv-cacheTriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Observes that pre-RoPE Q/K vectors concentrate around fixed centers, so attention can be predicted as a trigonometric series in token distance — a query-free KV eviction score that matches full-attention AIME25 accuracy at 2.5x throughput or 10.7x less KV memory.
quantizationTurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
A random rotation forces every coordinate into a known Beta distribution, so one precomputed Lloyd-Max codebook quantizes any vector stream online with zero calibration — plus a 1-bit QJL residual stage for unbiased inner products — provably within ≈2.7× of the information-theoretic distortion floor, yielding quality-neutral 3.5-bit KV caches and vector-search indexing in near-zero time.
kv-cacheVarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
VarRate is a training-free KV cache codec that allocates a per-token low-rank budget via water-filling over query salience instead of evicting tokens, staying within 3.5-5.5 points of uncompressed accuracy under query-agnostic cache reuse where query-aware eviction loses 11-15 points.