Paper Review
KIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache
Asymmetric KV quantization derived from element-distribution analysis — per-channel keys, per-token values — pushes the KV cache to tuning-free 2 bits, cutting peak memory 2.6x and lifting serving throughput up to 3.47x.
KV cache 원소 분포 분석에서 도출한 비대칭 양자화(K per-channel, V per-token)로 tuning 없이 2bit KV cache를 달성, peak memory 2.6배 절감과 최대 3.47배 throughput 향상을 얻는다.
Background LLM serving amortizes cost via batching, but at large batch and long context the KV cache — not the weights — becomes the memory ceiling. Decode is memory-bound: every token must re-read the whole cache, leaving compute cores idle while it streams in — the cook is fast but the pantry is far, so meal speed is set by how quickly ingredients arrive. The existing detours split three ways: MQA/GQA cuts KV heads but needs training from scratch or fine-tuning, token eviction discards information, and quantization is the remaining axis.
배경 LLM serving은 batching으로 요청당 비용을 낮추지만, batch가 커지고 context가 길어지면 weight가 아니라 KV cache가 메모리 상한이 된다. decode는 토큰마다 cache 전체를 다시 읽어야 하는 memory-bound 연산이라 KV 로딩 동안 연산 코어가 논다 — 요리사는 빠른데 팬트리가 멀어서, 식당의 속도를 재료가 도착하는 속도가 정하는 셈이다. 기존 우회로는 세 갈래인데, KV head 수를 줄이는 MQA/GQA는 처음부터 학습하거나 fine-tuning이 필요하고, token eviction은 정보를 버리며, quantization이 남은 세 번째 축이다.
Problem There was no way to push the KV cache to extreme low bit-width (2-bit, below 4-bit) without fine-tuning or calibration while preserving accuracy. The de-facto per-token quantization holds accuracy at INT4 but drops notably at INT2 (the paper's observation OB1). So the real question becomes: what structural fact about KV tensors would permit 2 bits at all?
문제 KV cache를 4bit 아래의 극저비트(2bit)로 줄이면서도 fine-tuning이나 calibration 없이 정확도를 유지하는 방법이 없었다. 통용되던 per-token 양자화는 INT4까지는 정확도를 지키지만 INT2로 내리면 뚜렷하게 무너진다(논문의 관측 OB1). 그래서 진짜 질문은 "KV tensor의 어떤 구조적 사실이 2bit를 허용하는가"가 된다.
Prior limitations Prior KV quantization treated K and V identically — mostly per-token uniform, since new tokens append along the token axis and that is the only choice that trivially fits streaming decode (FlexGen's 4-bit group-wise is the reference point; Atom-style schemes are also cited). The result was sharp collapse below 4 bits, with no in-depth study of KV cache element distributions. Per-channel was an essentially untried axis precisely because it appears to conflict with streaming.
기존 한계 기존 KV quantization은 K와 V를 같은 방식(주로 per-token uniform)으로 다뤘다 — 새 토큰이 token 축으로 append되는 streaming decode에 자명하게 맞는 유일한 선택이었기 때문이다(FlexGen의 4bit group-wise가 참조점, Atom 계열도 인용). 그 결과 4bit 밑에서 급격히 붕괴했고, KV cache 원소 분포 자체에 대한 심층 분석이 부재했다. per-channel은 streaming과 상충해 보인다는 이유로 사실상 시도되지 않던 축이었다.
Goal A tuning-free, plug-in 2-bit KV cache quantizer that keeps near-FP16 quality. Since the point is to convert the freed memory into larger batches and higher serving throughput, a hardware-friendly implementation compatible with streaming decode (kernel fusion included) is part of the method, not an accessory.
목표 FP16과 거의 같은 품질을 유지하는 tuning-free, plug-in 방식의 2bit KV cache quantizer. 절감된 메모리를 더 큰 batch로 환전해 serving throughput을 올리는 것까지가 목표라서, streaming decode와 호환되는 hardware-friendly 구현(kernel fusion 포함)이 부속물이 아니라 방법의 일부로 요구된다.
Method Quantize the key cache per-channel and the value cache per-token to 2 bits (group size 32), keeping recent tokens as a full-precision residual: keys accumulate until 128 tokens (R) and are then quantized wholesale and concatenated into the 2-bit store, while values sit in a length-R queue whose oldest entry is popped and quantized per-token. Like a shop till: today's takings (recent tokens) stay as cash in the drawer, and get deposited to the bank (the 2-bit store) in fixed bundles. This leaves a full-precision sliding window averaging R/2 tokens for keys and R for values, which the paper itself shows is crucial for hard tasks like GSM8K. Dequantization is fused into tiled matmul (the CUDA Q_MatMul kernel, plus Triton group-wise kernels), and the scheme composes with weight-only quantization.
방법 Key cache는 per-channel, value cache는 per-token으로 2bit 양자화하되(group size 32), 최근 토큰은 full precision residual로 남긴다: key는 128 토큰(R)이 쌓일 때마다 통째로 양자화해 기존 2bit store에 concat하고, value는 길이 R 큐에서 밀려나는 가장 오래된 토큰부터 per-token으로 양자화한다. 가게 금고에 비유하면 오늘 매출(최근 토큰)은 서랍에 현금으로 두고, 일정 묶음이 차면 은행(2bit store)에 예치하는 구조다. 그 결과 key는 평균 R/2, value는 R 길이의 full-precision sliding window가 남는데, 이 창이 GSM8K 같은 어려운 태스크의 정확도에 결정적이라고 논문 스스로 보인다. dequantization은 tiled matmul에 fuse한 Q_MatMul(CUDA)과 Triton group-wise kernel로 구현되어 weight-only quantization과도 호환된다.
Key idea One observation decides everything: K and V are statistically different objects. The key cache concentrates very large magnitudes in a few fixed channels, so a per-token scale gets dragged by those outliers — like setting one exposure per photo (token) when a few pixels always glare in the same spots; giving those columns their own exposure, per-channel quantization cuts attention-score error by about 5x. The value cache shows no outlier pattern, but the attention output is a sparse weighted sum over a few important tokens (measured sparsity 84.3%), so per-token quantization confines error to individual tokens and yields about 15x lower output error. The asymmetry is a prescription forced by measurement, not a design taste.
핵심 아이디어 K와 V는 통계적으로 다른 객체라는 관측이 전부를 결정한다. key cache는 소수의 고정 채널에 매우 큰 magnitude가 몰려 있어 per-token scale이 outlier 채널에 끌려간다 — 몇몇 픽셀이 항상 같은 자리에서 눈부신데 사진(토큰) 한 장마다 노출을 하나로 잡는 격이라, 그 열에 따로 노출을 주는 per-channel이 attention score 오차를 약 5배 줄인다. value cache는 outlier 패턴이 없는 대신 attention output이 소수 중요 토큰의 희소 가중합(측정 sparsity 84.3%)이어서, 오차를 토큰 단위로 가두는 per-token이 output 오차를 약 15배 줄인다. 비대칭은 취향이 아니라 측정이 강제한 처방이다.
Validation Evaluated Llama/Llama-2 (7B, 13B), Falcon-7B, and Mistral-7B on LM-Eval (CoQA EM, TruthfulQA BLEU, GSM8K EM) and eight LongBench tasks (Qasper, QMSum, MultiNews, TREC, TriviaQA, SAMSum, LCC, RepoBench-P), plus Needle-in-a-Haystack retrieval checks on Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2. Closed-ended tasks like MMLU were deliberately excluded because a single decoding step cannot expose the effect of a compressed KV cache — a point still relevant to KV-compression evaluation design. Throughput follows the vLLM protocol: a ShareGPT-based synthetic workload (average input 161, output 338 tokens) on a single A100 80GB.
검증 Llama/Llama-2(7B·13B), Falcon-7B, Mistral-7B를 LM-Eval(CoQA EM, TruthfulQA BLEU, GSM8K EM)과 LongBench 8개 태스크(Qasper, QMSum, MultiNews, TREC, TriviaQA, SAMSum, LCC, RepoBench-P)에서 평가하고, Llama-3-8B-Instruct·Mistral-7B-Instruct-v0.2로 Needle-in-a-Haystack 검색 능력까지 확인했다. MMLU 같은 closed-ended 태스크는 decoding step이 한 번뿐이라 압축된 KV cache의 영향을 드러내지 못한다는 이유로 의도적으로 제외했는데, KV 압축 평가 설계에서 지금도 유효한 지적이다. throughput은 vLLM 프로토콜대로 ShareGPT 기반 합성 workload(평균 입력 161, 출력 338 토큰)를 single A100 80GB에서 측정했다.
Results Maintains near-FP16 quality on most tasks — even on GSM8K, among the hardest generation tasks used, the drop is only about 2% — while cutting peak memory (model weights included) by 2.6x on Llama-2-7B. That memory saving converts into up to 4x larger batches, giving 2.35-3.47x higher throughput on the ShareGPT workload. Ablations show fine grouping itself matters at this bit-width: growing the group size from 32 to 128 drops GSM8K from 20.8 to 17.3.
결과 대부분의 태스크에서 FP16에 근접한 품질을 유지하고 — 가장 어려운 축인 GSM8K에서도 하락이 약 2% — Llama-2-7B 기준 weight 포함 peak memory를 2.6배 줄였다. 이 메모리 절감이 최대 4배 큰 batch로 환전되어 ShareGPT workload에서 2.35~3.47배 throughput 향상을 준다. ablation에서는 group size를 32→128로 키우면 GSM8K가 20.8→17.3으로 떨어져, 극저비트에서는 세밀한 grouping 자체가 실질임을 보였다.
Comparison Compared against FP16, 4-bit per-token, and four fake 2-bit configurations. On Llama-2-13B CoQA, 2-bit per-token/per-token collapses from 66.4 to 52.9, and any configuration that quantizes values per-channel implodes below 3% regardless of how keys are handled (OB2). Only K per-channel + V per-token survives at 2 bits, and adding the full-precision sliding window on top lets KIVI recover most of the fake-quant GSM8K gap.
비교 FP16, 4bit per-token, 그리고 네 가지 fake 2bit 조합과 비교했다. Llama-2-13B CoQA에서 2bit per-token/per-token은 66.4→52.9로 무너지고, value를 per-channel로 잡는 순간 key 방식과 무관하게 3% 미만으로 궤멸한다(OB2). K per-channel + V per-token만이 2bit에서 성립하는 조합이며, 여기에 full-precision sliding window를 더한 KIVI가 fake-quant 대비 GSM8K 격차의 대부분을 회복한다.
Significance It established the K/V asymmetry (per-channel K, per-token V) that most subsequent KV-cache quantization work inherits, deriving it explicitly from distribution measurements, and made a 2-bit KV cache a practical serving option. It is a textbook case of measuring the tensors first and letting the method fall out as a corollary, and its evaluation frame — memory saving converted into batch growth and then throughput — became standard for later work.
의의 이후 KV-cache quantization 연구 대부분이 계승하는 K/V 비대칭(per-channel K, per-token V)을 분포 측정으로부터 처음 명시적으로 정립했고, 2bit KV cache를 실제 serving 옵션으로 만들었다. "tensor를 먼저 측정하고 method는 그 따름정리로 나온다"는 전개의 교과서적 사례이며, 메모리 절감을 batch 확대→throughput으로 환전해 보이는 평가 프레임도 이후 연구의 표준이 됐다.
Limitations The paper itself reports that Falcon-7B (multi-query attention, a single KV head) is already highly compressed, so it degrades badly at 2 bits and needs 4-bit KIVI-4 — a signal that extreme-low-bit headroom depends on KV redundancy. Also (reviewer judgment) attribution of the full-precision window is blurry: the text calls a sufficiently large residual important, yet the R=32→128 sweep is nearly flat on GSM8K (20.6→20.8), suggesting the window's existence, not its length, carries the effect. The paper also concedes non-zero quantization overhead during prefill and decoding.
한계 논문 스스로 multi-query attention인 Falcon-7B는 KV head가 하나뿐이라 이미 고도로 압축된 상태여서 2bit 열화가 크고 4bit(KIVI-4)가 필요하다고 보고한다 — 극저비트 여유가 KV 중복도에 의존한다는 신호다. 또한 (리뷰어 판단) full-precision window의 기여 귀속이 모호하다: 본문은 충분히 큰 residual이 중요하다고 쓰지만 정작 R=32→128 sweep은 GSM8K 20.6→20.8로 거의 평평해서, 창의 길이보다 창의 존재 자체가 효과를 나르는 것으로 보인다. prefill·decode 중 양자화 연산의 오버헤드가 0이 아니라는 점도 논문이 인정한다.
Future work The paper explicitly names deeper fusion of the KV quantization step with preceding operations, to cut prefill/decoding overhead, as future work. (Reviewer judgment) GQA models (Llama-3-8B-Instruct, Mistral-Instruct) are shown working in the appendix, but the outlier-channel analysis is not redone for GQA/MLA caches, so why the structure transfers remains open — as does how the asymmetric prescription reshapes when composed with eviction methods like H2O.
향후 과제 논문은 KV cache 양자화를 선행 연산과 더 깊이 fuse해 prefill·decoding 오버헤드를 줄이는 것을 명시적 향후 과제로 든다. (리뷰어 판단) GQA 모델(Llama-3-8B-Instruct, Mistral-Instruct)은 부록에서 동작이 확인되지만 outlier-channel 분석 자체를 GQA/MLA cache에 다시 수행한 것은 아니어서 구조가 전이되는 이유는 열려 있고, H2O류 eviction 계열과의 결합에서 비대칭 처방이 어떻게 변형되는지도 열린 문제다.
Released resources Official code is on GitHub (jy-yuan/KIVI), built on HuggingFace Transformers and shipping the CUDA fused kernel (Q_MatMul) plus Triton group-wise quantization kernels. Being tuning-free and plug-in, it applies to existing checkpoints with no extra artifacts, and the KIVI-2 / KIVI-4 settings expose an accuracy-compression trade-off.
자원 공개 공식 코드가 GitHub(jy-yuan/KIVI)에 공개되어 있으며, HuggingFace Transformers 위에 구현되어 CUDA fused kernel(Q_MatMul)과 Triton group-wise quantization kernel을 포함한다. tuning-free plug-in 방식이라 별도 checkpoint 없이 기존 모델에 바로 적용되고, KIVI-2/KIVI-4 두 설정으로 정확도-압축 트레이드오프를 고를 수 있다.
The research thread
Where this paper sits 연구의 흐름
KIVI stands where two lines of work that ran separately through 2022-2023 meet in the same tensor. One is the quantization line: LLM.int8() and SmoothQuant showed that transformer activations hide a few persistently extreme channels, and that low-bit schemes live or die by how they treat those outliers. The other is the serving line that re-identified the bottleneck of batched long-context inference as the KV cache, not the weights — attacked architecturally by MQA/GQA, by token eviction in H2O (later DuoAttention), and by direct compression in FlexGen and Atom, whose group-wise quantization stopped at 4 bits with the per-token scales streaming append makes convenient. KIVI continues the compression branch while importing the quantization branch's outlier sensibility, sharing its moment with KVQuant, which reached per-channel keys almost simultaneously via calibration and non-uniform codebooks.
The conceptual change is not the bit count but the unit of analysis. Earlier work treated 'the KV cache' as one homogeneous object with one correct recipe; KIVI's opening move is to deny the noun phrase itself — K and V are statistically different objects, and each dictates its own axis. Keys glare through a few fixed channels, so scales must run per-channel; values hide no outliers but feed a sparse weighted sum over tokens, so error must be confined per-token. The axis stops being a preference and becomes something read off a histogram. The other half is making that reading executable: per-channel scaling looks incompatible with streaming decode, and the full-precision residual window — tokens deposited to the 2-bit bank only in bundles — is the small mechanism that legalizes it. Measure first, and the method falls out as a corollary.
The prescription stuck. Per-channel K + per-token V became the baseline later KV-quant work inherits or defines itself against, and its evaluation frame — memory saved, converted into batch, then throughput — became the genre's scoreboard. The paper also quietly names its own boundary. Falcon's multi-query attention, with a single already-dense KV head, collapses at 2 bits — extreme-low-bit headroom is really redundancy headroom. That reading grows heavier as architectures absorb exactly that redundancy: GQA is now the default and MLA folds the cache into a latent, yet KIVI shows GQA working only in an appendix, without re-running the distribution analysis that justified the recipe.
The open questions are compositional. Does the outlier structure that licensed per-channel keys survive in GQA/MLA caches, or does architectural compression launder it away? How does the asymmetric prescription reshape when eviction dynamically prunes the very token set being quantized? Below 2 bits, uniform integer grids likely run out, so the probable next moves are non-uniform and vector codebooks, plus deeper kernel fusion to erase the quantization overhead the paper concedes. KIVI's larger trajectory is quieter than any of these: the KV cache stopped being a passive buffer and became a measured, structured object — most compression papers since begin by measuring it.
KIVI는 2022~2023년을 따로 달려온 두 갈래 연구가 같은 tensor 위에서 만나는 지점에 서 있다. 한쪽은 quantization 계열이다 — LLM.int8()과 SmoothQuant가 transformer activation에는 항상 극단적 magnitude를 갖는 소수 채널이 있고, low-bit 방식의 성패가 그 outlier 처리에 달렸음을 이미 보여준 상태였다. 다른 쪽은 batched long-context inference의 병목이 weight가 아니라 KV cache임을 짚어낸 serving 계열로, MQA/GQA는 아키텍처로, H2O(이후 DuoAttention)는 token eviction으로, FlexGen과 Atom은 직접 압축으로 공략했지만 — 후자의 group-wise quantization은 streaming append에 편하다는 이유로 택한 per-token scale과 함께 4bit에서 멈춰 있었다. KIVI는 이 압축 분기를 이어가며 quantization 분기의 outlier 감각을 수입하고, calibration과 non-uniform codebook으로 거의 동시에 per-channel key에 도달한 KVQuant와 같은 시기를 공유한다.
개념적으로 바뀐 것은 bit 수가 아니라 분석의 단위다. 이전 연구가 'KV cache'를 하나의 균질한 객체로 놓고 하나의 올바른 레시피를 찾았다면, KIVI의 첫 수는 그 명사구 자체를 부정하는 것이다 — K와 V는 통계적으로 다른 객체이고, 각자가 자기 축을 지시한다. key는 고정된 몇 채널에서 빛나니 scale이 per-channel로 가야 하고, value는 outlier가 없는 대신 소수 토큰의 희소 가중합으로 흘러가니 오차를 per-token으로 가둬야 한다. 양자화 축이 설계 취향이 아니라 histogram에서 읽어내는 값이 된 셈이다. 전환의 나머지 절반은 그 읽기를 실행 가능하게 만드는 일이다: streaming decode와 상충해 보이던 per-channel scaling을, 토큰을 일정 묶음이 찰 때만 2bit 은행에 예치하는 full-precision residual window라는 작은 장치가 합법화한다. 먼저 재면 method는 따름정리로 떨어진다.
이 처방은 정착했다. per-channel K + per-token V는 이후 KV-quant 연구가 계승하거나 맞서 자신을 정의하는 기준선이 됐고, 절감한 메모리를 batch로, batch를 throughput으로 환전하는 평가 프레임은 이 장르의 표준 채점표가 됐다. 동시에 논문은 자기 경계도 조용히 명시한다. KV head가 하나뿐인 Falcon(MQA)은 2bit에서 무너진다 — 극저비트의 여유란 사실 중복도의 여유라는 뜻이다. 이 독해는 해가 갈수록 무거워지는데, 아키텍처가 바로 그 중복을 흡수해 가고 있기 때문이다. GQA는 이미 기본값이고 MLA는 cache를 latent로 접는데, KIVI는 GQA 동작을 부록에서 보일 뿐 레시피를 정당화한 분포 분석을 다시 수행하지는 않는다.
그래서 열린 질문들은 합성의 형태를 띤다. per-channel key를 허가했던 outlier 구조가 GQA/MLA cache에서도 살아남는가, 아니면 아키텍처 압축이 그것을 지워버리는가? 양자화 대상인 token 집합 자체가 eviction으로 동적으로 잘릴 때 비대칭 처방은 어떻게 변형되는가? 2bit 아래에서는 uniform integer grid가 바닥날 공산이 크니, 다음 수는 non-uniform·vector codebook과 논문 스스로 인정한 양자화 오버헤드를 지우는 더 깊은 kernel fusion일 것이다. 다만 KIVI가 놓은 더 큰 궤적은 이보다 조용하다: KV cache가 수동적 buffer이기를 멈추고 측정되는 구조적 객체가 됐다는 것 — 이후의 압축 논문은 대부분 그것을 재는 데서 시작한다.
Sparks — where one could take this 아이디어의 씨앗
-
One could re-run KIVI's element-distribution analysis on GQA/MLA caches to test whether the outlier-channel structure that justifies per-channel keys survives architectural KV compression. KIVI의 원소 분포 분석을 GQA/MLA cache에 다시 수행해, per-channel key를 정당화한 outlier-channel 구조가 아키텍처적 KV 압축 이후에도 살아남는지 확인해볼 수 있다.
-
One could run a controlled attribution study of whether the residual window helps through its existence or its length, given the paper's nearly flat R=32→128 sweep. full-precision residual window의 효과가 창의 존재에서 오는지 길이에서 오는지 — R=32→128 sweep이 거의 평평하므로 — 통제된 attribution 실험으로 분리해볼 수 있다.
-
One could measure how the per-channel-K / per-token-V prescription reshapes under H2O-style eviction, where the token set being quantized is itself dynamically pruned. H2O류 eviction과 결합해, 양자화 대상 token 집합이 동적으로 잘릴 때 per-channel-K/per-token-V 처방이 어떻게 변형되는지 측정해볼 수 있다.
Seeds grounded in the paper's own stated limitations and open questions. 논문이 스스로 밝힌 limitations와 open questions에 뿌리를 둔 씨앗입니다.
Notes
Notes
KIVI is the reference point for asymmetric KV-cache quantization: it turned a distributional observation — structured outlier channels in keys, none in values — into a quantization axis choice, and nearly every later KV-quant paper either inherits the per-channel-K/per-token-V split or defines itself against it. The full-precision residual window it introduced is now a standard ingredient, quietly doing accuracy work in most streaming low-bit schemes. Read alongside KVQuant for the non-uniform/outlier-isolation branch of the same problem, and H2O / DuoAttention for the orthogonal axis of dropping cache entries rather than shrinking them.
KIVI는 “tensor를 먼저 측정하고 method는 그 따름정리로 나온다”는 전개의 교과서적 사례로, KV quantization 계열을 읽을 때 출발점으로 삼기 좋은 논문이다.
Linked reviews
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Pre-RoPE per-channel key and per-token value quantization with sensitivity-weighted non-uniform datatypes plus ~1% fp16 outliers pushes the KV cache to 3 bits at under 0.1 perplexity cost, reaching million-token contexts on a single A100.
kv-cacheH2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs
A small set of "heavy-hitter" tokens dominates attention mass, so greedily evicting low-accumulated-attention entries keeps only ~20% of the KV cache while matching full-cache accuracy and raising serving throughput up to 29x.
kv-cacheDuoAttention: Efficient Long-Context Inference with Retrieval and Streaming Heads
Learns which attention heads actually retrieve from long context and gives only those a full KV cache, cutting long-context memory up to 2.55x with retrieval accuracy intact.
compressionBiSCo-LLM: Lookup-Free Binary Spherical Coding for Extreme Low-Bit Large Language Model Compression
BiSCo-LLM replaces stored VQ codebooks with a codebook-free binary spherical code (bit-packed sign stream) plus a residual BSQ stage and category-wise recovery distillation, reaching 10.18 WikiText-2 perplexity vs. 9.73 FP16 on Qwen3-8B at an extreme low-bit storage budget.
compressionBudget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
A systematic coupling study shows that quantization, pruning, and KV-cache compression do not simply add up when stacked -- and uses the resulting rules to fit a 70B model on a single 48GB GPU with a 10K-token context.
kv-cacheC²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
C2KV is a KV-cache reuse framework for long-context LLM inference that combines compression with non-prefix reuse via a lightweight sidecar Extractor and a structured attention flow, trained with a compression-concatenation co-training strategy, reaching up to 17x inference acceleration without modifying the base model.
diffusion-llmFast-dLLM: Training-Free Acceleration of Diffusion LLMs via KV Cache and Parallel Decoding
A block-wise approximate KV cache plus confidence-thresholded parallel decoding accelerates open diffusion LLMs by up to 27.6x end-to-end with 1-2 point accuracy loss, entirely training-free.
kv-cacheFlashDrive: Algorithm-System Co-Design for Real-Time VLA Autonomous-Driving Inference
Algorithm-system co-design cuts real-time reasoning-VLA latency 4.7x by matching a distinct lightweight fix to each of four structurally different inference bottlenecks.
servingFreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
An edge-native MoE serving system that continuously repartitions GPU memory between KV cache and resident experts at runtime, letting consumer GPUs serve up to 753B-parameter models with 1.3-2.3x higher decode throughput than prior edge engines.
quantizationGaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
GaugeQuant learns the quantization-optimal rotation basis during training itself -- via a LogSumExp outlier-suppression proxy loss with a stop-gradient, constrained to SO(d) by a Cayley transform -- rather than searching for it post-hoc on a frozen model with calibration data, cutting LLaMA-2 7B W4A16 perplexity degradation by 51% with no calibration set and negligible training overhead.
servingHyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory
HyMCache stages KV-cache blocks in a CXL-hybrid memory tier (a little device DRAM in front of large SSD-backed capacity) for multi-turn LLM serving, using request-level prefix prefetching to exploit the read-dominant, append-only nature of multi-turn access, reaching 3.0x the throughput of local LMCache at a matched DRAM budget while using 16x less DRAM than a 1TB distributed-DRAM Mooncake deployment.
kv-cacheA JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation
Treating the KV cache as a single three-axis tensor rather than a stack of matrices, JoLT compresses only the token and feature axes with a partial Tucker decomposition, restores the discarded energy with a low-bit rotated residual, and lets one shared optimizer decide how to split the byte budget between rank and residual bits and between keys and values -- reaching a near-lossless 2-3x compression zone on both a grouped-query and a multi-head attention model.
compressionKronQ: LLM Quantization via Kronecker-Factored Hessian
KronQ extends GPTQ-style post-training quantization with the output-side gradient covariance (Kronecker-factored Hessian), fixing a blind spot where prior compensation-based PTQ methods treat all output channels as equally important; on 2-bit LLaMA-3-70B it reaches 7.93 perplexity where GPTQ/GPTAQ diverge above 2000.
kv-cacheError Certificates for KV-Cache Eviction via Randomized Design
This paper proves that deterministic top-k KV-cache eviction cannot certify its own damage -- evicted values can be altered so everything the serving system retains is unchanged while the true attention-output error grows unboundedly -- and shows that a cheap randomized retention scheme (Poisson sampling plus one logit correction) restores enough statistical structure to build a real per-step error certificate, though pre-registered testing finds that certificate is worth its 2x overhead only for attributing and scheduling recovery from failures, not predicting them.
quantizationLiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection
A lift-then-project weight-quantization scheme that stores each weight block as a linear projection of a 1-bit lattice living in a higher-dimensional space, so the effective bit-width becomes a quasi-continuous dial (lifted dim / block dim) — e.g. Llama-3-70B at 2.4 bits exactly fills a 24GB GPU and clearly beats every 2-bit method forced onto the same card, with a lookup-table-free INT1-GEMM decode path.
kv-cacheWhat to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents
A survey reframes KV-cache eviction, prompt compression, bounded architectural state, and agent memory consolidation as one rate-distortion decision on a shared bytes-per-token-of-history axis, showing all four communities' scoring signals fail identically (deciding what to keep before the query is known) and proposing a COMPACT-Bench benchmark.
on-deviceMobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
A systematic sub-billion design-space study showing that at 125M-350M scale deep-thin layouts, embedding sharing, GQA, and block-wise layer sharing buy accuracy per byte, yielding state-of-the-art on-device LMs.
moePagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
PagedWeight treats MoE expert weights as elastic, pageable memory — like vLLM's PagedAttention does for the KV cache — dynamically re-quantizing routed experts at runtime to free GPU memory for a growing KV cache, reaching FP16-equivalent accuracy with up to 72.0% memory savings and 1.94x throughput.
kv-cacheTriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Observes that pre-RoPE Q/K vectors concentrate around fixed centers, so attention can be predicted as a trigonometric series in token distance — a query-free KV eviction score that matches full-attention AIME25 accuracy at 2.5x throughput or 10.7x less KV memory.
quantizationTurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
A random rotation forces every coordinate into a known Beta distribution, so one precomputed Lloyd-Max codebook quantizes any vector stream online with zero calibration — plus a 1-bit QJL residual stage for unbiased inner products — provably within ≈2.7× of the information-theoretic distortion floor, yielding quality-neutral 3.5-bit KV caches and vector-search indexing in near-zero time.
kv-cacheVarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
VarRate is a training-free KV cache codec that allocates a per-token low-rank budget via water-filling over query salience instead of evicting tokens, staying within 3.5-5.5 points of uncompressed accuracy under query-agnostic cache reuse where query-aware eviction loses 11-15 points.