Paper Review
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
A random rotation forces every coordinate into a known Beta distribution, so one precomputed Lloyd-Max codebook quantizes any vector stream online with zero calibration — plus a 1-bit QJL residual stage for unbiased inner products — provably within ≈2.7× of the information-theoretic distortion floor, yielding quality-neutral 3.5-bit KV caches and vector-search indexing in near-zero time.
Random rotation으로 좌표 분포를 알려진 Beta 분포로 고정해, 미리 계산한 Lloyd-Max codebook 하나로 어떤 vector stream이든 calibration 없이 online 양자화하고(1-bit QJL residual 단계로 inner product unbiased 보장), 정보이론적 distortion 하한의 ≈2.7배 이내임을 증명 — 3.5-bit KV cache 품질 중립과 indexing 시간 ~0의 vector search를 달성한 논문.
Background Vector quantization — compressing high-dimensional Euclidean vectors while preserving their geometry — is a classic problem rooted in Shannon's source-coding theory. Two modern inference workloads have pushed it onto the hot path: the KV cache in long-context LLM decoding and embedding stores in nearest-neighbor search, both of which consume vectors almost exclusively through inner products. In both settings vectors arrive as a stream, so the quantizer must operate online, without a calibration pass over the data it is about to compress.
배경 Vector quantization — 고차원 Euclidean vector를 기하 구조를 보존하며 압축하는 문제 — 는 Shannon source coding 이론에 뿌리를 둔 고전 문제다. 최근 두 inference workload가 이를 hot path로 끌어올렸다: long-context LLM decoding의 KV cache와 nearest-neighbor search의 embedding store로, 둘 다 vector를 거의 inner product로만 소비한다. 두 세팅 모두 vector가 stream으로 도착하므로, quantizer는 압축할 데이터를 미리 보는 calibration pass 없이 online으로 동작해야 한다.
Problem The problem is quantizing vectors on the fly, at any bit-width b and dimension d, while controlling two different distortions at once: mean-squared error on the vectors and error on inner products. The two objectives genuinely conflict — the paper shows that MSE-optimal quantizers yield biased inner-product estimates, and since attention only ever reads the KV cache through query-key dot products, that bias directly distorts attention scores. It is like music compression: what must survive is how the track sounds to the listener (the inner product with a query), not the exact waveform (MSE) — an encoder tuned purely for waveform error can systematically color the sound.
문제 임의의 bit-width b와 차원 d에서, vector에 대한 MSE와 inner product 오차라는 서로 다른 두 distortion을 동시에 제어하며 on-the-fly로 양자화하는 것이 문제다. 두 목표는 실제로 충돌한다 — 논문은 MSE-optimal quantizer가 inner product 추정에 bias를 만든다는 것을 보이는데, attention은 KV cache를 오직 query-key dot product로만 읽으므로 이 bias가 그대로 attention 점수를 왜곡한다. 음악 압축에 비유하면: 살아남아야 하는 것은 청자에게 들리는 소리(query와의 inner product)지 파형 자체(MSE)가 아니며, 파형 오차만 최적화한 encoder는 소리를 체계적으로 물들일 수 있다.
Prior limitations Product quantization reaches good recall but requires data-dependent codebook training — indexing takes 37–494 s in the paper's benchmarks — and offers no near-optimal distortion guarantee. QJL is data-oblivious and unbiased but fixed at 1 bit, RaBitQ lacks a fully vectorized GPU implementation (per the paper), and KV-cache scalar schemes like KIVI carry no formal distortion guarantee at all. Token-eviction methods (SnapKV, PyramidKV) discard whole tokens and visibly fail long-context retrieval at the same memory budget.
기존 한계 Product quantization은 recall은 좋지만 data-dependent codebook 학습이 필요해 — 논문 벤치마크에서 indexing에 37–494초 — near-optimal distortion 보장이 없다. QJL은 data-oblivious이고 unbiased지만 1-bit 고정이고, RaBitQ는 (논문 주장으로는) fully vectorized GPU 구현이 없으며, KIVI 같은 KV-cache scalar 방식은 형식적 distortion 보장이 전혀 없다. Token eviction 계열(SnapKV, PyramidKV)은 token을 통째로 버려 같은 메모리 예산에서 long-context retrieval이 눈에 띄게 무너진다.
Goal Build a single data-oblivious online quantizer that provably approaches the information-theoretic distortion floor for both MSE and inner product, across all bit-widths and dimensions. Practically: a KV cache below 4 bits per channel with no quality loss, and vector-search indexing whose codebook cost is effectively zero.
목표 모든 bit-width와 차원에서 MSE와 inner product 둘 다에 대해 정보이론적 distortion 하한에 증명 가능하게 근접하는, 단일 data-oblivious online quantizer를 만드는 것이 목표다. 실용적으로는 품질 손실 없는 4-bit 미만 KV cache와, codebook 비용이 사실상 0인 vector-search indexing이다.
Method TurboQuant-mse first applies a random rotation, after which every coordinate follows a known Beta distribution (converging to N(0, 1/d) in high dimension) regardless of the input, with near-independent coordinates — so a precomputed optimal Lloyd-Max scalar codebook, solved once as a continuous 1-D k-means, quantizes each coordinate with no training data. TurboQuant-prod spends b−1 bits on that MSE stage, then encodes the residual with the 1-bit Quantized-JL (QJL) transform, cancelling the bias to yield an unbiased inner-product estimator. For the KV-cache experiments, channels are split into outlier and non-outlier groups, each quantized by an independent TurboQuant instance (the 2.5-bit setting is 32 outlier channels at 3 bits plus 96 channels at 2 bits). The rotation works like a casino shuffle: whatever order the deck arrives in, after shuffling every position follows the same known distribution, so one pre-designed codebook fits all inputs.
방법 TurboQuant-mse는 먼저 random rotation을 적용한다 — 회전 후 모든 좌표는 입력과 무관하게 알려진 Beta 분포(고차원에서 N(0, 1/d)로 수렴)를 따르고 좌표 간 near-independence가 성립하므로, continuous 1-D k-means로 한 번 풀어 미리 계산해 둔 optimal Lloyd-Max scalar codebook을 학습 데이터 없이 좌표별로 적용한다. TurboQuant-prod는 b−1 bit를 이 MSE 단계에 쓰고 residual을 1-bit Quantized-JL(QJL) transform으로 인코딩해, bias가 상쇄된 unbiased inner-product estimator를 만든다. KV cache 실험에서는 채널을 outlier/non-outlier 그룹으로 나눠 각각 독립적인 TurboQuant 인스턴스로 양자화한다(2.5-bit 세팅 = outlier 32채널 3-bit + 나머지 96채널 2-bit). Rotation은 카지노 셔플과 같다: 덱이 어떤 순서로 오든 셔플 후에는 모든 위치가 같은 알려진 분포를 따르므로, 미리 설계한 codebook 하나가 모든 입력에 들어맞는다.
Key idea Randomness turns a data-dependent problem into a data-free one — rotating first fixes the coordinate distribution in advance, so codebook design happens once, offline, against a distribution known before any data is seen. The second insight is resolving the conflict between MSE-optimality and unbiasedness hierarchically: quantize coarsely for MSE, then spend exactly one bit on an unbiased sketch of the leftover residual. That residual bit acts like fair rounding in accounting — always rounding one way accumulates error, while a randomized correction on the remainder makes the totals come out right on average.
핵심 아이디어 Randomness가 data-dependent 문제를 data-free 문제로 바꾼다 — 먼저 회전하면 좌표 분포가 사전에 고정되므로, codebook 설계를 데이터를 보기도 전에 알려진 분포에 대해 offline으로 한 번만 하면 된다. 두 번째 통찰은 MSE-optimality와 unbiasedness의 충돌을 계층적으로 푸는 것: MSE 기준으로 거칠게 양자화한 뒤, 남은 residual에 정확히 1 bit를 써서 unbiased sketch를 만든다. 이 residual bit는 회계의 공정한 반올림과 같다 — 한 방향으로만 반올림하면 오차가 누적되지만, 나머지에 대한 randomized 보정은 평균적으로 합계를 정확히 맞춘다.
Validation Three theorems — an MSE upper bound of (√3π/2)·4^(−b), an unbiased inner-product bound, and matching information-theoretic lower bounds — plus synthetic experiments showing empirical distortion tracks the bounds. KV-cache evaluation runs on Llama-3.1-8B-Instruct and Ministral-7B-Instruct: Needle-in-a-Haystack from 4k to 104k tokens and LongBench-V1, against KIVI, PolarQuant, SnapKV, and PyramidKV at a fixed 25% memory budget. Vector search is evaluated on GloVe (d=200) and OpenAI3 embeddings (d=1536/3072) against product quantization and RaBitQ.
검증 세 개의 정리 — MSE 상한 (√3π/2)·4^(−b), unbiased inner-product 상한, 그리고 이에 대응하는 정보이론적 하한 — 에 더해 empirical distortion이 bound를 따라감을 보이는 synthetic 실험으로 이론을 검증했다. KV cache는 Llama-3.1-8B-Instruct와 Ministral-7B-Instruct에서 4k–104k token Needle-in-a-Haystack과 LongBench-V1으로, KIVI·PolarQuant·SnapKV·PyramidKV 대비 25% 고정 메모리 예산에서 평가했다. Vector search는 GloVe(d=200)와 OpenAI3 embedding(d=1536/3072)에서 product quantization·RaBitQ 대비로 평가했다.
Results Distortion is provably within √3π/2 ≈ 2.7 of the lower bound at every bit-width (tighter at low bits), and the prod variant is exactly unbiased. On Needle-in-a-Haystack TurboQuant scores 0.997 — identical to the full-precision cache — at 4× compression (KIVI 0.981, PolarQuant 0.995, SnapKV 0.858), and on LongBench-V1 3.5-bit is quality-neutral while 2.5-bit degrades only marginally at ≥4.5× compression. In nearest-neighbor search it beats PQ and RaBitQ on recall across all three datasets while cutting 4-bit quantization time to 0.0007–0.0021 s, versus 37–494 s for PQ and 597–3957 s for RaBitQ.
결과 Distortion이 모든 bit-width에서 하한의 √3π/2 ≈ 2.7배 이내임이 증명되고(저비트에서는 더 타이트), prod 변형은 정확히 unbiased다. Needle-in-a-Haystack에서 TurboQuant는 4× 압축에도 full-precision cache와 동일한 0.997을 기록했고(KIVI 0.981, PolarQuant 0.995, SnapKV 0.858), LongBench-V1에서는 ≥4.5× 압축으로 3.5-bit가 품질 중립, 2.5-bit가 근소한 열화에 그친다. Nearest-neighbor search에서는 세 데이터셋 모두 PQ·RaBitQ보다 높은 recall을 보이면서 4-bit quantization 시간을 0.0007–0.0021초로 줄였다(PQ 37–494초, RaBitQ 597–3957초).
Comparison The KV results split cleanly by method class: guarantee-carrying quantizers (TurboQuant, PolarQuant) sit above the heuristic scalar scheme KIVI, and every quantizer dominates token eviction (SnapKV, PyramidKV) at the same memory budget. Unlike KIVI and PolarQuant, TurboQuant also quantizes tokens generated during decoding rather than leaving them in full precision. Against calibration-based recipes in the KVQuant line, its codebook is universal by construction — no calibration set, no per-model statistics.
비교 KV 결과는 방법 계열별로 깔끔하게 갈린다: 이론 보장을 가진 quantizer(TurboQuant, PolarQuant)가 heuristic scalar 방식인 KIVI보다 위에 있고, 모든 quantizer가 같은 메모리 예산의 token eviction(SnapKV, PyramidKV)을 압도한다. KIVI·PolarQuant와 달리 TurboQuant는 decoding 중 생성되는 token까지 양자화하며 full precision으로 남겨두지 않는다. KVQuant 계열의 calibration 기반 레시피와 달리 codebook이 구성상 universal이라 calibration set도 모델별 통계도 필요 없다.
Significance It puts a Shannon-style floor under the largely empirical KV-quantization literature: a data-oblivious online VQ is shown to be near-optimal for both MSE and inner product at all bit-widths, so "how low can KV bits go" gets a principled answer rather than a benchmark-by-benchmark one. It also unifies KV-cache compression and vector-database quantization as one primitive with one analysis. Knowing the lower bound is like knowing the speed of light: it tells the whole field how much room is left, and TurboQuant shows you can drive within about 2.7× of that limit.
의의 경험적으로만 굴러가던 KV-quantization 문헌에 Shannon 스타일의 바닥을 깔았다: data-oblivious online VQ가 모든 bit-width에서 MSE와 inner product 둘 다에 near-optimal임을 보였으므로, "KV bit를 얼마나 낮출 수 있나"가 벤치마크 단위가 아닌 원리적 답을 얻는다. KV-cache 압축과 vector-database quantization을 하나의 primitive, 하나의 분석으로 통합한 것도 의의다. 하한을 아는 것은 빛의 속도를 아는 것과 같다 — 분야 전체에 남은 여지가 얼마인지 알려주며, TurboQuant는 그 한계의 약 2.7× 안쪽까지 달릴 수 있음을 보였다.
Limitations The paper has no explicit limitations section. The analysis assumes unit-norm vectors (norms can be stored separately, which the authors argue is standard and not restrictive), and entropy coding of codebook indices — which would push the average bit-width down toward the coordinate entropy — is deliberately left out for simplicity and speed. (Reviewer judgment) arXiv v1 reports quantization wall-clock only in the vector-search setting; end-to-end decode latency with the rotation and codebook lookups on the attention hot path is not measured in this version, and the LLM evidence covers only two 7–8B models.
한계 논문 내 명시적 limitations 절은 없다. 분석은 unit-norm vector를 가정하고(norm은 따로 저장하면 되므로 저자들은 표준적이며 제약이 아니라고 주장), codebook index의 entropy coding — 평균 bit-width를 좌표 entropy 수준까지 더 낮출 수 있는 — 은 단순성과 속도를 위해 의도적으로 제외했다. (리뷰어 판단) arXiv v1은 quantization wall-clock을 vector-search 세팅에서만 보고하며, rotation과 codebook lookup이 attention hot path에 올라간 end-to-end decode latency는 이 버전에서 측정되지 않았고, LLM 증거도 7–8B 모델 두 개에 한정된다.
Future work No explicit future-work section is stated; the paper itself only points to entropy-encoding the codebook pointers as an unexploited efficiency gain. (Reviewer judgment) Natural next steps are fused attention kernels and serving-stack integration, evaluation on larger models and GQA/MLA cache layouts, and combination with orthogonal eviction or token-selection methods.
향후 과제 논문 내 명시적 future-work 절은 없으며, 논문 스스로는 codebook pointer의 entropy encoding을 미활용 효율 이득으로만 언급한다. (리뷰어 판단) fused attention kernel과 serving stack 통합, 더 큰 모델과 GQA/MLA cache layout에서의 평가, 직교적인 eviction·token-selection 방법과의 결합이 자연스러운 다음 단계다.
Released resources No official code link appears in arXiv v1 (not stated in the paper). The paper is public on arXiv (CC BY 4.0) and, as an ICLR 2026 paper, on OpenReview with reviews. Third-party open-source reimplementations exist on GitHub, but as none are author-official they are not linked here.
자원 공개 arXiv v1에는 공식 코드 링크가 없다(논문 내 명시 없음). 논문은 arXiv(CC BY 4.0)와 ICLR 2026 OpenReview(리뷰 포함)에 공개돼 있다. GitHub에 제3자 오픈소스 재구현이 여럿 있으나 저자 공식이 아니어서 여기에는 링크하지 않는다.
The research thread
Where this paper sits 연구의 흐름
TurboQuant stands at the junction of two traditions that rarely cite each other. The older one is classical source coding — Lloyd-Max quantization, Shannon's rate-distortion floors, the sketching lineage of Johnson-Lindenstrauss — which these authors had already carried into LLM serving with QJL (Zandieh, Daliri, Han, 2024), a 1-bit JL transform giving unbiased inner products for KV caches, and its sibling PolarQuant. The younger one is the empirical KV-cache quantization literature — KIVI's per-channel-key/per-token-value asymmetry, KVQuant's calibrated non-uniform codebooks. Next door, vector search had its own compression canon in product quantization (Jégou et al.) and RaBitQ, and the random-rotation move itself was living a parallel life in weight quantization (QuIP#, QuaRot), Gaussianizing coordinates before rounding. TurboQuant is what happens when the sketching lineage stops being a 1-bit specialist: extend QJL to arbitrary bit-widths, put Lloyd-Max on the rotated coordinates, and bring matching lower bounds along.
The conceptual shift is this — where prior KV work measures the data, TurboQuant erases it. A calibration recipe is a portrait of one model's cache; the rotation makes the coordinate distribution a known constant, so the codebook is derived once, offline, against mathematics rather than measurements. The other half is that quality claims change type. "3.5-bit is quality-neutral" stops being an empirical episode and becomes an instance of a theorem — distortion provably within ≈2.7× of the information-theoretic floor, with unbiasedness for the one statistic attention actually reads: the inner product. And because the proof holds for arbitrary vector streams, KV caches and vector-database embeddings fold into a single primitive — same rotation, same codebook, same analysis.
What it opens is a change of scoreboard. Once a lower bound with a named constant exists, every future quantizer can be graded against the floor instead of last year's baseline; the interesting quantity becomes the remaining 2.7× gap, and the paper itself points at its first slice — entropy coding of codebook indices, omitted for speed. The residual tension is that the guarantee lives one layer away from where LLMs live: the theorem bounds per-vector distortion, yet the KV experiments still had to reintroduce outlier-channel splitting by hand — distribution-free optimality and model-specific structure are not yet unified; the theorem covers the rotation's output, not the transformer's tolerance.
The next moves are visible from here. On the systems side, wall-clock evidence exists only for vector-search indexing, not the attention hot path, so fused kernels, serving-stack integration, and evidence beyond two 7–8B dense models come first. On the theory side, expect pushes from unbiased inner products toward end-to-end attention error through the softmax, and codebooks that close the constant-factor gap. And on the vector-database side, near-zero indexing time quietly deletes a pipeline stage: when a codebook costs milliseconds instead of minutes, streaming and re-indexable ANN systems stop being exotic designs.
TurboQuant는 서로 인용이 드문 두 전통의 교차점에 서 있다. 오래된 쪽은 고전 source coding이다 — 1950년대 말의 Lloyd-Max scalar quantization, Shannon의 rate-distortion 하한, 그리고 Johnson-Lindenstrauss embedding의 sketching 계보. 같은 저자들이 이미 QJL(Zandieh, Daliri, Han, 2024) — KV cache에 unbiased inner product를 주는 1-bit JL transform — 과 그 형제 격인 PolarQuant로 이 계보를 LLM serving에 들여온 바 있다. 젊은 쪽은 경험적 KV-cache quantization 문헌이다 — KIVI의 per-channel key/per-token value 비대칭, KVQuant의 calibration 기반 non-uniform codebook. 옆 동네 vector search에는 product quantization(Jégou et al.)과 RaBitQ라는 자체 압축 정전이 있었고, random rotation이라는 수 자체는 weight quantization 쪽(QuIP#, QuaRot)에서 rounding 전에 좌표를 Gaussianize하는 용도로 병행하는 삶을 살고 있었다. TurboQuant는 sketching 계보가 1-bit 전문가이기를 그만둘 때 나오는 결과다: QJL을 임의 bit-width로 확장하고, 회전된 좌표 위에 Lloyd-Max를 얹고, 대응하는 하한까지 함께 가져왔다.
개념적 전환은 이렇다 — 이전 KV 연구가 데이터를 측정했다면 TurboQuant는 데이터를 지운다. calibration 레시피는 특정 모델 cache의 초상화지만, rotation은 좌표 분포를 알려진 상수로 만들어 codebook을 측정이 아닌 수학에 대해 offline으로 한 번만 유도하게 한다. 전환의 나머지 절반은 품질 주장의 유형이 바뀐다는 점이다. "3.5-bit는 품질 중립"이 경험적 에피소드이기를 멈추고 정리(theorem)의 사례가 된다 — distortion이 정보이론적 바닥의 ≈2.7배 이내임이 증명되고, attention이 실제로 읽는 유일한 통계량인 inner product에 대해 unbiasedness가 보장된다. 그리고 임의의 vector stream에 대해 증명했기 때문에, KV cache와 vector-database embedding이 같은 rotation, 같은 codebook, 같은 분석을 쓰는 하나의 primitive로 접힌다.
이것이 여는 것은 채점표의 교체다. 이름 붙은 상수를 가진 하한이 존재하는 순간, 이후의 모든 quantizer는 작년 baseline이 아니라 바닥을 기준으로 채점될 수 있다. 흥미로운 양은 남은 2.7× 갭이 되고, 논문 스스로 그 첫 조각을 가리킨다 — 속도를 위해 생략한 codebook index의 entropy coding. 남은 긴장은 보장이 LLM에게는 한 층 어긋난 곳에 산다는 점이다: 정리는 per-vector distortion을 묶지만, KV 실험은 여전히 outlier-channel 분리를 손으로 다시 들여와야 했다 — distribution-free optimality와 모델 특유의 구조가 아직 통합되지 않았다는 신호다. 정리가 커버하는 것은 rotation의 출력이지 transformer의 허용 오차가 아니다.
다음 수는 여기서 보인다. 시스템 쪽에서는 wall-clock 증거가 측정된 곳 — vector-search indexing — 에만 있고 attention hot path에서는 아직이므로, fused kernel과 serving stack 통합, 그리고 7–8B dense 모델 두 개 너머의 증거가 자연스러운 후속이다. 이론 쪽에서는 unbiased inner product에서 softmax를 통과한 end-to-end attention 오차로의 확장과, 상수 갭을 좁히는 codebook이 나올 것이다. 그리고 vector database 쪽에서는 indexing 시간 ~0이 파이프라인 한 단계를 조용히 삭제한다 — codebook 구축이 분 단위가 아니라 밀리초 단위가 되면, streaming·re-indexable ANN 시스템은 더 이상 이국적인 설계가 아니다.
Sparks — where one could take this 아이디어의 씨앗
-
One could measure how far entropy coding of the codebook indices — the headroom the paper names but leaves out — pushes the effective bit-width below nominal, and at what decode-throughput cost. 논문이 스스로 미활용 여지로 지목하고도 생략한 codebook index entropy coding이 유효 bit-width를 명목치 아래로 얼마나 끌어내리는지, 그리고 decode throughput 비용은 얼마인지 측정해볼 수 있다.
-
One could fuse the rotation and codebook lookup into an attention kernel and report end-to-end decode latency at serving batch sizes, closing arXiv v1's vector-search-only wall-clock gap. rotation과 codebook lookup을 attention kernel에 fuse해 serving batch 규모에서 end-to-end decode latency를 보고함으로써, vector-search 세팅에만 wall-clock을 보고한 arXiv v1의 공백을 메워볼 수 있다.
-
One could test whether the distortion guarantees survive composition with SnapKV-style token eviction under a joint memory budget, where quantization and eviction currently split the budget without a shared analysis. quantization과 eviction이 같은 메모리 예산을 공유 분석 없이 나눠 쓰는 현 상태에서, distortion 보장이 SnapKV류 token eviction과 결합해도 유지되는지 공동 예산 하에서 검증해볼 수 있다.
Seeds grounded in the paper's own stated limitations and open questions. 논문이 스스로 밝힌 limitations와 open questions에 뿌리를 둔 씨앗입니다.
Notes
Notes
TurboQuant reads as the theory-first counterpoint to the KIVI/KVQuant line: where those papers derive per-channel and per-token recipes from measured cache statistics, TurboQuant removes the data from the problem entirely — a random rotation makes the coordinate distribution known in advance, so the optimal codebook is a mathematical constant rather than a calibration artifact. The most instructive detail is the mse/prod crossover in Figure 3: the unbiased two-stage variant wins at low bit-widths where bias dominates the error, while plain MSE-optimal quantization pulls ahead beyond roughly 3 bits — a clean empirical picture of the trade the two theorems formalize. The work builds directly on the authors’ own QJL transform (Zandieh, Daliri, Han, 2024) and sits beside PolarQuant from the same group; the ICLR 2026 camera-ready on OpenReview extends the arXiv v1 reviewed here.
KIVI·KVQuant가 관측된 cache 통계에서 레시피를 도출하는 노선이라면, TurboQuant는 rotation으로 데이터 자체를 문제에서 제거하는 노선이다 — distortion 하한 대비 ≈2.7× 갭이라는 “남은 여지”를 분야 전체에 명시해 준 것이 이 논문의 가장 큰 선물이다.
Linked reviews
KIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache
Asymmetric KV quantization derived from element-distribution analysis — per-channel keys, per-token values — pushes the KV cache to tuning-free 2 bits, cutting peak memory 2.6x and lifting serving throughput up to 3.47x.
kv-cacheKVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Pre-RoPE per-channel key and per-token value quantization with sensitivity-weighted non-uniform datatypes plus ~1% fp16 outliers pushes the KV cache to 3 bits at under 0.1 perplexity cost, reaching million-token contexts on a single A100.
kv-cacheH2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs
A small set of "heavy-hitter" tokens dominates attention mass, so greedily evicting low-accumulated-attention entries keeps only ~20% of the KV cache while matching full-cache accuracy and raising serving throughput up to 29x.
kv-cacheDuoAttention: Efficient Long-Context Inference with Retrieval and Streaming Heads
Learns which attention heads actually retrieve from long context and gives only those a full KV cache, cutting long-context memory up to 2.55x with retrieval accuracy intact.
attentionAVQ-Attention: Adaptive Vector-Quantized Attention
Vector-quantized attention cuts quadratic attention cost by clustering keys into a fixed codebook, but spends the same resolution everywhere regardless of where attention mass actually concentrates; AVQ-Attention adaptively refines codebook resolution toward high-attention regions using a hierarchical parent-child codebook, implemented as custom Triton kernels fused directly into Flash-Attention-style tiling, reaching up to 127x over Flash Attention at 64k tokens.
quantizationExTernD: Expanded-Rank Ternary Decomposition — Ternary LLM PTQ with Accuracy Approaching Any Quantization Level
ExTernD factors each LLM weight matrix into two ternary (-1/0/+1) matrices plus a real scale vector, deliberately expanding the inner rank past full rank so later components provably, monotonically correct the residual error of earlier ones, turning ternary quantization''s usual fixed-plane accuracy ceiling into a continuous dial that matches Q4_K accuracy at 5.2-5.5 effective bits per weight.
kv-cacheFlashDrive: Algorithm-System Co-Design for Real-Time VLA Autonomous-Driving Inference
Algorithm-system co-design cuts real-time reasoning-VLA latency 4.7x by matching a distinct lightweight fix to each of four structurally different inference bottlenecks.
quantizationGaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
GaugeQuant learns the quantization-optimal rotation basis during training itself -- via a LogSumExp outlier-suppression proxy loss with a stop-gradient, constrained to SO(d) by a Cayley transform -- rather than searching for it post-hoc on a frozen model with calibration data, cutting LLaMA-2 7B W4A16 perplexity degradation by 51% with no calibration set and negligible training overhead.
kv-cacheA JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation
Treating the KV cache as a single three-axis tensor rather than a stack of matrices, JoLT compresses only the token and feature axes with a partial Tucker decomposition, restores the discarded energy with a low-bit rotated residual, and lets one shared optimizer decide how to split the byte budget between rank and residual bits and between keys and values -- reaching a near-lossless 2-3x compression zone on both a grouped-query and a multi-head attention model.