Generating text from an LLM is dominated by reading its weights — at small batch the GPU is a memory pump — so shrinking the weights shrinks both the memory bill and, with the right kernel, the runtime.
An LLM's weights are fp16 numbers, two bytes each: 326 GB for GPT-3-scale. Each generated token re-reads essentially all of them once. That is why a weights-only trick can matter at all — and why this paper cares about nothing except making the weights small.
Replace every fp16 weight of a 100B+-parameter model with one of just 16 (4-bit) or 8 (3-bit) values, in one shot, without retraining — and lose almost no accuracy.
Quantization means choosing those few levels and snapping every weight onto them — turning a continuous number line into a lattice. The obvious way is RTN — round to nearest: snap each weight, independently, to its closest level.
The whole of "quantization", in one picture — per row of the matrix, a min–max grid is fixed and every weight moves to its nearest level.
The rounding error of one weight is its distance to the nearest level. RTN stops here: 16 independent little errors per row, no questions asked. The whole paper is about what to do with those errors instead of ignoring them.
Not a schematic: 2.4M real weights from one matrix of the model we quantized, with the actual 4-bit grid that row 0 was assigned.
Weights pile up near zero, so the levels near the middle do most of the work and the outer ones are nearly idle — one reason low-bit grids hurt, and the reason later papers (AWQ, SmoothQuant) obsess over outliers.
RTN survives 4-bit but collapses at 3-bit (perplexity 27.7 → 1,300 on our probe), while the accurate second-order alternative (OBQ) tops out near 100M parameters — three orders of magnitude short of GPT scale.
The reason RTN breaks is not that individual weights move far. It is what a layer computes: y = Wx. What matters is never how much W changed, but how much the output changed — ‖ΔW·x‖ on the inputs x the model actually sees. That reframing is the door the whole paper walks through.
Keep OBQ's error-compensating accuracy but make it run on OPT-175B in ~4 GPU-hours — a ~1000× scaling of the same idea, with 3–4-bit weights and near-zero perplexity loss.
GPTQ walks the network one weight matrix at a time, quantizing each column left to right while the not-yet-quantized columns absorb every rounding error — guided only by the input co-firing statistics, H = 2XXᵀ.
Before stepping through the algorithm, place it. GPTQ never sees "the model" — it walks through the network one weight matrix at a time. The picture below drills from the whole model down to the single matrix the demo operates on.
OPT-125M as the running example — the model our A100 runs quantize. The orange-outlined piece is the one being zoomed into.
Take the smallest layer that can show it: one output, two inputs — y = w₁x₁ + w₂x₂ — and let the two inputs be strongly correlated: when x₁ fires, x₂ fires nearly alike. Round w₁ to its grid and every sample's output picks up an error (q₁−w₁)·x₁. But w₂ is still free, and x₂ can imitate x₁ — so shift w₂ by the right amount and it cancels most of that error without touching the rounded q₁. That is error compensation, the entire idea. Everything else in GPTQ is bookkeeping to do this optimally across 768 correlated inputs at once: H = 2XXᵀ records who co-fires with whom, and the paper's Eq. 2 is the closed form of the slider below.
The same tiny layer drawn twice. Left: round w₁ and do nothing. Right: round w₁ the same way, then shift w₂. The rounded value is identical — only the output error differs.
Numbers from the demo below (ρ ≈ 0.93): rounding error δ = −0.22 on w₁ either way; letting w₂ move by +0.205 cuts the output RMS error from 0.225 to 0.087. Nothing about w₁ was "fixed" — the error was re-routed through a correlated input.
90 samples of correlated inputs (corr ≈ 0.93). ① rounds w₁ onto a coarse grid; then drag Δw₂ — or take the optimal step — watching the output error, not the weights.
The residual you cannot remove is the part of x₁ that x₂ fails to mimic — with correlation ρ, a fraction √(1−ρ²) of the error survives. Uncorrelated inputs ⇒ no compensation possible. This is why the Hessian — pure co-firing statistics, no labels, no gradients — is the only thing GPTQ needs to know about the data.
After ②, w₂ still has to be rounded eventually — with one fewer helper left. Play the game forward: what happens to the LAST column's error, and who absorbs it?
Where did the "optimal step" in the demo come from? Not from a neural network — from four lines of algebra. Walk them; each line is one idea.
One more mechanical step in the paper: repeatedly deleting row q from H⁻¹ (Eq. 3) is numerically fragile at scale, and it turns out all the needed rows can be precomputed at once as a Cholesky factor — that is the only reason "Cholesky" appears in the algorithm below.
Left: the 16×16 H of the toy layer below (same synthetic inputs). Right: the Hessian diagonal of a real OPT-125M layer from our probe run.
The band structure of the toy H is why compensation in the stepper smears into neighboring columns. The real diagonal spans five orders of magnitude: some inputs are enormously more expensive to get wrong — H knows which.
So much for where H comes from. The question that actually decides the algorithm is how it gets used, thousands of times, at run time. The answer is smaller than the derivation: H is inverted and Cholesky-factored once, and after that each column just reads one row of numbers off the factor and adds a multiple of its own error to the columns on its right.
Four weights of one row, a 4-level grid, and the real U = chol(H⁻¹) of this toy layer. Step through: watch a column round, spill its error to the right, and then watch the NEXT column get quantized at its new value — not its original one.
Three things to notice. The only thing taken from H is one row of U per column — everything else was precomputed before the loop started. The spill is strictly rightward, so a locked column never moves again. And each column is quantized at the value it has after receiving everyone else's error, which is why the same weights end up on different grid levels than plain rounding would pick.
Now scale the two-weight move up: 16 columns instead of 2, every column both receiving compensation and later needing it, and the optimal shifts read off the Cholesky factor of H⁻¹. This is the paper's Algorithm 1, verbatim.
A toy linear layer: 8 output rows × 16 input columns, correlated synthetic inputs X (so the Hessian H = 2XXT is non-trivial), per-row min–max grid. Exactly Algorithm 1 of the paper, Cholesky form included.
final objective — GPTQ: – · RTN: – · ratio –
The blue curve (GPTQ) and orange curve (RTN) quantize the same columns in the same order on the same grid — the ONLY difference is the compensation update after each column (Eq. 2). At 2 bits the gap is dramatic; try 4 bits to see why coarse grids are where compensation earns its keep. Reset re-draws W and X from a new seed.
Step to the last few columns and watch ΔW: why does compensation get weaker near the end — and how does that explain the paper's observation that greedy ordering barely matters?
The differentiator is not compensation itself (that is OBQ) but three scaling moves: one shared column order for all rows, lazily batched updates, and a one-shot Cholesky in place of thousands of fragile inverse downdates.
The compensation idea is from 2022's OBQ — which already did it optimally, but kept separate Hessian bookkeeping per row and a greedy per-row order: O(d_row·d_col³), an hour for 25M parameters, hopeless at 175B. GPTQ's three moves, each forced by one observation: (1) order barely matters → use one shared column order for all rows, so the Hessian bookkeeping is shared — three orders of magnitude less of it (its fine print is Q11); (2) column-at-a-time global updates starve the GPU → lazily batch them in 128-column blocks; (3) thousands of repeated downdates of H⁻¹ turn numerically toxic at billion scale → precompute every needed row up front as one Cholesky factor, plus 1% damping. Result: OPT-175B in ~4 GPU-hours — and our OPT-125M in 30 seconds.
Mean |W − Ŵ| per input column after 4-bit quantization, GPTQ vs plain rounding on the identical grid.
Counter-intuitive until you remember Q3: GPTQ deliberately parks weights further from their original values, because the quantity being minimized is the layer's OUTPUT error, not the weight error. A method judged by ‖W−Ŵ‖ would call this worse; the model disagrees.
The paper's yardstick is perplexity on WikiText2/PTB/C4 (plus zero-shot tasks) across the OPT and BLOOM families, against an FP16 baseline and RTN; our GPU runs are probes of the same protocol at OPT-125M scale — there to pull real numbers and tensors into this page, not to re-prove the paper.
Every measured figure below carries its artifact name; the runner was verified against the authors' unmodified code (0.004% ppl difference), and calibration follows the paper exactly — 128 random 2048-token C4 excerpts the model has never been tuned on.
Across the whole OPT family, 4-bit GPTQ tracks the FP16 baseline within a fraction of a perplexity point where RTN drifts or breaks, and at 3-bit RTN collapses by orders of magnitude while GPTQ stays usable — with the gap closing as models grow.
Lower is better. Both axes log — perplexity has to be, because 3-bit RTN leaves the chart entirely. Hover any point for the number.
Two readings. Vertically: at every size, compensation beats rounding — and the 3-bit pair is a different universe, not a different score. Horizontally: bigger models are EASIER to quantize (GPTQ-3bit at 175B loses 0.34 ppl; at 125M it loses 26). The one bump against the trend, OPT-66B, is the paper's own footnote: that checkpoint has many dead units in early layers.
Quality cost on the left, the payoff on the right.
The speedup is not from faster math — the kernel dequantizes on the fly and the arithmetic stays fp16. It comes from reading 3 bits per weight instead of 16, which is why the cheaper card (A6000, less bandwidth to spare) gains more than the A100. Quantizing all 175B parameters took ~4 GPU-hours, once.
At GPT scale the only baseline that even runs is RTN — GPTQ beats it by whole regimes (3-bit: 54.7 vs 1,277 on our probe); at small scale it ties the accurate-but-slow school (AdaRound, BRECQ, OBQ) while finishing in seconds instead of hours.
3-bit OPT-175B fits on ONE A100 (down from five), the dequant kernel turns saved bytes into 3.25–4.5× faster generation, and the recipe seeded the open-source quantization ecosystem that made local LLMs ordinary.
The speedup is memory-traffic only — no low-bit arithmetic, so compute-bound batch inference gains nothing — activations stay fp16, small models still lose real accuracy at 3-bit, and the "any column order works" licence holds for wide layers, not narrow ones.
That last point is worth pausing on, because the paper hedges it in one clause: order is free "in particular on large, heavily-parametrized layers". On a narrow layer the pool of still-free columns runs out fast, so a bad order leaves late columns with no one to absorb their error. It is the same intuition as the two-weight demo, played to the end: the last column has no helper at all.
The paper points to activation quantization and finer grouping; history answered with activation-outlier methods — SmoothQuant and AWQ, the next papers in this series — and later with 2-bit vector quantizers built on the same Hessian machinery.
Authors: full code with the 3-bit CUDA kernel (IST-DASLab/gptq). Us: every number on this page links to a run artifact (JSON + ledger), produced by a runner verified against that code to 0.004%.