Generating one token at batch 1 reads every weight once and only a handful of activation vectors, so weight bytes outnumber activation bytes by orders of magnitude — which makes W4A16 the right target on device, and W8A8 the wrong one.
This is the fork in the road with the previous page. SmoothQuant aims at W8A8 because a datacentre GEMM is compute-shaped; AWQ aims at 4-bit weights and FP16 activations because a phone decoding one token at a time is bandwidth-shaped. Same observation about outliers underneath, opposite deployment, and therefore a different method. The byte accounting is on the foundation page.
Plain rounding survives INT8 and starts to break at INT4, and at INT3 it takes OPT-1.3B from perplexity 14.6 to 119 — so something has to be treated differently, and the question is what.
Twenty years of pruning says a weight's importance is its magnitude — and here that heuristic performs no better than picking channels at random.
This is the paper's most quotable experiment, and it is one you can run at layer scale in seconds. Keep a small fraction of weight channels in full precision and quantize the rest to three bits. The fraction barely matters. The criterion decides everything.
Relative output error of a real Qwen3-8B layer, group-128 weight quantization, keeping a few channels in bf16.
Protecting 1% of channels chosen by how loud their activations are removes a large part of the error. Protecting 1% chosen by weight norm removes almost none of it — it tracks the random line. The salient weights are not the big ones; they are the ones reading the big inputs, which is a statement about the tensor next door, not about W.
Every input channel of the layer, ranked by activation magnitude and by weight-column norm.
The weight curve is nearly flat: no column stands far above the others, so "keep the biggest weights" has almost nothing to select. The activation curve has a tail that leaves the frame. The two top-1% sets are almost disjoint, and the counter says by how much — that disagreement is the whole reason one criterion works and the other does not.
Keeping 1% of channels in FP16 costs nothing in bits and ruins the kernel, so the goal is to get the same protection while every weight in the tensor stays a plain 4-bit integer.
Rounding error is roughly uniform whatever the value, so scaling a weight up by s before quantizing and its input down by s afterwards leaves the absolute error the same and the relative error smaller by 1/s — as long as the group's scale Δ does not move.
And there is the catch, which is the interesting half. Δ is set by the largest weight in the group. Push one channel up far enough and it becomes the largest, Δ grows, and every other weight sharing that group gets a coarser grid. Protection is not free; it is bought from the neighbours.
96 real weights from one row of Qwen3-8B, quantized as a single group. The row is a typical one for this channel. Drag s and watch Δ.
While the scaled weight stays inside the group's existing range, Δ does not move and the protection is free: the same rounding error now stands against a value s times larger. The moment it reaches past that range — in this row it goes downward, since the weight is negative — Δ starts climbing and the grid coarsens for all 95 others. Everything AWQ does after this point is about where to stop.
s = s_Xα with a single α grid-searched to minimise ‖Q(W·diag(s))(diag(s)⁻¹X) − WX‖, so the layer itself decides how much protection its salient channels are worth.
The search space is one number, and its shape comes from the observation: saliency is set by the activation scale, so the family of scales worth trying is powers of the per-channel activation magnitude. No gradients, no reconstruction — just a grid over α and the layer's own output error as the judge.
Left: our layer, INT3-g128, scaling the loudest 1% by a fixed s. Right: their Table 2, OPT-6.7B perplexity under the same experiment.
Two different models, two different metrics, the same shape — and the same minimum at s = 2. The middle rows are the mechanism: the fraction of groups whose Δ has moved climbs with s, and the moment enough of them move, the neighbours' losses overtake the salient channels' gains. Our error-ratio column lands within a percent of theirs, which it should: that column is arithmetic, not a measurement.
Relative output error against α, with round-to-nearest and the mixed-precision oracle marked for reference.
The searched scale lands close to the mixed-precision oracle while every weight stays a 4-bit integer, which is the trade the paper is making. Ours sits slightly above the oracle because this reproduction stops at the scale search — the paper also clips the weight range, which buys back part of the remaining gap.
INT3 and INT4 at group 128 on OPT, LLaMA, Llama-2, Mistral, Vicuna and the VILA vision-language models, plus a deployment story on an RTX 4090 and a Jetson Orin through their TinyChat runtime.
At INT3-g128 on OPT-6.7B, round-to-nearest gives perplexity 23.54, keeping 1% in FP16 gives 11.39, and AWQ gives 11.39 with a uniform data type.
Their Table 3 — WikiText perplexity at INT3, group 128, across OPT sizes. Lower is better.
The line to watch is the gap between the mixed-precision oracle and AWQ: they sit on top of each other at every size, while round-to-nearest is off the top of the chart at three of five. The 2.7B column is the reminder that these collapses are not monotone in model size — RTN is worse at 2.7B than at 13B.
GPTQ reaches similar accuracy by solving a second-order reconstruction against a calibration set; AWQ measures one statistic per channel, which is why it needs less calibration data and generalises further from it.
The relationship with the previous page is closer than it looks. Both methods apply the same object — an offline per-input-channel scale s, cancelled by its inverse on the other operand, with a single exponent α tuning how far to go. SmoothQuant spends it making X quantizable for an INT8 kernel; AWQ spends it making W's salient columns survive four bits. One observation, one mechanism, two destinations.
The lasting idea is not the scale search — it is that a weight's importance is a property of the activations it meets, which is why every serious weight-only method since is activation-aware.
The search space is deliberately tiny — one α over a fixed family — so it cannot express a scale that is not a power of the activation magnitude, and it optimises each layer's output error rather than the model's.
Our sweep shows the cost of that choice at the two ends: α too small protects nothing, α too large hands the salient channels a grid the neighbours pay for. The minimum is real but shallow, which is a hint that the family, not the search, is the binding constraint.
A per-channel scale is the diagonal case of a more general change of basis, and the methods that followed — Hadamard rotations, learned transforms — are what happens when you stop insisting the matrix be diagonal.
Paper: arXiv 2306.00978
(MLSys 2024, best paper). Code and the TinyChat runtime:
mit-han-lab/llm-awq. Every
probe number here comes from one CPU run over Qwen3-8B — 8
sequences × 256 tokens of WikiText-2, hooks on the inputs of
layers.23.self_attn.q_proj and layers.23.mlp.down_proj, quantizers
written from the paper's equations, group size 128. Numbers badged
paper are the authors'.