A model weight is not "a number" — it is sixteen bits split into a sign, an exponent and a mantissa, and every design decision downstream is about spending those bits differently.
Qwen3-8B stores its weights in bfloat16: one sign bit, eight exponent bits, seven mantissa bits. The exponent gives it the enormous range of fp32; the seven mantissa bits are all the precision it has. Flip the bits below and watch which ones matter.
The exact layout Qwen3-8B's files use. Sign · 8 exponent · 7 mantissa.
Two things to take away. The exponent bits move the value by factors, the mantissa bits by small increments — which is why floats are dense near zero and sparse far from it. And 16 bits is 65,536 possible values; a 4-bit weight will have 16. That is the whole compression story in one comparison.
Quantizing means choosing a small set of levels, storing only which level each weight landed on, and keeping two extra numbers per group — a scale and a zero-point — that map those level indices back to real values.
There is no such thing as storing "a 4-bit float". What is stored is an integer index q ∈ {0…15}. The real value is reconstructed as w ≈ s·(q − z), where s is the step between levels and z shifts the grid so it can cover asymmetric ranges. Drag the weight and watch the round trip.
A group of 8 weights sharing one scale and zero-point. Drag the highlighted weight; the grid is fixed by the group's min and max.
The reconstruction error is the distance to the nearest level, and it is bounded by half a step: s/2. So everything about accuracy comes down to how big s is — and s is set by the range the grid has to cover, divided by the number of levels.
Because the grid must stretch to cover the largest value in its group, a single outlier widens every step — so the other weights, which never moved, lose precision. This one fact is the seed of half the quantization literature.
Drag the outlier and watch the other seven weights. Their values are unchanged; their errors are not. Then shrink the group and watch the tax get contained.
Eight weights, 4-bit grid. The last one is an outlier you control.
Splitting the group is the fix, and it is not free: every group needs its own scale and zero-point. That trade — resolution against bookkeeping — is what a "group size" hyperparameter really is, and the next section shows what it costs in bits.
A 4-bit tensor is stored as integers packed several to a machine word, next to a separate array of scales and zero-points — which is why "4-bit" weighs more than four bits per weight, and why the group size is a storage decision as much as an accuracy one.
Memory has no 4-bit addresses. Eight 4-bit indices are packed into one 32-bit word, and the kernel unpacks them by shifting and masking. Alongside sits the metadata: one scale and one zero-point per group.
The actual bit positions, and the metadata that rides along.
This is the number nobody quotes: at group size 128 with an fp16 scale and an fp16 zero-point, a "4-bit" weight costs 4.25 bits. Shrink the group to 32 and it costs 5 bits — you bought accuracy with a quarter of your compression.
In Qwen3-8B the MLP matrices alone are two thirds of the file, the attention projections another sixth, and the embedding and LM head — usually left in bfloat16 — are a stubborn 2.3 GiB that no weight-quantization paper touches.
Every number below is read from the model's own safetensors headers. Toggle what gets quantized and watch the file shrink — and notice where it stops shrinking.
Generating one token reads every weight once, so the file size is also the per-token memory traffic — quantization is a bandwidth technique that happens to save disk, and the dequantization arithmetic it adds is free because the arithmetic units were idle anyway.
The kernel reads packed integers, unpacks and rescales them inside registers, and multiplies in bf16. Nothing is ever written back in high precision, so what you save on the read is what you save on the clock.
Bytes crossing HBM per generated token, at batch 1.
This is also the honest ceiling: with embeddings and the LM head left in bf16, 4-bit weights do not make the model 4× faster to read. The parts you refused to quantize become the floor.
Weights sit in a narrow band, but the activations that multiply them are a token × channel × value volume in which a handful of channels stand tens to hundreds of times above the rest, on every token — and that ridge, not the weights, is what the next papers in this track are about.
Every matmul in the model is y = Wx, and so far this page has only drawn W. The x is just as real: one row per token, one column per channel. Below is a measured one — Qwen3-8B, run on CPU, hooks on the modules whose input gets multiplied by quantized weights. Raise the surface and look along the token axis. The tall thing is not a bump on one token; it is a ridge.
|x| entering the matmul, for the prompt "The tower is 324 metres tall, about the same height as an 81-storey building."
Flatten the view and the loud channel is still there as a dark column — but only as dark. Colour saturates long before 30× does, so the flat view says that the channel is different without saying by how much; height is what carries the ratio. The other half of the point is the shape along the token axis: it is the same channel on every token, so the damage can be met with one number per channel instead of with more careful rounding. Switch to the weights, on the identical axis, and the surface is a plain — the loudest row barely doubles a typical one. That asymmetry is why weight-only quantization got easy and activation quantization did not.
Nothing on this page was a research idea — it was arithmetic about bits and bytes. But each fact forces a consequence, and each consequence has exactly one obvious escape, and those escapes are the papers. Step through the chain.
Each step is a claim you can check against the figures above.
Read the chain backwards and every paper in this track becomes obvious in hindsight — which is the point. A method is rarely a leap; it is usually the only move left once you have taken the physics seriously.