An autoregressive LLM must push its entire stack of weights through the GPU to produce a single token, and cannot start the next one until this one exists — so latency is token count × one pass, and that pass is spent mostly moving weights, not computing.
That last part is the opening the whole field walks through. At batch size 1 the GPU reads ~16 GB of weights to do a handful of multiplications per weight; the arithmetic units idle. If you could check several candidate tokens in one pass instead of producing one, the extra ones would come nearly free.
Eight tokens of ordinary decoding from Qwen3-8B, drawn to scale in time. Every box is one full forward pass through the model.
We measured one pass at 27.3 ms on our A100. Nothing in this paper makes that box narrower — the entire method is about getting more than one token out of a box like it.
Cut generation latency without changing a single output token — a constraint that turned urgent once reasoning models began emitting thousands of chain-of-thought tokens before answering.
"Without changing a single output token" is the strict part. Sampling a smaller model is fast and wrong; distillation changes the model. Speculative decoding is the one family that promises the identical distribution: guess ahead, then let the real model audit the guesses in a single pass and discard whatever it disagrees with.
Every strong speculative method before this one drafts with a small autoregressive model, so producing γ guesses costs γ sequential passes — which forces the drafter to be tiny (EAGLE-3 uses one transformer layer), which caps how many guesses survive, which caps speedup at 2–3×.
It is a trap with three walls. A deeper drafter guesses better but costs linearly more time. Longer guesses raise the ceiling but cost linearly more time. A shallower drafter is cheap but its guesses stop surviving. The measurement below is that trap drawn to scale.
Same time axis as the clock above. Top: what autoregressive drafting has to do. Bottom: what a block-diffusion drafter does.
An autoregressive drafter pays per token, so its cost is a staircase; a diffusion drafter fills the whole block in one pass, so its cost is a single step. The consequence is not "somewhat cheaper" — it is that guess length stops being a cost at all.
Measured in steady-state decoding, 50 timed repetitions after 10 warmups (σ < 0.1 ms), log scale. The dashed line is γ × one target pass — what serial drafting with a target-sized model would cost. EAGLE-3's real 1-layer drafter sits far below that line but keeps its slope; the diffusion drafter's line is flat.
Break the 2–3× ceiling by making drafting cost independent of guess length, then spend the freed budget on drafter depth and on conditioning it with the target's own hidden states — while verification keeps the output provably unchanged.
Each cycle costs one drafter pass plus one target pass — a fixed ~35 ms on our A100, no matter how many tokens are involved — and yields τ committed tokens, so every idea in the paper is an attempt to raise τ without widening that box.
τ (tau) is the number this method lives on, and it is easy to state and hard to feel: the average number of tokens that survive one cycle. The figure makes it physical. Both strips cover the same 300 ms of wall-clock. The top strip is ordinary decoding. The bottom is DFlash — the same fixed cycle repeated, τ tokens falling out of each. Drag τ and watch the bottom strip fill.
Drafting 5.3 ms + verification 30.2 ms = a 35.5 ms cycle. The cycle never changes width; only how many tokens come out of it does.
Two things this makes obvious. τ = 1 is worse than not speculating at all — you paid for a drafter and still emit one token per cycle. And returns are not linear: 3.2 → 6.4 nearly doubles throughput, while 12 → 16 barely moves it, because the cycle's fixed cost has stopped being the bottleneck.
τ is an average, and averages hide their shape. This is every cycle of a 128-problem run, bucketed by how many tokens it committed.
The spike at the right edge is the whole block surviving; the mass at 1–3 is the drafter being wrong early. τ is the mean of this distribution, so raising it means shifting mass rightward — not making a few lucky cycles luckier.
So the drafter must produce 15 tokens in one pass. An autoregressive model structurally cannot: token 3 is a function of token 2. A block-diffusion model can, because it was trained on a different task — fill in masked positions given everything around them, all at once, the way BERT-style models do, but over a block of future tokens.
One anchor token (the last one the target committed) followed by 15 mask slots. One pass. Bidirectional attention inside the block — position 9 may look at position 14 while both are still masks.
This is also why the drafter can afford depth: its cost is one pass regardless of block size, so the paper spends the savings on five layers instead of one — and five layers of drafting still cost a fifth of a single target pass.
Here is the step that makes the output provably unchanged, and it is simpler than it sounds. The target model runs once over the whole block. That single pass produces, at every position, the token the target itself would have generated. Compare position by position; keep the longest prefix that matches; at the first mismatch take the target's own token instead and throw the rest away.
An actual cycle from a GSM8K solve on our A100 — the drafter's 15 guesses, and what the target's single verification pass did to them.
The bonus token is the quiet trick: the verification pass already computed the target's own next token at the mismatch position, so it comes along free. That is why a cycle accepting zero guesses still emits one token, and why τ ≥ 1 always.
Step through a whole solve. Watch where drafts die — and where the drafter runs 16-for-16.
Step until you find a cycle that accepts everything, and one that accepts nothing. What kind of text is the drafter good at — and what does that say about what it learned?
A five-layer drafter guessing from scratch is not good enough — the paper's own ablation puts it at 2.8–3.7×. What makes DFlash's guesses survive is that the drafter never guesses from scratch: it is fed the target model's internal representation of the context, which already encodes information about tokens the target has not emitted yet. The mechanism is where that signal is injected.
Two conditioning strategies, from the paper's Table 9. Toggle between them.
That is the design. The part that only shows up when you follow the loop is where those features come from on cycle number two — and the answer closes the circle neatly: from the verification pass that just ran.
Four beats. Follow what is computed, what is reused, and what is thrown away.
Diffusion drafting removes the cost of guessing far ahead, which makes a deep drafter affordable; injecting the target's features into every draft layer's K/V keeps that depth from being wasted — either half alone lands near 3×.
The paper's ablations are the cleanest statement of this, because each one removes exactly one of the two halves.
GSM8K unless noted. Left bar is acceptance length τ, right bar is end-to-end speedup.
Read the top three rows as the argument: a bare diffusion drafter is ordinary; adding the target's features at the input helps a little; moving that same signal into every layer's K/V is what makes it work. The bottom rows are the depth trade-off — τ keeps climbing with layers, but at 8 layers drafting time finally starts eating the gain, so the paper ships 5.
Two metrics on eight benchmarks across math, code and chat — acceptance length τ and end-to-end speedup over plain autoregressive decoding — for Qwen3 and LLaMA-3.1 targets, in both a research backend (Transformers) and a production one (SGLang, where concurrency matters).
Since the output distribution is unchanged by construction, accuracy is not the question; speed is. The probe figures on this page are our own runs of the same protocol on one A100 — there to put real tokens and real milliseconds into the diagrams, not to re-prove the paper.
On Qwen3-8B with greedy decoding, DFlash runs 4.7–6.1× faster than autoregressive decoding on math and code and about 2.4× faster than EAGLE-3 at a matched drafting budget — with chat (MT-Bench) the notable weak spot at 2.75×.
Greedy decoding, Transformers backend, H200. Parenthesised numbers are the drafting budget: EAGLE-3 with a 16-token tree matches DFlash's block size; the 60-token tree is EAGLE-3's own preferred setting.
The gap tracks acceptance length: where DFlash gets τ ≈ 7–8 (structured math and code, where the next dozen tokens are predictable) it runs away; on open-ended chat τ falls to 4.2 and so does the speedup. Note also that DFlash at a 16-token budget beats EAGLE-3 at 60 — a bigger search tree costs verification time that acceptance does not repay.
Against plain autoregressive decoding as the 1.0× floor and EAGLE-3 as the state of the art; the other diffusion-drafter papers (DiffuSpec, SpecDiff-2) are argued against rather than benchmarked, because they had no public implementation — they buy acceptance with 7B drafters and pay it back in drafting latency.
The comparison that matters conceptually is PARD, which trains small autoregressive models to imitate parallel generation. Same goal, opposite direction: PARD keeps the drafter architecture and changes its behaviour; DFlash changes the architecture so that parallelism is structural.
It moves speculative decoding from "worth about 2×" to "worth about 5×" on the workloads people actually pay for — long reasoning traces and code — and it reframes diffusion language models as specialist drafters rather than failed generalists.
That reframing is the part with legs. Diffusion LLMs have struggled to match autoregressive quality end to end; as drafters they never have to, because verification supplies the quality guarantee. A weak model with a strong auditor is a different product than a weak model alone.
The drafter must be trained per target model, open-ended chat gets roughly half the speedup that math and code get, and the whole advantage erodes as serving concurrency rises — because a busy GPU is no longer waiting on memory.
Qwen3-8B. Same method, same model, only more simultaneous requests.
The mechanism is Q1 run backwards. Speculation converts idle arithmetic into free verification; with 32 requests in flight the arithmetic units are no longer idle, so there is less to convert. Speculative decoding is a latency-regime optimisation, and this chart is where that shows.
The paper's own loose end is adaptive block-size scheduling — its ablation shows a drafter trained at block 16 degrades gracefully when run at block 8, so a server could shrink the block under load; the authors leave the scheduler unbuilt.
We probed that generalisation on our A100: running the released block-16 checkpoint at block 8 on MATH-500 gives τ 5.66 against 7.78 at block 16 — a real cost, but a graceful one, matching the pattern the paper reports for its own ablation model. A follow-up, DFlash 2, has since appeared.
The authors ship code and drafter checkpoints for Qwen3, LLaMA-3.1 and more (z-lab/dflash), with SGLang and vLLM integrations; our probe runs used the public checkpoint unmodified.