There is a particular kind of engineering result that feels like getting something for nothing. You change no hardware, you buy no new GPUs, you do not degrade the quality of your output by even one token, and yet the same machines suddenly produce answers twice as fast.
That is what speculative decoding has done for the MindRouter cluster over the past week. This post explains what it is, how it works, where the idea came from, and what it has actually delivered on our own AI infrastructure investments.
The Machine That Waits
To understand why this trick works, you have to understand a slightly embarrassing fact about how language models generate text.
A model produces one token at a time. A token is roughly a word fragment, so "understanding" might be two or three tokens. To produce a single token, the GPU must read the model's entire set of parameters out of its memory. Every layer, every parameter, all of it.
For a 120 billion parameter model, that is well over a hundred gigabytes of data moved across the memory bus. To produce one word fragment. Then it does the whole thing again for the next one. Again and again for each new token in the output.
Here is the uncomfortable part: the actual arithmetic involved is trivial. An NVIDIA H200 can perform roughly a thousand trillion operations per second. Generating one token needs a vanishingly small fraction of that. The chip is not thinking hard. It is waiting for groceries to arrive.
Engineers call this being memory bandwidth bound rather than compute bound. The practical consequence is that when a single user sends a single request, a modern datacenter GPU costing tens of thousands of dollars runs at a small percentage of its theoretical capability. It is a delivery truck making a separate round trip for every item on the shopping list.
That inefficiency is an opening to be exploited.
The Trick
Since reading the weights (model parameters) is the expensive part, and since reading them once lets you do a great deal of math essentially for free, the question becomes obvious: could we produce more than one token per trip?
The answer is yes, and it works like this:
- Something fast and cheap guesses the next several tokens. Call them draft tokens.
- The big model then checks all of those guesses at once, in a single forward pass.
- Guesses that match what the big model would have produced on its own are accepted and kept.
- At the first guess that does not match, everything from that point on is discarded, and generation continues normally from there.
The magic is entirely in step two. Verifying eight proposed tokens costs almost exactly the same as generating one token, because the expensive part, reading all those weights out of memory, happens exactly once either way. You are filling the truck instead of sending it out with a single item.
If the guesser is good, you collect several tokens for the price of one. If the guesser is wrong, you throw the bad guesses away and you are no worse off than you started, minus a small amount of overhead.
That asymmetry is the whole game: Correct guesses pay enormously. Incorrect guesses cost very little.
Your CPU Has Been Doing This Since the 1960s
If this pattern sounds familiar to anyone who has studied computer architecture, it should. The processor in the machine you are reading this on has been running the identical strategy for decades.
When a CPU encounters a branch in a program, something like "if the user is logged in, do this, otherwise do that," it faces a dilemma. Determining the answer might take many cycles, and the pipeline would sit idle the whole time.
So the CPU does not wait. It predicts which way the branch will go, and it charges ahead executing instructions on that assumption. If the prediction was right, the work is already done and the results are committed. If it was wrong, the speculative work is thrown away and the pipeline restarts from the correct path.
This is called speculative execution, and it is one of the foundational ideas in modern processor design. Branch prediction goes back to the IBM Stretch in 1961. By the time the Intel Pentium Pro shipped in 1995, aggressive speculation was mainstream, and it has been in essentially every performance processor since.
Modern branch predictors are startlingly good, frequently exceeding 95%. That prediction accuracy is why your CPU feels fast.
Speculative decoding in LLMs is the same insight, applied to a machine learning workload roughly sixty years later. Guess ahead. Verify cheaply. Keep what was right, discard what was wrong.
Who Actually Invented This?
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit at Google published work on blockwise parallel decoding in 2018. They attached extra prediction heads to a model so it could propose several tokens at once, then verify them. Shazeer and Uszkoreit were both co-authors of the original famous Attention is All You Need paper, so this was very much a case of the people who built the engine noticing where it idled.
Speculative decoding in its modern form appeared essentially twice, independently, within a few months of each other.
- Yaniv Leviathan, Matan Kalman, and Yossi Matias at Google Research published "Fast Inference from Transformers via Speculative Decoding" in November 2022, which went on to appear at ICML 2023.
- Charlie Chen, Sebastian Borgeaud, and colleagues at DeepMind published "Accelerating Large Language Model Decoding with Speculative Sampling" in February 2023.
Both groups landed on the same core insight, and both proved the critical property: done correctly, this technique does not change the model's output distribution at all!
Multi token prediction (MTP), the specific variant that most of MindRouter now runs, came from Meta AI in April 2024, in a paper by Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve titled "Better and Faster Large Language Models via Multi-token Prediction." Their argument was that training a model to predict several tokens ahead does not merely make it faster, it can actually make it better, because anticipating further into the future forces the model to plan rather than simply react.
DeepSeek-V3, released in December 2024, put that idea in front of everyone by shipping multi token prediction in a frontier open weight model. After that, adoption moved quickly.
Two Ways To Guess Ahead
In practice there are two distinct mechanisms, and we run both on our MindRouter cluster.
The Separate Drafter
Pair the large model with a much smaller companion model trained on similar data. The small model is quick, so it can propose several tokens rapidly, and the large model verifies them in one pass.
The tradeoff is that you have to load and serve two models on the same GPU, and you need a drafter that genuinely thinks like its larger sibling. A poorly matched drafter guesses wrong constantly, and you spend effort on work you throw away.
This is what our gemma-4-31b deployments use, paired with a purpose built assistant model from Google.
The Built-in Head (MTP)
The more elegant approach is to train the ability directly into the model. Multi token prediction, or MTP, adds a small number of extra layers whose job is to predict not just the next token but the one after that, and the one after that.
The advantages are considerable. There is no second model to load, no matching problem to solve, and the prediction head has access to the large model's own internal state, which makes it a far better guesser than any external model of comparable size. The cost in memory is minimal.
Most of our MindRouter GPU fleet runs this style, including the qwen3.5 and qwen3.6 families and NVIDIA's Nemotron-3-Super.
Why This Is Genuinely Free
Most performance optimizations in machine learning are really trade-offs.
Quantization, for instance, shrinks a model by storing its numbers at lower precision. It is often an excellent trade, but it is a trade: you accept some quality loss for speed and memory. The same is true of pruning, distillation, and aggressive caching.
Speculative decoding is not a trade. The verification step uses an acceptance rule specifically designed so that the sequence of tokens you end up with follows exactly the same probability distribution as if the large model had generated every token itself, one at a time, with no speculation at all.
Not approximately the same. Exactly the same.
You are not getting a cheaper answer faster. You are getting the same answer sooner. The only cost is a small amount of computational overhead on guesses that get rejected, which is why the technique is worth using only when the guesser is right reasonably often.
What We Actually See On Our Own Hardware
Enough theory. Here is what this delivers on the MindRouter cluster. Every model below runs on NVIDIA H200 GPUs under vLLM 0.25.1.
The "tokens guessed" column is how many tokens we let the model propose before verifying. The "accepted" column is how often those guesses turn out to be correct. "Speed gain" is measured against the identical model on the identical GPU with speculation turned off.
| Model | How it guesses | Tokens guessed | Guesses accepted | Speed gain |
|---|---|---|---|---|
| qwen3.6-27b | Built-in MTP head | 3 | 73 to 85% | about 2.3x |
| qwen3.6-35b | Built-in MTP head | 3 | high | about 2.3x |
| gemma-4-31b | Separate drafter model | 4 | high | about 2.0x |
| qwen3.5-122b | Built-in MTP head | 2 | 83% then 71% | about 2.0x |
| Nemotron-3-Super-120b | Built-in MTP head | 2 | high | about 1.7x |
| qwen3.5-35b | Built-in MTP head | 2 | 73% | about 1.6x |
Read that table again, because the summary is remarkable. Fifteen of our production model endpoints are now producing between 1.6 and 2.3 times as many tokens per second as they did last week, on exactly the same GPUs, with byte-for-byte identical output quality.
We purchased no hardware to achieve this. We degraded nothing. The gains came entirely from algorithms.
These figures are strongest when a model is serving a small number of simultaneous requests, which is the common case for interactive chat and coding assistance. When a GPU is saturated with many concurrent users, it is no longer waiting on memory, it is genuinely busy computing, and the free lunch shrinks. In our testing the technique still helps or stays neutral under heavy load, but the dramatic numbers above are single stream and low concurrency figures.
Why This Matters More Than It Sounds
It is tempting to file this under routine optimization. That undersells it.
The public conversation about AI capacity is almost entirely a conversation about hardware: how many GPUs, which generation, what the power budget looks like, how long the procurement will take. Those constraints are real, and for a university research cluster they are sharply real. GPUs are expensive, they are difficult to acquire, and they consume electricity whether or not they are doing useful work.
Speculative decoding is a reminder that hardware is only one of the two levers, and often not the faster one to pull.
A roughly two times throughput improvement from an algorithmic change is, in practical terms, equivalent to having doubled part of our GPU fleet. Except it arrived through a software update, it cost nothing in capital, it consumed no additional power, and it required no data center space. In an era when a single H200 represents a serious purchase and a rack of them represents a serious institutional commitment, that is not a footnote.
This pattern keeps repeating. FlashAttention rewrote attention to respect the memory hierarchy and made long contexts practical. Paged attention, the idea vLLM was built around, borrowed virtual memory paging from operating systems and dramatically improved how many users a single GPU can serve. Quantization made large models fit on hardware that had no business running them. Each of these advances extended the useful life and effective capacity of hardware that was already installed and paid for.
The models keep getting larger. The hardware improves on its own schedule, governed by physics and manufacturing and supply chains. In the space between those two curves sits a great deal of clever algorithmic work, and that work is what determines whether a research group with a finite budget can run competitive models at all.
Our GPUs did not change. The mathematics we run on them did, and everyone using the cluster now waits about half as long.
That is a good trade, especially when it is not really a trade at all.
The MindRouter cluster serves our entire user community. Speculative decoding is enabled by default across supported models. If you are curious about learning more, please contact us: mindrouter@uidaho.edu