Qwen 3.6 27B: MTP Benchmark on a 24 GB GPU

Can Multi-Token Prediction double inference speed on a 24 GB GPU?

If you run LLMs locally, you already know one of the main problems: generation speed. Some tasks feel fine, others quickly become painful. Qwen 3.6 27B is no exception.

This is where Multi-Token Prediction, or MTP, comes in. In the right conditions, it can speed up inference quite a bit by letting the model draft multiple future tokens ahead of time instead of decoding strictly one by one.

The catch is that it does not help equally on every workload. Code generation tends to benefit a lot. Reasoning still improves, but less. Short answers often do not gain enough to justify the extra VRAM.

To see how much difference it really makes, I benchmarked Qwen 3.6 27B with and without MTP on an RTX 3090 24 GB setup, using llama.cpp. I tested 3 different use cases and compared draft depths from 1 to 4.

Before looking at the numbers, here is the test setup.

Setup

If you are running Qwen 3.6 for the first time, start with the step-by-step setup guide with Podman and llama.cpp.

The goal here was simple: benchmark Qwen 3.6 27B and its MTP variant on the same workstation, under controlled conditions, using the same runtime and the same prompt set.

Hardware:

• CPU: AMD Ryzen 9 5950X (16 cores)
• RAM: 64 GB DDR4
• GPU 1: NVIDIA RTX 3090 (24 GB VRAM) - primary inference device - power capped at 280W
• GPU 2: NVIDIA GTX 1070 Ti (8 GB VRAM) - not used in this test

Software:

• OS: Fedora 44 Workstation
• podman 5.8.4
• NVIDIA driver version: 580.173.02 (kept to preserve support for the GTX 1070 Ti, since newer drivers drop legacy architecture support)
• NVIDIA Container Toolkit 1.19.0
• Container image: ghcr.io/ggml-org/llama.cpp:full-cuda (CUDA 12 - based image)
• llama.cpp version: 9917 (4a7ee3126)
• Models:
  • Qwen 3.6 27B - Unsloth GGUF UD-Q4_K_XL quantization
  • Qwen 3.6 27B MTP - Unsloth GGUF UD-Q4_K_XL quantization

The system was otherwise idle when running the tests. No other heavy processes were active, and the secondary GPU was left unused.

Presets

To make the benchmark more useful, I did not test only one configuration. I used 4 practical presets based on my previous Qwen 3.6 setup, each aimed at a different kind of workload.

• thinking-coder - f16 KV cache, 49152 context, temp 0.6. Intended for coding tasks with reasoning enabled.
• thinking-coder-max-ctx - q8_0 KV cache, 98304 context, temp 0.6. Same general profile, but with a much larger context window at the cost of cache precision.
• thinking-general - f16 KV cache, 49152 context, temp 1.0. A general-purpose reasoning preset with higher sampling temperature.
• instruct - f16 KV cache, 49152 context, temp 0.7, reasoning disabled. Meant for simpler, direct answers.

Here is the thinking-coder preset used as the reference example:

[qwen36-thinking-coder]
model = /models/Qwen3.6/Qwen3.6-27B-UD-Q4_K_XL.gguf
cache-type-k = f16
cache-type-v = f16
ctx-size = 49152
temp = 0.6
min-p = 0.0
top-p = 0.95
top-k = 20
presence-penalty = 0.0
repeat-penalty = 1.0
chat-template-kwargs = {"preserve_thinking": true}

The presets share the same core sampling defaults: min-p 0.0, top-k 20, presence-penalty 0.0, and repeat-penalty 1.0. What changes is the context size, KV cache type, temperature, and whether reasoning is enabled.

The thinking-general preset is basically thinking-coder with a higher temperature. The instruct preset is different enough that it should be treated as its own family, not compared directly with the thinking presets.

To enable MTP, I switched to the MTP model file and added two options:

model = /models/Qwen3.6/Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf
spec-type = draft-mtp
spec-draft-n-max = 1

spec-type = draft-mtp turns on speculative decoding. spec-draft-n-max controls the draft depth, from 1 to 4 in this benchmark.

That gives a full test matrix of 20 configurations: 4 presets x 5 MTP profiles (no-mtp, mtp-1, mtp-2, mtp-3 and mtp-4)

Compared to my non-MTP setup, I had to reduce the context sizes to make room for the larger MTP model and its extra VRAM overhead. In practice, that meant going from 65536 to 49152 with f16 KV cache, and from 131072 to 98304 with q8_0 KV cache.

Prompts

To make the benchmark useful, I picked 3 prompts that stress MTP in different ways.

• Coding - Ansible playbook to deploy an Nginx reverse proxy on AlmaLinux 10. About 100 input tokens, with a 12000-token output cap. This is the kind of structured output where speculative decoding should excel.
• Reasoning - Security incident analysis with partial evidence and constrained mitigation steps. About 300 input tokens, with a 12000-token output cap. Reasoning is less predictable than code, so draft token acceptance should drop.
• Quick answer - Short explanation of a reverse proxy. About 20 input tokens, with a 2000-token output cap. This is the kind of short response where MTP has much less time to pay for itself.

Each prompt was executed 3 times for every preset and MTP depth. The output limits were set high enough to avoid truncation during the runs.

The benchmark script starts a fresh Podman container for each run, sends a warmup request to load the model into VRAM, and then executes the real prompt while polling GPU metrics through nvidia-smi every 500 ms. Results are appended to a CSV file immediately so that each run is recorded as soon as it finishes.

The full configuration, including prompts and preset definitions, is available in matrix.yml.

Results

The numbers below are averages across 3 runs. This is still a relatively small benchmark, so differences of only 1-2% should be treated as a tie.

For comparison, I use generation throughput as the main metric because it is normalized for output length. Total execution time is still useful, but by itself it can be misleading: a shorter run may simply mean the model generated fewer tokens, not that decoding was faster.

Coding - thinking-coder-max-ctx (q8_0, 98304 context)

This is the workload where MTP helps the most. The output is long, repetitive, and highly structured, which gives speculative decoding plenty of opportunities to reuse accepted draft tokens.

MTP-3 delivers the best throughput here. Acceptance stays high enough through depth 3 to keep speculative decoding efficient, but by depth 4 the acceptance rate drops enough to reduce the benefit.

MTP-2 has the shortest total runtime, but that is mostly because it produced fewer tokens than MTP-3. If the goal is raw decoding speed, generation throughput is the better comparison, and by that metric MTP-3 is the winner.

The prefill penalty is visible, but in practice it is small. On a run that lasts two to five minutes, an extra 80 ms at prompt processing time is not something you are likely to notice.

Compared to the smaller thinking-coder preset, thinking-coder-max-ctx starts from a slower no-MTP baseline. With MTP enabled, though, it catches up almost completely.

For workloads that actually need the larger context window, that makes the max-context preset much more practical than it looks at first glance. Without MTP it is clearly slower; with MTP it lands near throughput parity with the smaller preset in this benchmark.

The trade-off is straightforward: max-ctx gives you more room, but it gets there by using a q8_0 KV cache instead of f16.

Reasoning - thinking-coder (f16, 49152 context)

Reasoning is where MTP becomes more sensitive to prediction quality. The output is less repetitive than code, so the model has a harder time guessing several tokens ahead.

Here, MTP-1 is the most practical choice. MTP-2 reaches nearly the same throughput, but it uses about 150 MiB more VRAM and already shows a much lower acceptance rate.

That drop matters. Acceptance falls from 81.2% at depth 1 to 68.7% at depth 2, which is a sign that deeper drafting is starting to overshoot what the model can reliably reuse.

By depth 4, only about half of the speculative tokens are still being accepted. At that point MTP is still faster than the baseline, but the return on extra VRAM has clearly flattened out.

Prefill changes only slightly, from about 0.62 s to 0.67 s. On a run that lasts around two minutes, that is a small cost compared to the decoding gain.

I also ran the same reasoning prompt with the thinking-general preset to see how a higher temperature changes things. That matters because as sampling becomes less predictable, speculative decoding usually gets less efficient.

That is exactly what happens here. At temperature 1.0, the model explores a wider range of candidates for the future tokens, fewer draft tokens are accepted, and the MTP speedup shrinks across the board.

In practical terms, MTP still helps, but deeper draft depths become harder to justify. For this particular reasoning prompt, temperature 0.6 gave better MTP throughput than temperature 1.0. This comparison is about inference speed, not answer quality.

Quick answer - instruct (f16, 49152 context, no reasoning)

Short answers are the least favorable case for MTP. There simply is not much decoding work to accelerate.

MTP-1 is still the best option here, but the gain is modest in absolute terms. Going from 3.7 s to 2.8 s is measurable, but it is not the kind of difference that changes the overall experience very much.

As draft depth increases, the acceptance rate drops quickly and the speedup fades with it. That makes short responses the clearest example of why MTP is not an automatic win.

If your typical workload looks like this, the simpler non-MTP setup may still be the better default.

Summary

The pattern across workloads is fairly clear. MTP works best when the model is generating long, structured output with a high acceptance rate.

Here, “recommended” means the best practical balance between throughput and VRAM use, not always the single fastest configuration in absolute terms.

The full dataset is available as results.csv.

Trade-offs

MTP can be a very effective speedup, but it is not free. The trade-offs become pretty clear once you look at the benchmark as a whole.

• Generation speed vs. prefill speed. MTP speeds up decoding, but prompt processing becomes slightly slower. On long runs that extra prefill time is almost irrelevant; on very short responses it becomes a more noticeable part of the total runtime.
• VRAM. Each increase in draft depth adds roughly 150 MiB of VRAM overhead. On a 24 GB GPU, that extra usage matters because it reduces the headroom available for the KV cache.
• Diminishing returns. Going from depth 1 to depth 2 can still help a lot, especially for structured output. Going deeper than that often gives smaller gains, and on reasoning workloads it can start working against you.
• Larger model size. The MTP variant is bigger than the standard model because it includes the extra weights needed for speculative decoding. That affects download time and disk usage.
• Temperature sensitivity. MTP benefits from predictable output. As temperature goes up, acceptance rates go down, and the speedup drops with them.

Taken together, those trade-offs explain why there is no single “best” MTP setting for every use case. The right choice depends on whether you care more about raw throughput, VRAM efficiency, context size, or simplicity.

Conclusion

On this 24 GB RTX 3090 setup, MTP is most useful for long code-generation workloads. That is where speculative decoding has enough structured output to work with, and where the throughput gains are large enough to clearly justify the extra VRAM.

The best measured throughput in this benchmark came from thinking-coder-max-ctx with MTP-3 at 48.52 tok/s, which is an 87% improvement over its no-MTP baseline. At the same time, thinking-coder with MTP-3 reached almost the same throughput at 47.96 tok/s while using about 0.8 GiB less VRAM, which makes it the more balanced default if you do not need the larger context window.

For lower-temperature reasoning, MTP still helps, but the sweet spot is shallower. In this benchmark, MTP-1 and MTP-2 gave the best trade-off, while deeper drafting added VRAM cost faster than it added useful speed.

For short answers, MTP does improve throughput, but the real-world gain is small. In that case, the standard model remains a perfectly reasonable choice unless you are comfortable paying the extra VRAM cost for a modest latency reduction.