Draft-token setting and speed on Gemma 4 31B
- Date
- 24 October 2026
- Task type
- Local models
- Status
- Complete
Models tested
| Model or tool | Provider | Open? | How run |
|---|---|---|---|
| Gemma 4 31B (gemma4:31b, quantised weights, through Ollama, with a draft component for multi-token prediction) | Open weights | Local |
Purpose
Measure how the number of draft tokens (multi-token prediction) changes generation speed for a local model used for batch text processing.
Context
Gemma 4 31B served through Ollama on an NVIDIA DGX Spark. Three real inputs from a cataloguing task, each sent twice per setting. The run date is not recorded; the figures come from a report dated 24 October 2026.
Method
The draft setting was set to 0, 3 and 5 in turn. Each of the three inputs was sent twice at each setting, with warm-up requests excluded. Median generation speed and median request time (excluding reported model load) were recorded, and the outputs were scored on 6 automated checks.
Results
| Draft setting | Median generation speed (tokens/s) | Median request time (s) | Automated checks passed |
|---|---|---|---|
| 0 | 10.6 | 28.73 | 4 of 6 |
| 3 | 32.6 | 10.23 | 4 of 6 |
| 5 | 41.3 | 8.46 | 4 of 6 |
Caveats
- Three inputs, two requests each, one machine; a small workload.
- No quality difference between the settings was established; all three passed the same 4 of 6 checks.
- The exact run date is not recorded.
What this suggests
Turning draft tokens on roughly tripled generation speed on this model and workload, and 5 was faster than 3. The checks did not show a quality difference between the settings.
Source
entries/2026-10-24-gemma-31b-draft-tokens.md in the GitHub repository (Markdown with YAML frontmatter).