BLUF
On my 32 GB Mac mini (M2 Pro), going from an 8B dense model to a 14B dense model drops generation throughput from 26.1 tok/s to 14.7 tok/s — a real, felt slowdown. But two MoE models, qwen3-coder:30b and gpt-oss:20b, hit 40.0 tok/s and 32.8 tok/s respectively, both faster than the little 8B Llama — which is not what the “bigger = slower” rule would predict.
Why I ran this
The standard advice for local LLMs is: pick the biggest model your RAM can handle and accept the speed penalty. I wanted to stress-test that assumption across a range of model sizes and architectures on a single machine with fixed RAM. Specifically, I was curious whether MoE (Mixture of Experts) models — which have large total parameter counts but only activate a small subset per token — actually deliver on their theoretical throughput advantage in practice under Ollama.
Setup and method
- Hardware: Mac mini, Apple M2 Pro, 32 GB unified memory, macOS
- Software: Ollama 0.32.1, default quantizations for each model
- Benchmark: Single fixed prompt,
num_predict=256,temperature=0.3, thinking off - Reported numbers: Warm average of 2 runs per model
- Metrics: Load time (seconds), generation throughput (tok/s), prompt eval throughput (tok/s)
Honest limitations: This is throughput-only — it says nothing about output quality, reasoning depth, or instruction-following. One machine, one prompt, two runs. Default quants vary by model, so you’re not comparing apples-to-apples weights. Don’t extrapolate to different hardware or different Ollama versions without re-testing.
Results
| Model | Architecture | Load (s) | Gen (tok/s) | Prompt eval (tok/s) |
|---|---|---|---|---|
| llama3.1:8b | Dense 8B | 7.6 | 26.1 | 1318.6 |
| phi4:14b | Dense 14B | 11.6 | 14.7 | 739.8 |
| gpt-oss:20b | Dense(?) 20B | 16.9 | 32.8 | 1131.2 |
| gemma4:26b | MoE ~4B active | 12.9 | 30.1 | 501.3 |
| gemma4:31b | MoE | 15.0 | 6.2 | 108.3 |
| qwen3-coder:30b | MoE ~3B active | 20.3 | 40.0 | 1973.2 |
What the numbers actually show
Dense models: bigger really does mean slower
For the dense models, the pattern is exactly what you’d expect. llama3.1:8b generates at 26.1 tok/s. phi4:14b drops to 14.7 tok/s — a significant slowdown you’ll notice in conversation. Load times track parameter count too: 7.6 s for the 8B, 11.6 s for the 14B, 16.9 s for gpt-oss:20b.
What’s interesting is that gpt-oss:20b punches above its weight on throughput — 32.8 tok/s from a 20B model is faster than both dense models below it. I don’t have a confirmed explanation for this from the data alone; it could be quantization level, architecture efficiency, or something else. Worth investigating further.
MoE: the pattern fractures — sometimes spectacularly, sometimes badly
The MoE results are the most striking part of this benchmark, and they cut both ways.
qwen3-coder:30b (MoE, ~3B active parameters per token) is the fastest model in the entire test at 40.0 tok/s generation and 1973.2 tok/s prompt eval. It loads in 20.3 s — the slowest load time — but once it’s running, it outpaces every other model including the 8B dense. That’s the MoE promise delivering: 30B of total capacity, small-model inference speed.
gemma4:26b (MoE, ~4B active) also beats the 8B Llama at 30.1 tok/s. Prompt eval at 501.3 tok/s is on the lower end, but generation speed is solid.
gemma4:31b is the cautionary tale. Despite also being a MoE model, it crawls at 6.2 tok/s generation and 108.3 tok/s prompt eval — the slowest numbers in the entire table, slower than every other model including the dense 14B. Something about this particular model’s configuration — possibly larger active parameter count, higher quant size pushing memory bandwidth limits, or the full 32 GB being taxed — causes it to fall apart on this machine. MoE is not a magic speed card; it depends heavily on the specific model’s implementation and how well it fits your available memory bandwidth.
Practical takeaways for 32 GB Mac users
- Don’t assume dense 8B is your speed floor. qwen3-coder:30b at 40.0 tok/s and gemma4:26b at 30.1 tok/s prove you can run much larger models faster.
- phi4:14b at 14.7 tok/s is the slowest generator here — if you’re using it for interactive chat, that’s a noticeable pace.
- MoE can fail on this hardware. gemma4:31b at 6.2 tok/s is a reminder to always benchmark before committing to a model for daily use.
- Load time is a separate axis. qwen3-coder takes 20.3 s to load but then runs fast; llama3.1:8b loads in 7.6 s. If you’re switching models frequently, load time matters. If you keep one model resident, it doesn’t.
- Prompt eval throughput varies wildly — from 108.3 tok/s (gemma4:31b) to 1973.2 tok/s (qwen3-coder:30b). For long-context use cases, this matters as much as generation speed.
Method reminder
All numbers come from a single fixed prompt, num_predict=256, temp=0.3, thinking off, warm average of 2 runs, default Ollama quantizations, one machine. This measures throughput only — not output quality, coherence, or task accuracy. Your numbers will differ on different hardware or with non-default quants.
Running similar benchmarks on your own machine? Drop your model, hardware, and tok/s numbers in the comments — I’d like to see how M3, M4, and non-Apple silicon compare on the same models.
