Choosing an inference stack

Hardware and workload pick the runtime, not preference. MLX on Apple Silicon, where it has measured near 3 times llama.cpp on the same model. llama.cpp anywhere, as the portable default. ExLlamaV3 for single-user NVIDIA, vLLM or SGLang once you are serving concurrent requests. A large MoE model that needs CPU expert offload overrides all of this and wants the hybrid path.

Formula

stack = f (platform, full-memory fit, concurrency, dense or MoE)

Worked example: Qwen3-Coder-30B-A3B on an M4 Pro

MLX reaches about 130 tokens per second here. The older llama.cpp backend reaches 43. Nothing about the hardware or the model changed between those two numbers, which is why runtime selection comes before quant tuning, not after.

MLX: 130 tps llama.cpp: 43 tps \frac{130}{43} = 3.0 \times

Try a Qwen model on an M4 Pro with MLX