Prefill vs decode

Prefill reads the whole prompt at once and saturates the compute units. Decode emits one token at a time and is memory-bandwidth-bound, because a single token cannot fill the arithmetic pipeline no matter how wide it is. That split is why one machine reports two very different token rates: prompt length over prefill throughput gives time to first token, and decode throughput sets the pace you watch.

Formula

prefill tps = \frac{MFU \times peak FLOPS}{2 \times P _{active}} TTFT = \frac{prompt tokens}{prefill tps}

Worked example: A 4k prompt on an RTX 4090

At 165 FP16 TFLOPS and 40 percent utilization, an 8B model prefills at 4,125 tokens per second, then decodes at 153. The same card, the same model, a 27-fold gap. Prefill fills the arithmetic pipeline; a single decoded token cannot.

\frac{0.40 \times 165 \times 1 0 ^{12}}{2 \times 8 \times 1 0 ^{9}} = 4, 125 tps \frac{4 , 096}{4 , 125} \approx 0.99 s to first token

Try a 20B model with a 4k prompt