Why bandwidth decides speed
During single-user decode, the machine repeatedly reads model weights to produce one token. That workload uses only about 1 to 2 FLOP per byte in the research model, so memory bandwidth is the binding limit. Memory bandwidth utilization, or MBU, accounts for the share of peak bandwidth the runtime actually reaches.
Formula
decode tps ≈ MBU × memory bandwidth (GB/s) ÷ bytes read per token (GB)
Worked example: RTX 4090 with an 8B Q4 model
The RTX 4090 has 1008 GB/s of peak bandwidth. At 70% MBU with about 4.6 GB read per token, the estimate lands inside the measured 140 to 180 tps band.
1008 × 0.70 ÷ 4.6 ≈ 153 tps