Why bandwidth decides speed

Decode reads the weights once per token and does about 1 to 2 FLOP per byte, so the memory bus finishes last and sets the pace. This is why quantization speeds things up at all: integer and float multiplies run at similar rates on a modern GPU, so the gain is not cheaper arithmetic but fewer bytes to fetch. Memory bandwidth utilization, or MBU, is the share of peak the runtime actually reaches, and no runtime reaches all of it.