Bits per weight

Bits per weight, or bpw, is what one parameter costs on disk and in memory. The quoted rate already includes the scales and offsets each block carries, which is why it is never a round number: Q8_0 is 8.5 because a block stores 32 int8 weights plus one fp16 scale in 34 bytes, and Q4_K_M is 4.5 because 256 weights and their constants fit in 144. Halving bpw halves the bytes read per token, so it buys size and speed at once. Below Q3 the loss stops being invisible.

Formula

size (GB) = \frac{P _{billions} \times bpw}{8} bpw = \frac{8 \times bytes per block}{weights per block}

Worked example: Where 4.5 comes from

Quoted bpw is not the nominal bit width plus a fudge factor. It is the block layout, exactly. A Q8_0 block holds 32 int8 weights and one fp16 scale in 34 bytes. A Q4_K super-block holds 256 weights in 144. Divide bytes by weights and you get 8.5 and 4.5, the two numbers this simulator uses.

Q8_0: \frac{8 \times 34 B}{32 w} = 8.5 bpw Q4_K: \frac{8 \times 144 B}{256 w} = 4.5 bpw 27 B at 4.5 : \frac{27 \times 4.5}{8} = 15.2 GB

Try a 20B model at Q4_K_M on an RTX 4090