Low-bit training

Post-training quantization rounds a finished model down. It holds at 8 bits and starts failing below 4, because rounding error the model never saw during training has nowhere to go. Quantization-aware training fixes that by keeping master weights in full precision for the backward pass while quantizing them on the fly for every forward pass, so the model learns weights that survive rounding. The 1-bit LLMs are the limit case and the name is marketing: ternary weights need log2(3), or 1.58 bits, and only the linear layers convert. Embeddings, attention, and normalization stay full precision.

Formula

ternary floor = lo g_{2} 3 \approx 1.585 bpw packed rate = \frac{bits per group}{weights per group}

Worked example: Storing a ternary weight

Three values need log2(3) bits, and no hardware has a 1.58-bit type. BitNet packs three weights into one 5-bit code instead, landing at 1.667 against a floor of 1.585 and beating the 2 bits a naive field would spend. Only the linear layers convert, so no checkpoint reaches this rate end to end.

floor = lo g_{2} 3 = 1.585 naive 2-bit field = 2.000 3 weights in 5 bits = \frac{5}{3} = 1.667

Compare against an ordinary 4-bit model