Multi-GPU scaling

A second card adds its memory almost in full, minus 1 to 2 GB of reserve, but adds bandwidth only as well as the two cards can talk. NVLink tensor parallelism reaches about 0.8 times the card count, PCIe about 0.55. Pipeline parallelism reaches 1.0 at batch one: it buys capacity and no decode speed, which is the right trade when the alternative is not running at all.

Formula

BW_{eff} \approx BW_{1} \times k k_{NVLink} = 0.8 N k_{PCIe} = 0.55 N k_{pipeline} = 1

Worked example: Two RTX 3090s over NVLink

Two cards give 48 GB of pool before reserves, but 1,498 GB/s rather than the 1,872 the raw specs suggest. Over PCIe the same pair reaches 1,030. The memory you get is what you bought; the bandwidth is what the link allows.

pool = 2 \times 24 = 48 GB before reserves NVLink: 936 \times (2 \times 0.8) = 1, 498 GB/s PCIe: 936 \times (2 \times 0.55) = 1, 030 GB/s

Try a 120B model on two RTX 3090s