What is distributed inference? Pipeline vs tensor parallelism
Distributed inference runs one model across several devices. Pipeline parallelism gives each device a block of layers, which pools memory but does not speed up a single request. Tensor parallelism splits every layer across devices, so they work on each token together, which can make a single request faster when the link between them is fast.
Pipeline parallelism needs little bandwidth and works over ordinary networks. One request passes through the devices in turn, so single-user speed stays about the same as one device, minus some network delay.
Tensor parallelism needs many small exchanges for every token, so latency between devices matters. That is why fast, low-latency links such as RDMA over Thunderbolt 5 make it practical on Macs.
A third approach, disaggregated prefill and decode, uses a compute-heavy device to read the prompt and a bandwidth-heavy device to write the answer.
Sources
Updated