How fast is local AI?
Fast enough for interactive use on the right hardware: large open models generate about 20 to 50 tokens per second for a single user on Mac Studio clusters, faster than people read. Speed depends mostly on memory bandwidth and on how many parameters are active for each token.
Generation (decode) reads the active weights once per token, so it is limited by memory bandwidth. Reading the prompt (prefill) is limited by compute, which is why very long prompts take longer on Macs than on NVIDIA GPUs.
The software matters as much as the hardware. On the same four Macs, exo with RDMA ran DeepSeek V3.1 at 32.5 tokens per second; llama.cpp over a regular network managed 14.6.
| Setup | Model | 1 Mac | 2 Macs | 4 Macs |
|---|---|---|---|---|
| exo + RDMA, M3 Ultra (Geerling) | DeepSeek V3.1 671B | 21.1 | 27.8 | 32.5 tok/s |
| exo + RDMA, M3 Ultra (Geerling) | Qwen3-235B-A22B | 19.5 | 26.2 | 31.9 tok/s |
| exo + RDMA, M3 Ultra (Geerling) | Kimi K2 Thinking, 4-bit | — | 21.6 | 28.3 tok/s |
| llama.cpp over TCP, M3 Ultra (Geerling) | DeepSeek V3.1 671B | 14.8 | 16.0 | 14.6 tok/s |
| MLX tensor parallel, M5 Ultra (Apple) | Qwen3.8-27B, BF16 | 19.7 | 33.3 | 50.9 tok/s |
Sources
Updated