---
title: "How fast is local AI?"
description: "Fast enough for interactive use on the right hardware: large open models generate about 20 to 50 tokens per second for a single user on Mac Studio clusters, faster than people read. Speed depends mostly on memory bandwidth and on how many parameters are active for each token."
url: https://exolabs.net/faq/how-fast-is-local-ai/
updated: 2026-10-26
---

# How fast is local AI?

Fast enough for interactive use on the right hardware: large open models generate about 20 to 50 tokens per second for a single user on Mac Studio clusters, faster than people read. Speed depends mostly on memory bandwidth and on how many parameters are active for each token.

Generation (decode) reads the active weights once per token, so it is limited by memory bandwidth. Reading the prompt (prefill) is limited by compute, which is why very long prompts take longer on Macs than on NVIDIA GPUs.

The software matters as much as the hardware. On the same four Macs, exo with RDMA ran DeepSeek V3.1 at 32.5 tokens per second; llama.cpp over a regular network managed 14.6.

Measured generation speed, one request at a time

| Setup | Model | 1 Mac | 2 Macs | 4 Macs |
| --- | --- | --- | --- | --- |
| exo + RDMA, M3 Ultra (Geerling) | DeepSeek V3.1 671B | 21.1 | 27.8 | 32.5 tok/s |
| exo + RDMA, M3 Ultra (Geerling) | Qwen3-235B-A22B | 19.5 | 26.2 | 31.9 tok/s |
| exo + RDMA, M3 Ultra (Geerling) | Kimi K2 Thinking, 4-bit | — | 21.6 | 28.3 tok/s |
| llama.cpp over TCP, M3 Ultra (Geerling) | DeepSeek V3.1 671B | 14.8 | 16.0 | 14.6 tok/s |
| MLX tensor parallel, M5 Ultra (Apple) | Qwen3.8-27B, BF16 | 19.7 | 33.3 | 50.9 tok/s |

**Where exo fits:** exo includes exo-bench, which measures prompt and generation speed for every way of placing a model on your cluster, so you can see real numbers before you commit.

Sources:
- [Jeff Geerling: 1.5 TB of VRAM on Mac Studio, RDMA over Thunderbolt 5](https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5)
- [Apple Machine Learning Research: distributed inference with MLX (Oct 15, 2026)](https://machinelearning.apple.com/research/mlx-distributed-inference)
- [exo on GitHub (README)](https://github.com/exo-explore/exo)

Page: https://exolabs.net/faq/how-fast-is-local-ai/ · Updated 2026-10-26 · Deployments: https://exolabs.net/#business (sales@exolabs.net)