How many users can a local AI cluster serve?
It depends on the model, the hardware and how heavily people use it. A Mac cluster running a large model typically serves a handful of simultaneous requests well, which can cover a team of dozens of people because most people are not generating text at the same moment. Heavy concurrent use, such as many coding agents running at once, calls for more machines or datacenter GPUs.
Measure concurrent requests, not head count. Ten people chatting may produce one or two active requests at a time. Ten coding agents running continuously produce ten.
Every active request needs memory for its context (the KV cache). Long contexts per user reduce how many requests fit at once.
Macs are strongest at low to moderate concurrency, where memory capacity and bandwidth matter most. Datacenter GPUs pull ahead on cost per token when many requests are batched together.
Sources
Updated