What is the best hardware for running LLMs locally?
For the largest models per dollar, Apple silicon Macs with lots of unified memory: a Mac Studio offers up to 256 GB today at about $37 per GB, and several can be clustered. For serving many users at once, NVIDIA datacenter GPUs. For small models, any recent laptop or desktop GPU with 16 to 32 GB of memory.
Decide by model size first. The model must fit in fast memory, so memory capacity rules out options before anything else does.
Then look at memory bandwidth, which sets generation speed, and compute, which sets how fast long prompts are read.
Consider the whole system: power, noise, space and who will maintain it.
Sources
Updated