Local AI FAQ
Local AI, answered.
How to cut AI token spend, run open models on hardware you own, and deploy private AI across a company. Plain answers, with sources.
Cost and saving money
When owning hardware beats paying per token, and how to cut an AI bill.
How can I reduce my AI API token costs?
Measure where tokens go, then cut in this order: send easy requests to small, cheap models; use prompt caching for repeated context; trim what you send; batch work that can wait; and move steady, high-volume work to open models on hardware you own, where each extra token costs only electricity.
Full answer →Is running AI locally cheaper than paying for an API?
It is when your usage is steady and heavy and you would otherwise pay frontier-model prices. Three 256 GB Mac Studios, enough for a frontier-size open model such as DeepSeek V3.1, cost $28,497, or about $960 a month over three years including power. A workload that costs thousands a month in frontier API fees pays that back in months. Against cheap hosted open models, owning costs about the same, and the case for local rests on privacy and control.
Full answer →How do I cut the token costs of AI coding agents?
Coding agents resend large amounts of code and history on every step, so input tokens dominate the bill. Use prompt caching, keep context lean, reserve frontier models for hard tasks, and run routine agent work on an open coding model hosted on your own hardware, where tokens have no per-unit price.
Full answer →How much does it cost to run a frontier-size open model locally?
About $28,000 to $57,000 in Mac hardware as of October 2026, sized with room for context. Three 256 GB Mac Studios ($28,497) run DeepSeek V3.1 or GLM-5.3 at 4-bit. Four ($37,996) run Kimi K2 Thinking. Six ($56,994) run DeepSeek V4 Pro. Holding the same weights on 80 GB NVIDIA H100s takes at least five GPUs for DeepSeek V3.1 and twelve or more for DeepSeek V4 Pro, at roughly $25,000 to $40,000 each.
Full answer →Should I buy hardware or rent cloud GPUs for AI inference?
Rent for short or spiky work; buy for steady, long-running work. An H100 rents for about $3.29 to $4.29 an hour, or $2,400 to $3,100 a month if it runs nonstop. A large open model such as DeepSeek V3.1 at 4-bit needs five to eight H100s, or $12,000 to $25,000 a month rented, while three 256 GB Mac Studios that run it cost $28,497 once.
Full answer →How much electricity does running AI locally use?
Less than most people expect. Apple measured an M5 Ultra Mac Studio at a maximum of 385 W from the wall, so even at full load around the clock it uses about 281 kWh a month, roughly $56 at $0.20 per kWh. A single NVIDIA RTX 5090 is rated at 575 W, and an H100 SXM at up to 700 W, before the rest of the server.
Full answer →
Enterprise deployment
Running private AI across a company: architecture, models, security, users and agents.
What does an enterprise local AI deployment look like?
A typical setup has four layers: machines that hold the models (one server or a cluster), inference software that serves an OpenAI- or Anthropic-compatible API, a gateway that adds sign-in, quotas and logging, and the apps people use, such as a chat interface, coding agents and internal tools. Everything runs inside the company network.
Full answer →What is the best self-hosted ChatGPT alternative for a business?
Pair a strong open-weight model with a self-hosted chat interface. A common stack is Open WebUI for the chat experience, connected to a local inference server that runs a large open model from the DeepSeek, Kimi, Qwen or GLM families. Employees get a ChatGPT-like experience, and every prompt and document stays on company hardware.
Full answer →How do I secure a self-hosted LLM?
Treat the model server like any internal service: keep it off the public internet, put authentication and logging in front of it, and control what goes into it. Download weights only from trusted publishers in the safetensors format, keep remote code execution off, and limit what tools an AI agent can call.
Full answer →What is sovereign AI?
Sovereign AI means an organization or country controls its own AI: the models, the hardware they run on, and the data that flows through them, without depending on a foreign or outside provider. For a company, it usually means running open-weight models on infrastructure it owns, in a location it chooses.
Full answer →How do I deploy private AI for my team or company?
Pick the models your team needs, size hardware so those models fit in memory with room for context, install inference software that exposes a standard API, and point your team’s apps, chat interface and coding agents at that endpoint. Start small with one or two machines and grow from real usage.
Full answer →How many users can a local AI cluster serve?
It depends on the model, the hardware and how heavily people use it. A Mac cluster running a large model typically serves a handful of simultaneous requests well, which can cover a team of dozens of people because most people are not generating text at the same moment. Heavy concurrent use, such as many coding agents running at once, calls for more machines or datacenter GPUs.
Full answer →Can I use local models with coding agents like Claude Code, Codex or OpenCode?
Often, yes. Many coding agents let you set a custom API base URL, and local servers such as exo expose OpenAI- and Anthropic-compatible endpoints. Point the agent at your local endpoint and choose a large open model that is strong at tool calling, such as a recent GLM, Kimi, DeepSeek or Qwen coding model.
Full answer →How do I switch an app from the OpenAI API to a local model?
Run a local server with an OpenAI-compatible API, then change the base URL in your app (for example to http://localhost:52415/v1) and set the model name to a local model. Most OpenAI SDKs and tools need no other code changes.
Full answer →Can I run AI in an air-gapped or on-premises environment?
Yes. Open-weight models and local inference software run with no internet connection. Download the models and software on a connected machine, move them into the secure network, and run everything there.
Full answer →
Hardware and memory
How much memory you need, which machines work well, and what decides speed.
How much memory do I need to run an LLM locally?
Multiply the parameter count in billions by the bits per weight and divide by 8 to get gigabytes, then add 10 to 25% for context and the system. At 4-bit, a 70B model needs about 40 GB, a 235B model about 130 GB, and a 671B model such as DeepSeek V3.1 about 380 GB.
Full answer →What is the best hardware for running LLMs locally?
For the largest models per dollar, Apple silicon Macs with lots of unified memory: a Mac Studio offers up to 256 GB today at about $37 per GB, and several can be clustered. For serving many users at once, NVIDIA datacenter GPUs. For small models, any recent laptop or desktop GPU with 16 to 32 GB of memory.
Full answer →Are Macs good for running AI models?
Yes, especially for large models. Apple silicon shares one pool of fast unified memory between the CPU and GPU, so a Mac Studio with M5 Ultra can give a model up to 256 GB at 1.2 TB/s. That is more model memory than several high-end graphics cards, in a quiet desktop that draws under 400 W.
Full answer →Mac Studio vs NVIDIA GPUs for local AI: which is better?
Mac Studio wins on memory per dollar, power and simplicity: about $37 per GB of model memory versus roughly $54 for a DGX Spark and $310 or more for an H100. NVIDIA wins on raw compute, which speeds up long prompts and serving many users at once. Many teams want the memory more than the compute.
Full answer →Can I run DeepSeek, Kimi or GLM locally?
Yes, if you have enough memory. DeepSeek V3.1 needs about 378 GB at 4-bit, which runs comfortably on three 256 GB Mac Studios. Kimi K2 Thinking needs about 594 GB, or four. DeepSeek V4 Pro ships at about 893 GB, or six. With exo, four Mac Studios ran DeepSeek V3.1 at 32.5 tokens per second.
Full answer →How fast is local AI?
Fast enough for interactive use on the right hardware: large open models generate about 20 to 50 tokens per second for a single user on Mac Studio clusters, faster than people read. Speed depends mostly on memory bandwidth and on how many parameters are active for each token.
Full answer →Can I run an LLM without a GPU?
Yes, but large models will be slow. Tools such as llama.cpp run on ordinary CPUs, and speed is limited by system memory bandwidth, about 90 GB/s for a typical dual-channel DDR5 desktop. Small models (7 to 14 billion parameters at 4-bit) are usable; 70B models crawl.
Full answer →
Clusters and distributed inference
Running one model across several machines, and when that makes it faster.
Can I connect several computers to run one AI model?
Yes. Distributed inference software such as exo splits a model across machines so their memory adds up. Jeff Geerling connected four Mac Studios with 1.5 TB of combined memory and ran the 671B-parameter DeepSeek V3.1 with exo.
Full answer →What is distributed inference? Pipeline vs tensor parallelism
Distributed inference runs one model across several devices. Pipeline parallelism gives each device a block of layers, which pools memory but does not speed up a single request. Tensor parallelism splits every layer across devices, so they work on each token together, which can make a single request faster when the link between them is fast.
Full answer →What is RDMA over Thunderbolt 5, and why does it matter for AI?
RDMA (remote direct memory access) lets one computer read and write another computer’s memory directly, without going through the usual network stack. Apple added RDMA over Thunderbolt 5 in macOS Tahoe 26.2. It cuts latency between Macs enough to make tensor parallelism practical, so a cluster can run a model faster than a single Mac.
Full answer →Does adding more computers make an AI model run faster?
It can, but only with the right setup. With tensor parallelism over a fast, low-latency link, exo reports up to 1.8× speedup on 2 Macs and 3.2× on 4. Over a slow network, or with pipeline parallelism, extra machines add memory but not single-user speed.
Full answer →Can I mix Macs and NVIDIA hardware for AI inference?
Yes, and the two are good at different things. NVIDIA GPUs have more compute, which speeds up reading long prompts (prefill). Macs have more memory bandwidth per dollar, which speeds up writing the answer (decode). Exo Labs measured a 2.8× speedup by prefilling on an NVIDIA DGX Spark and decoding on a Mac Studio.
Full answer →
Privacy and control
Where your data goes, compliance, and why companies keep AI in-house.
Is local AI private? Does my data leave my network?
When the model runs on your own hardware, your prompts, files and outputs stay on your machines. Nothing is sent to an AI provider, so there is no third party that can log, retain or train on your data.
Full answer →Does local AI help with HIPAA, GDPR or data-residency rules?
It can make compliance simpler, because sensitive data never goes to an outside AI provider, so there is no extra processor to vet and no cross-border transfer. Local AI does not make a system compliant by itself: you still need access controls, logging, retention rules and the rest of your normal safeguards.
Full answer →Why do companies run AI on their own hardware?
The common reasons are privacy, predictable cost, and control. Data stays in-house, spending is a one-time hardware cost instead of a growing per-token bill, and the company decides which models run, when they change, and who can use them.
Full answer →
Local AI basics
What local AI is, what it can do today, and what it means for quality.
What is local AI?
Local AI means running AI models on hardware you control, such as a laptop, a desktop or a cluster in your office, instead of sending requests to a cloud provider. The model weights live on your machines, and your prompts and outputs never leave them.
Full answer →Can I run a large language model on my own computer?
Yes. A recent laptop with 16 GB of memory can run small models of about 7 to 14 billion parameters. A desktop with 64 to 128 GB can run models in the 70 to 120 billion range. The largest open models, from about 700 billion to over 2 trillion parameters, need hundreds of gigabytes or more and usually a cluster of machines.
Full answer →Are open models as good as ChatGPT or Claude?
For many everyday tasks, the best open-weight models are close to the leading closed models. On the hardest reasoning and long agentic coding tasks, the top closed models usually still lead. The gap has narrowed to months rather than years.
Full answer →What is an open-weight model?
An open-weight model is an AI model whose trained weights are published for download. You can run it on your own hardware, fine-tune it and, depending on its license, use it commercially. It is not always fully open source: the training data and code are often not released.
Full answer →What is quantization, and does it hurt model quality?
Quantization stores model weights with fewer bits, for example 4 or 8 bits instead of 16. A 4-bit model needs about a quarter of the memory of a 16-bit one and usually runs faster. At 8 bits the quality loss is usually hard to notice, and at 4 bits it is small for most tasks.
Full answer →Can I run AI without an internet connection?
Yes. Once the model and software are downloaded, local AI runs fully offline. You need internet access only to download new models or updates.
Full answer →
Comparisons
How exo and local AI compare with other tools and with cloud AI services.
exo vs Ollama, LM Studio and llama.cpp: what is the difference?
Ollama, LM Studio and llama.cpp are built mainly to run a model on one machine. exo is built to run one model across many machines: it pools their memory and, with RDMA over Thunderbolt 5, their compute. Use a single-machine tool when your model fits on one computer, and exo when it does not or when you want a cluster to run it faster.
Full answer →exo vs vLLM: which should I use?
Use vLLM to serve models on datacenter GPUs, mostly NVIDIA, at high concurrency. Use exo to run large models on Apple silicon Macs, especially when one model needs the memory of several machines. vLLM is built for throughput in a GPU server; exo is built to turn desk-side hardware into one cluster with no configuration.
Full answer →ChatGPT Enterprise vs self-hosted AI: which is right for a company?
ChatGPT Enterprise, Claude for Work and similar services give you the newest closed models with no hardware to run, billed per seat or per token. Self-hosted AI gives you full control of data, fixed costs and no vendor lock-in, using open models on your own hardware. Many companies use both: cloud models for the hardest tasks, local models for sensitive data and high-volume work.
Full answer →
About exo
What exo is, what it runs on, and how to get help from Exo Labs.
What is exo?
exo is free, open-source software from Exo Labs that connects your devices into one AI cluster. It splits large models across machines so you can run models bigger than any single device holds, and with RDMA over Thunderbolt 5 it can make them run faster as you add devices.
Full answer →Is exo free and open source?
Yes. exo is open source under the Apache 2.0 license, which allows commercial use. The code is on GitHub at github.com/exo-explore/exo.
Full answer →What hardware does exo support?
exo runs on Apple silicon Macs. Tested and maintained platforms are the M3 Ultra Mac Studio, M4 Pro Mac mini, and M4 Max and M5 MacBook Pro. The macOS app needs macOS Tahoe 26.2 or later. On Linux, exo currently runs on CPU; NVIDIA DGX Spark support is planned.
Full answer →How do I install exo?
On a Mac with macOS Tahoe 26.2 or later, download the exo app from exolabs.net or run brew install --cask exo. Install it on each Mac you want in the cluster. The devices find each other, and the dashboard and API run at http://localhost:52415.
Full answer →Does Exo Labs help businesses deploy local AI?
Yes. Exo Labs helps teams plan and deploy private AI on hardware they own: choosing models, sizing Mac clusters, and connecting exo to the tools and agents the team already uses. Start with the form at exolabs.net or email sales@exolabs.net.
Full answer →