How do I secure a self-hosted LLM?
Treat the model server like any internal service: keep it off the public internet, put authentication and logging in front of it, and control what goes into it. Download weights only from trusted publishers in the safetensors format, keep remote code execution off, and limit what tools an AI agent can call.
Network: bind the inference API to a private network or VPN. Many local servers have no authentication of their own.
Access: put a gateway in front for sign-in, per-user keys, rate limits and audit logs.
Supply chain: record model checksums, prefer safetensors (which cannot run code when loaded), and review any model that asks for trust_remote_code.
Agents: an AI agent with tool access can act on what it reads, including instructions hidden in documents or web pages (prompt injection). Give agents the least access they need and require approval for risky actions.
Sources
Updated