How to Self-Host an AI Model: A Complete Setup Guide
How to self-host an AI model: infrastructure options from laptop to cluster, security basics, latency expectations, version pinning, and a worked example.
Self-hosting an AI model used to mean a GPU rig and a weekend. For small and mid-sized models in 2026, it can run on a single CPU VM. This guide walks the decision from "should I" through infrastructure options, security, latency, and updates, with a decision model (Saina Helm) as the worked example. It covers the decisions; a separate hands-on walkthrough covers every command end to end.
Should you self-host an AI model?
Self-host when:
- Data must not leave. Policy, contract, or regulation. This is the reason that overrides everything else.
- You want a flat cost at high volume. Per-token billing is cheap until volume × price isn't.
- You need offline operation. Field sites, air-gapped networks, ships, factories.
- You need version control the provider can't take away. You pin the weights.
Don't self-host when a hosted API meets those constraints (most teams, most of the time; for decision steps that includes hosted decision models such as OpenAI's GPT-6 Luna Decisions), or when your workload is bursty and tiny. If you're weighing on-premise specifically, the requirements and the security review get their own guide.
Infrastructure options
| Option | Hardware | Fits | Trade-offs |
|---|---|---|---|
| Laptop / desktop (Ollama, MLX) | Any recent machine | Dev, personal automation, Apple silicon (MLX) | Shares resources with your actual work; no HA |
| Single VM + Docker | 4–8 GB RAM CPU for small models; GPU optional | One service, one box; most self-hosters | Single point of failure; you manage the VM |
| Kubernetes (run the Saina Helm container as a Deployment) | Cluster + GPU nodes as needed | Multiple models, multiple replicas, a platform team | Complexity; only worth it if you already run k8s |
| GPU workstation/server | Discrete GPU | Bigger models, throughput | Cost, power, noise; often unnecessary for <1B models |
For decision models, the sizing note: Start with at least 4 GB RAM on a CPU and measure peak memory and latency on your inputs. The October 6 timing was for Helm 1 and does not establish Helm 2 performance. A 27B reasoning model is a different conversation and a different guide.
Security basics (non-negotiables)
- TLS, always. Terminate at a reverse proxy (Caddy/nginx) or at Cloudflare Tunnel if the box has no public ingress. No plain-HTTP over networks you don't own.
- Authentication. An API key at minimum; one per consumer (Helm's server accepts a comma-separated list in
SAINA_API_KEY). A model endpoint on a LAN without auth is a future incident report. - Bind sensibly. Serve on localhost or a private interface; tunnel or proxy for external access. Don't expose the model port directly.
- Lock down cross-origin access. Set
SAINA_CORS_ORIGINSto an empty string on a private server; explicitly allow a browser origin only when needed. - No credentials in workflow definitions. Keys live in the secret store (n8n credentials, env vars), not in exported JSON.
- Audit logging is the caller's job. The Helm server doesn't write request or audit logs; log timestamps, question IDs and outcomes in the calling workflow. Be careful what you capture: warning and error logs can include request content. If you run the server with uvicorn directly,
--no-access-logkeeps request metadata out of its logs.
The worked example: Saina Helm on one VM
Use the release-pinned walkthrough for the complete Helm 2 setup, readiness check, authenticated request and n8n HTTP Request configuration. It targets saina==0.6.1 and weights revision 723490269872074d677c491f449a32cffe919c84.
The earlier Docker 0.1.2 example targeted Helm 1. Public Docker tags 0.6.0 and 0.6.0-cuda returned 404 on 2026-10-23, so use the walkthrough's pinned pip setup until the container release has been verified. Helm 2 needs a compatible server; changing only the model name in an old recipe is insufficient.
Latency and throughput expectations
- Measure on your hardware with your prompt sizes; every published number is someone else's machine.
- For Helm, measure latency on the CPU configuration you plan to deploy. Decide if that fits your workflow's patience; routing queues usually tolerate it; interactive UIs may not.
- Server 0.6.1 has a bounded queue with one execution slot by default; queue overflow returns
429and queue expiry returns503. HonorRetry-After. Configure retries in your client or n8n node, and for more throughput run several replicas behind a load balancer. If you need many decisions per second, that's the cluster row of the table.
Updates and model pinning
- Pin the model revision (Helm:
SAINA_REVISION, an immutable Hub commit). Providers deprecate; your pinned weights can't be deprecated out from under you. - Pin the server version (the verified package
saina==0.6.1; verify container availability before choosing an image) and upgrade deliberately: read the changelog, re-run your labeled test set, then roll forward. - Test the model change as well as the deploy: run the new revision in shadow mode against current answers before switching.
Checklist
- Reason written down (data / flat cost / offline / version control).
- VM sized (4 GB RAM CPU for Helm-class models); GPU only if needed.
- TLS + auth + private binding;
SAINA_CORS_ORIGINSempty; keys in the secret store. - Model revision pinned; server version pinned.
- Retries configured; replicas if one request at a time isn't enough.
- Labeled test set ready for upgrade rituals (50–100 labeled examples for a first check, 200+ for tuning thresholds).
- Monitoring: request rate, error and 429/503 rate, fallback rate, latency.