Private AI Inference: What It Is and How to Set It Up
Private AI inference compared: self-hosted, VPC-hosted, confidential computing and provider zero-retention, what each protects, and how to set up self-hosting.
Private AI inference is four different arrangements sold under one marketing word. They differ in what they actually protect, and confusing them is how teams end up with a compliance story that doesn't match their architecture. This guide defines the patterns and what each one gives you.
The four patterns of private AI inference
1. Self-hosted inference
The model runs on hardware you control; requests never leave your boundary.
- Protects: content confidentiality (nothing egresses), availability from provider outages, version pinning.
- Does not protect: your own misconfiguration, such as an unauthenticated endpoint on the LAN.
- Costs: you run a server; sizing starts small for decision workloads (a 0.8B decision model like Saina Helm runs on a CPU VM; sizing details).
- Pattern example: see the worked deployment shape in the on-premise guide.
2. VPC-hosted (private endpoint) inference
A provider's model, exposed only inside your cloud VPC; traffic never crosses the public internet.
- Protects: exposure to the public internet; often satisfies "data doesn't traverse the open internet" requirements.
- Does not protect: the provider's internal handling; your content reaches their infrastructure under their terms.
3. Confidential computing
Model execution inside trusted execution environments (hardware enclaves with cryptographic attestation), so even the operator of the machine can't read the data in use.
- Protects: use-time confidentiality against the infrastructure operator, with cryptographic attestation.
- Does not protect: what the enclave's operator still controls (code updates, interfaces); availability is workload-dependent.
4. Provider zero-retention options
Hosted APIs with contractual/technical no-training, no-logging terms. OpenAI and other large providers offer zero-retention or no-training terms for eligible API use; read the current terms for the endpoint you use. Saina's own hosted API is another example: the same server as the self-hosted one, under a zero-retention policy (free account).
- Protects: storage and secondary use (training on your data), within the contract's definitions and jurisdiction.
- Does not protect: transmission (data still egresses to the provider), jurisdiction questions, or changes to the terms. If your actual requirement is "no storage and no training", this pattern meets it with the least work.
What each pattern actually protects
| Self-hosted | VPC-hosted | Confidential computing | Zero-retention | |
|---|---|---|---|---|
| Data egresses your boundary | No | To the provider, never over the public internet | Yes, to attested hardware; encrypted in use | Yes |
| Provider sees content | n/a | Yes, under their terms | Not in use | Yes, transit/processing |
| Trains on your data | n/a | Per terms | n/a | Contractually no |
| You pin the model version | Yes | Provider-managed | Deployment-specific | Provider-managed |
| Ops burden | Yours | Low | Medium-high | None |
Read that table as a requirements-matching exercise: write down which row you actually need, and the pattern picks itself.
Setting up the self-hosted pattern
For decision workloads, the shortest path (the full commands are in the step-by-step walkthrough):
- Choose the workload: the decision steps (route, tag, gate, rate), not generation, first.
- Run the model: Saina Helm in Docker (
ghcr.io/run-saina/saina-server:0.1.2) or via pip on an internal VM, weights revision pinned. - Boundary discipline: bind privately, TLS at the proxy, per-consumer API keys; set
SAINA_CORS_ORIGINS=""so no browser origin can call it, and run with--no-access-log(pip/uvicorn) if request metadata shouldn't be logged. The calling workflow logs decisions with reasons; keys in a secret store, never in workflow exports. - Prove the boundary: pre-stage the pinned checkpoint first (the first start otherwise downloads ~1.7 GB of weights from Hugging Face), then apply egress-deny rules on the box that would fail loudly if anything tried to phone home; audit what's logged and retained.
- Scope the claim: "no data leaves the organization" is a claim about this pattern only. VPC-hosted and zero-retention setups can be excellent; they just don't earn that sentence.
Checklist
- The actual requirement named (no egress? no training? no public transit?).
- Pattern chosen to match the requirement, not the marketing.
- For self-hosted: private binding, TLS, auth, CORS off, checkpoint pre-staged, egress-deny proof.
- Terms read and quoted for any hosted component, with their date.
- Data-flow diagram written down and current.