TL;DR
- Self-hosted AI is not a preference; it is a requirement when data residency, vendor independence, or deterministic control drive the decision.
- The five criteria that trigger a self-hosting decision are regulatory, contractual, competitive, availability, and cost at scale.
- The deployment pattern is mature in 2026: open-weight models (Llama, Mistral, Qwen, Gemma) on client GPUs via Ollama or vLLM, fronted by an orchestration layer that owns the tool registry, retrieval, and audit.
- The quality gap to frontier cloud models is narrowing fast on specific enterprise workloads.
The default narrative is wrong for regulated enterprises
The industry narrative — "everyone is using the frontier cloud LLM; you should too" — is accurate for a lot of workloads. It is often wrong for regulated enterprises, and the cost of getting it wrong is not a small efficiency loss. It can be a compliance incident, a contract breach, or a competitive-information leak.
Every serious AI program we have run in banking, healthcare, or government has hit one of the same five decision points where self-hosting became not an optimization but a precondition.
The five criteria
1. Regulatory
The client operates under a regulation — HIPAA, GLBA, SOX, PCI-DSS, FedRAMP, state privacy laws — that constrains where protected data can flow and who can process it. Business-associate agreements and data processing addenda from cloud LLM vendors exist, but they do not solve every case. For some workloads, the regulatory answer is simply: the data does not leave the environment.
2. Contractual
The client's contracts with their own customers or partners prohibit sharing content with third parties beyond a defined list. Adding a new cloud LLM vendor to that list is a renegotiation the client does not want to run. Self-hosting keeps the vendor list short.
3. Competitive
The workload processes proprietary research, strategic planning, pricing models, or underwriting criteria where sending the content to a third party — even one with a strong privacy posture — creates competitive risk the client cannot accept.
4. Availability and reliability
The workload is operationally critical and cannot tolerate the variability of a cloud LLM provider: latency spikes, rate limits, outages, deprecation of a model version. Self-hosting puts the availability curve under the client's control.
5. Cost at scale
At high enough volume, self-hosted inference on GPUs the client owns becomes cheaper per token than cloud API pricing. This is the weakest of the five criteria on its own — the break-even volume is higher than most enterprise workloads hit — but combined with any of the first four it is the additional reason to make the move.
The deployment pattern that works
Our 2026 reference architecture for self-hosted AI in regulated enterprises:
- Model layer. Open-weight models (Llama 3.3 70B, Qwen 2.5 series, Mistral Medium, Gemma 27B) served via Ollama or vLLM on client GPUs. Multi-model routing where different sub-tasks fit different models.
- Retrieval layer. Embedding model (BGE, E5, or a domain-tuned alternative) plus a vector store — pgvector is our default for operational simplicity, with native Databricks vector indexes or dedicated engines where scale requires.
- Orchestration layer. Agent framework with tool registry, approval gates, and full trace capture. This is where MCP typically lives as the tool protocol.
- Governance layer. Audit log with append-only retention matched to the regulatory regime, evaluation pipeline running continuous grounding and accuracy checks, PII redaction and content-safety pre-filters.
- Observability. Standard SRE tooling (metrics, traces, logs) plus AI-specific observability (token counts, confidence distributions, evaluation scores).
Our self-hosted reference deployment
TWSS Commercial Credit AI runs this pattern in production: a 3-model ensemble (Qwen 2.5, Gemma 27B, Llama 3.3 70B), zero external API calls, full audit per loan, and the platform economics that make it MBE/GSA-procurable. See the full case study for the architecture detail.
The decision framework in one paragraph
If one of the five criteria applies — regulatory, contractual, competitive, availability, cost — self-hosting is not a preference discussion, it is the only answer. If none of them apply, cloud APIs are the right starting point and the operational simplicity is worth the trade. Most of the friction in real decisions comes from leaders who assume "we will add self-hosting later if we need to" — and then discover that after-the-fact re-platforming of an AI workload is 5-10x the cost of building it self-hosted from the start. The decision is better made early.
For broader context on our practice, see the AI & Generative AI service and the accelerators portfolio.