Loading insights
Loading insight article
Insights / Production Engineering
Open-weight models and local runtimes make self-hosted legal AI practical in 2026. Here's what it takes to run an agent on your own infrastructure including hardware, architecture, real costs and firm size where it starts to make sense.
Most legal AI in 2026 means one thing: log into a vendor's cloud and hope their security page holds up. A parallel option is growing fast; running the same kind of agent on infrastructure your firm actually owns, where the model, the data and its memory never leave your network. This is realistic now because Open-weight models such as Llama 3.3 and Qwen 2.5 are now capable enough for many legal tasks and local runtimes like llama.cpp, vLLM, and Ollama make running them in-house practical. But what hardware do you need and when does self-hosting break even with SaaS?
With self-hosted AI, law firm data sovereignty is achievable. The technology is here, but deploying it inside a firm's own infrastructure comes with practical requirements around hardware, architecture, capabilities, and cost. In this piece, we will answer
Client confidentiality isn't optional in legal work, it's a professional duty. Attorneys are expected to make reasonable efforts to prevent unauthorized disclosure of client information, which means knowing exactly where that data goes and who can access it. A public AI tool or even a shared multi-tenant product, puts privileged material in the hands of a third party on terms the firm didn't fully negotiate. Three concerns are pushing firms toward infrastructure they control:
Not every "AI agent for law firms" runs the same way under the hood. They typically choose among three AI deployment patterns: public AI tools, private cloud tenants and true on‑premise setups. Each changes where data lives, how much training risk remains, how isolated the system is and who owns the audit trail.
Scroll horizontally to view all columns.
| Public AI tool (standard SaaS) | Private cloud tenant (single-tenant VPC/VNET) ) | True on-premise (firm servers | |
| Data location | Vendor's multi-tenant cloud; data may replicate across regions | Firm-controlled cloud account or dedicated tenant in AWS/Azure/GCP (often in a specific region) | Firm's own data center or colocation; no external replication |
| Training risk | Highest, unless explicit zero-data-retention/no-training clauses are contractually secured; inputs can be logged and used for model improvement | Medium–low, vendor may still operate the model, but data stays in firm's tenant; no-training must be contractually guaranteed and technically enforced | Lowest, data never leaves firm network; no external party can access prompts/completions for training |
| Isolation | Low, shared infrastructure and inference layer with other customers | Medium, logically isolated from other customers, but cloud provider still has underlying access | High, physically or logically disconnected from public internet; updates via controlled media/processes |
| Audit ownership | Vendor controls logs; firm gets limited export or dashboard access | Firm typically owns or co-owns logs; can export to SIEM and retain for audits | Firm fully owns and controls immutable audit logs; can integrate with internal compliance systems |
Firms can now run capable, private AI agents entirely on their own infrastructure. Below are the core building blocks that are production‑ready today.
These run open-weight models like Llama 3.3 and Qwen 2.5 without sending data to an external API.
Scroll horizontally to view all columns.
| Runtime | Best for |
| llama.cpp | Lightweight, runs on CPU, Apple Silicon, or GPU, best for smaller models and edge/on-prem setups with no GPU budget |
| vLLM | Built for production GPU serving; roughly 19x Ollama's throughput in benchmark testing thanks to PagedAttention, but needs real GPU infrastructure |
| Ollama | Simplest to set up, OpenAI-compatible API, ideal for prototyping or a single small team but degrades past 5-6 concurrent users |
This is the agent's memory layer, indexing matter documents for retrieval.
Scroll horizontally to view all columns.
| Vector Store | Best for |
| Qdrant | Production-grade vector DB with strong filtering, hybrid search, and multi-tenant isolation; widely used for RAG in regulated environments |
| Weaviate | Supports hybrid (BM25 + dense) search and rich metadata filtering; well-suited to legal corpora where exact terms and semantics both matter |
| Chroma | Simple, embeddable vector store; good for pilots, smaller knowledge bases, and internal tools where operational complexity must be minimal |
Connectors are typically implemented via secure APIs, service accounts, and on‑prem middleware that enforces matter‑level access controls.
Self-hosted systems still need patching and oversight. This gives the firm a clear record: who used the agent, on what data, with what output.
Not everything needs to be fully isolated. A common pattern: privileged matter work (contract review, due diligence, internal memos) stays on the local model, while general legal research routes to a vetted external API keeping cost and complexity down without exposing sensitive work.
Private Ai for law firms targets high-volume, repetitive workflows where time savings are measurable within 30 days.
SaaS pricing ranges from about $19.99/month (TheLawGPT, entry tier) to $3,000/user/year for enterprise platforms like Legora, which carries a 10-seat minimum (~$30,000+/year to start).
Self-hosted costs instead go into GPU hardware ($1,600 for a single-GPU workstation up to $10,000+ for multi-user 70B-class serving), the model itself (open-weight, free), a vector database (free self-hosted or a few hundred to low-thousands per month managed) and ongoing IT time.
Deployment by firm size
Before any agent, self-hosted or SaaS, touches live matter files, these controls need to already be in place.
Running an AI agent on your own servers isn't a future bet anymore; it's a live option, with real tools, real costs and real trade-offs already mapped out above. The question was never whether agentic AI for law firms works; it's whether your firm's size, workload and risk tolerance make self-hosting worth the investment or whether a well-chosen SaaS tool gets you 90% of the value with none of the infrastructure headache. Either way, AI for law firms is no longer optional, the firms figuring out this decision now will be the ones setting the pace in 2027.
Not sure which deployment model fits your firm? Book a 20-minute infrastructure assessment, we'll map your firm's size, data sensitivity and budget to tell you honestly whether self-hosting is worth it for you.
Yes. Law firms can deploy AI agents on firm-controlled servers or private infrastructure using self-hosted models and supporting software. This allows the agent to work with internal firm data without necessarily sending that data to a public AI provider.
Not automatically. Private deployment gives the firm greater control over where data is processed and stored, but the system still needs strong access controls, encryption, monitoring, audit logs, and human oversight.
Not necessarily. Self-hosting can reduce dependence on recurring SaaS fees, but the firm takes on hardware, deployment, maintenance, security, updates, and support costs. The economics depend on the firm's size, usage, workload, and infrastructure.
It can be, particularly when a firm handles sensitive documents or has enough recurring AI usage to justify its infrastructure costs. Smaller firms may find a private cloud or hybrid approach more practical than maintaining their own physical servers.
Tell me about the system or workflow you are working on.
30-minute intro call on Calendly
