Konstantin Semenenko
July 6, 2026
4
minutes read
Regulated teams (healthcare, finance, legal, government) can use AI without sending sensitive data to a third-party cloud, but it takes deliberate architecture, not a settings toggle. The main patterns are on-prem or private-cloud deployment, bring-your-own-model with local inference, and strict data-isolation boundaries around what a model is ever allowed to see. The goal is that regulated data never leaves your control while the AI still does useful work.
A regulated team can absolutely use AI without handing its sensitive data to a public API, but privacy is an architectural property, not a checkbox. Keeping regulated data on your side means deciding, deliberately, where the model runs, what it is allowed to see, and where the boundary sits between your protected data and any external service. The three building blocks are deployment (on-prem or private cloud instead of a shared API), model choice (a model you host, including local open-weight models), and isolation (hard boundaries controlling what data ever reaches the model). Get those right and AI does real work while HIPAA, GDPR, or your regulator's requirements stay satisfied.
We build AI for teams with real compliance constraints, so this is the practical architecture, what the patterns are, when each fits, and where the boundaries have to sit.
The default way to add AI, call a hosted model API, sends your input data to a third party. For most products that is fine. For a regulated team it is often a non-starter, because the data in the prompt, a patient record, a financial statement, a privileged legal document, is exactly the data a regulator says cannot leave your controlled environment or be processed by an unapproved sub-processor.
The problem is not that hosted models are insecure. It is that sending regulated data to them creates a data-processing relationship your compliance framework may not permit, or permits only under specific contractual and technical conditions. So the architecture question for a regulated team is not "which model is best," it is "how do I get AI's value without the regulated data crossing a boundary it is not allowed to cross." Everything below is an answer to that.
The most direct pattern is to run the model inside your own environment, on your own servers or in your private cloud tenant, so the data never leaves infrastructure you control. Inference happens on your side of the boundary, and no prompt is sent to an external API. For the strictest requirements, this is often the only architecture that passes review, because it removes the third-party processor entirely.
The trade-off is that you own the operational burden: provisioning, scaling, monitoring, and updating the inference infrastructure. That cost is real, and it is why this pattern fits teams whose compliance requirements genuinely demand it, rather than being the default for everyone. When the requirement is real, though, the control it provides is exactly what the regulator is asking for.
You do not always need a frontier model. For many regulated workflows, an open-weight model running locally does the job, and running it locally means the data never leaves the machine. Tools in this space, including our own dotPilot, run AI agents entirely on your hardware: a local-first desktop app built on Microsoft Orleans and the Microsoft Agent Framework, one control plane for multiple agents, with adapters for Codex, Claude Code, GitHub Copilot, and local runtimes, so data stays on the device by design rather than by policy.
This pattern fits teams that want AI assistance on sensitive material without any network egress of that material. The model may be smaller than a hosted frontier model, but for classification, extraction, summarization, and structured tasks, capable local models are often more than sufficient. And when a task genuinely needs a larger model, a well-designed system can route only the non-sensitive parts outward while keeping the regulated data local, which is the isolation pattern below.
The deepest lever is not where the model runs, it is what the model is ever allowed to see. Even with a hosted model, careful architecture can keep regulated data out of the prompt: redact or tokenize sensitive fields before the model sees them, retrieve only the minimum context a task needs, and enforce hard boundaries so protected data and external calls never occur in the same operation. Sensitive-data anonymization for compliance workloads is a real engineering component here, and one we maintain as an open-source library for exactly this. The model works on a sanitized view; the regulated data stays behind the boundary.
This is where most of the real engineering lives, because it is per-workflow. It means deciding, for each task, exactly what the model needs to see to be useful and ensuring it sees nothing more. Done well, data isolation lets a team use more capable infrastructure for the parts of a workflow that carry no regulated data, while the sensitive parts never cross the line. Done poorly, it is the exact hole that ships a data-isolation failure to production, the same failure mode that breaks agent systems generally.
The patterns are not mutually exclusive, and the right mix depends on how strict the requirement is. As a rough guide: the strictest data-residency and no-third-party-processor requirements point to on-prem or private-cloud with local models, where nothing leaves your environment. A requirement to keep specific regulated fields protected, but not to avoid hosted models entirely, points to data isolation with redaction and tokenization around a hosted model. And a requirement driven by full privacy or air-gapped operation points to fully local inference with a tool that runs on your own hardware.
The engineering discipline is the same across all three: know exactly where your regulated data is at every step, and make the boundary a designed, enforced thing rather than an assumption. Compliance frameworks like HIPAA and GDPR do not ban AI, they constrain where regulated data can go, and each of these patterns is a way to satisfy that constraint while still getting the work done.
Regulated teams do not have to choose between using AI and protecting their data. They have to architect for both: run the model where compliance allows (on-prem, private cloud, or local), choose a model they can host when residency demands it, and isolate regulated data so the boundary it cannot cross is enforced by design. Privacy is not a feature you turn on; it is the shape of the system you build.
The pattern under all of it is the one behind every serious AI deployment: the model does the work, and the architecture around it, here the deployment, model choice, and data boundaries, is what makes it safe to use. If you are trying to bring AI into a regulated environment without your data leaving your control, that architecture is where our AI Discovery work starts.
Can regulated companies use AI without sending data to the cloud? Yes. By running models on-prem or in a private cloud, using local open-weight models, or isolating regulated data so it never reaches an external model, a regulated team can use AI while keeping sensitive data inside its controlled environment.
What is BYOM (bring-your-own-model)? BYOM means hosting a model you choose, often an open-weight model, in your own environment instead of calling a third-party API. Combined with local inference, it keeps prompt data on your infrastructure, which suits strict data-residency requirements.
Does HIPAA or GDPR prohibit using AI? No. They constrain where regulated data can go and who can process it. AI is compliant when the architecture keeps regulated data within allowed boundaries, through local inference, private deployment, or data isolation and redaction around any external model.
What is data isolation in an AI system? Designing each workflow so the model only ever sees the minimum, sanitized data it needs, by redacting or tokenizing sensitive fields and preventing protected data and external calls from occurring together. It keeps regulated data behind a boundary even when a hosted model is used.
Is a local open-weight model good enough for regulated work? Often yes. For classification, extraction, summarization, and structured tasks common in regulated workflows, capable local models are frequently sufficient, and they keep data on-device. Larger models can still be used for non-sensitive parts of a workflow through careful isolation.
