HIPAA Is an Architecture Decision, Not a Final-Week Review
There's a version of healthcare AI that looks finished and isn't. The model works, the demo is impressive, the client is excited — and then someone asks where the patient data goes when it hits the language model, who can see it, and whether there's a signed agreement with every vendor in the path. If those answers weren't designed in from the start, the honest response is a partial rebuild.
HIPAA — the US law governing protected health information — doesn't sit on top of a healthcare AI system as a compliance layer you add before launch. It shapes the architecture: where data lives, how it moves, who can touch it, what gets logged, and which vendors you're even allowed to use. Get those decisions right early and compliance is a manageable, ongoing discipline. Get them wrong and you're unwinding foundational choices under pressure.
This guide is engineering guidance, not legal advice. It walks through the technical backbone of a HIPAA-aware AI system — the encryption, audit logging, access control, and vendor-agreement decisions that a security auditor will actually examine — and it's honest about one thing that gets lost in marketing: "HIPAA-compliant" describes a whole program and posture, not a checkbox you tick on a single tool. Certification and attestation require an auditor, not a self-assessment. This sits directly on top of the same foundations as any AI agent handling sensitive data — HIPAA just adds legal teeth.
What Actually Counts as PHI
Before any architecture, you need to know what you're protecting. Protected health information is health information tied to an individual — and the identifier is broader than most teams assume. It's not only the diagnosis or the lab result. It's the eighteen categories of identifiers that, combined with health context, make data protected: names, dates more specific than a year, phone numbers, email addresses, medical record numbers, device identifiers, and more, down to full-face photographs and any other unique identifying number.
This matters enormously for AI, because the risk isn't only in the obvious fields. A free-text clinical note dictated into an AI medical scribe is dense with PHI — patient names, ages, addresses, family history — scattered through prose, not neatly boxed in database columns. An audit log that records "which patient record was accessed" is itself PHI. The retrieval context you feed a model in a medical LLM RAG system is PHI. Treating PHI as "the fields marked sensitive" while free text and metadata leak everywhere is one of the most common design failures in healthcare AI.
The practical consequence: your data-flow map has to trace PHI through every hop — ingestion, storage, the model call, the response, the logs, the analytics. Anywhere PHI travels, HIPAA's requirements travel with it.
The Third-Party LLM Problem: BAAs and Where Data Goes
Here is the challenge specific to AI, the one that doesn't exist in a traditional healthcare app. Most AI systems send data to a third-party model API — and if that data is PHI, you have a vendor in your compliance perimeter who now handles protected health information on your behalf.
Under HIPAA, any vendor that creates, receives, maintains, or transmits PHI for you is a business associate, and you need a signed Business Associate Agreement (BAA) with them. That includes your model provider. The critical, frequently-missed detail: standard and consumer tiers of most LLM APIs do not come with a BAA. The default terms often permit data to be retained or used to improve services, which is incompatible with handling PHI. You typically need a specific enterprise or healthcare arrangement — one that offers a BAA, excludes your data from training, and commits to defined retention and deletion.
So the architecture question becomes concrete: for each model call that touches PHI, is there a BAA covering that provider under the plan we're actually using? If the answer is no, you have three real options — move to a BAA-covered tier, de-identify the data before it leaves your environment, or run a model inside infrastructure you already control. This is exactly the third-party model risk that applies to any AI agent, but in healthcare the consequence of getting it wrong is a reportable breach rather than an awkward conversation.
De-identification and Data Minimisation Before the Model
The cleanest way to reduce risk is to reduce what leaves your controlled environment in the first place. Two techniques do most of the work.
Data minimisation means each model call carries only the PHI the task genuinely requires. If a summarisation prompt needs the clinical narrative but not the patient's full demographic block, don't send the demographic block. This is the least-privilege principle applied to data flow rather than accounts, and it shrinks the blast radius of any single failure.
De-identification goes further: stripping or replacing identifiers so the data reaching the model is no longer PHI. HIPAA recognises two routes — Safe Harbor, which removes the eighteen identifier categories, and Expert Determination, where a qualified statistician certifies that re-identification risk is very small. In practice, many AI systems tokenise or redact identifiers before the model call and re-attach them afterward inside the trusted boundary, so the model works on de-identified text and the identified record never leaves your environment.
Two honest caveats. First, de-identification is genuinely hard on free text — a name buried in a dictated note or a rare condition that is itself identifying can slip through automated redaction, so this needs validation, not blind trust. Second, if de-identification is imperfect or reversible in your design, treat the data as PHI and keep the BAA in place. De-identification is a risk-reduction technique, not a licence to be careless.
Encryption, Access Control and Audit Logging
Three controls form the backbone an auditor expects to see, and each has a specific AI wrinkle.
Encryption at rest and in transit. PHI should be encrypted in transit (TLS on every hop, including the call to the model API) and at rest (encrypted databases, object storage, vector stores, and backups). The AI-specific catch is the vector store: embeddings generated from clinical text are derived from PHI and often reconstruct it, so the vector database holding your RAG index needs the same encryption and access posture as the primary record store. Teams routinely lock down the database and forget the vector index sitting beside it.
Least-privilege access control (IAM). Every human and every service account gets the minimum access required, no more. The AI-specific failure mode is the over-permissioned service account: an AI agent given broad database credentials "to speed up the build" that were never narrowed. If your scribe integration can read every patient's record when it only ever needs the record for the active encounter, that's a finding waiting to happen. Scope every query to the authenticated context, and give the AI's service account exactly the tables and fields it needs — the same excessive-permissions discipline that applies to any agent, here with regulatory weight.
Comprehensive audit logging. HIPAA expects you to know who accessed which PHI, when, and what they did. For an AI system that means logging not just human access but every model call: which record was in context, which user or process triggered it, what action followed. This is what lets you answer an auditor's questions and detect misuse — and it's also why the logs themselves contain PHI and must be protected like any other PHI store. Tamper-evident, access-controlled, retained per policy.
| HIPAA requirement | What it means for an AI system | How to implement |
|---|---|---|
| Encryption in transit & at rest | PHI protected on every hop, including the model call and the vector index | TLS everywhere; encrypted DB, object storage, backups, and vector store |
| Business Associate Agreement | A signed BAA with every vendor that touches PHI, including the LLM and cloud provider | Use BAA-covered enterprise/healthcare tiers; sign the cloud provider BAA |
| Minimum necessary / least privilege | AI service accounts and users see only the PHI the task needs | Scoped service accounts; per-context query scoping; no admin keys |
| Audit controls | Record who accessed which PHI and every model call, when | Immutable, access-controlled logs covering human and model access |
| Access authorisation | Only authorised identities reach PHI or trigger the AI | Role-based access, MFA, per-session isolation, deprovisioning |
| Integrity & transmission security | PHI isn't altered or exposed in transit to third-party APIs | Validated payloads, minimised prompts, redaction before external calls |
HIPAA-Eligible Cloud, and the Program Around It
Most healthcare AI runs on cloud infrastructure, and the major providers offer HIPAA-eligible services — a defined subset of their services that may be used with PHI, provided you sign the cloud provider's BAA and configure them correctly. Two conditions there are load-bearing. You have to actually sign the BAA, and you have to stay within the eligible-service list; standing up PHI on a service outside that list, or without the agreement, doesn't become compliant just because it's the same cloud.
And this is the point where the honest framing matters most. HIPAA compliance is not a single tool you buy or a badge the cloud hands you. It's a program: administrative, physical, and technical safeguards; risk assessments; workforce training; incident-response and breach-notification procedures; and periodic review. The encryption and logging above are the technical safeguards — a necessary part, not the whole. A tool that measures or enforces some controls is helpful, but measuring controls is not the same as being compliant. That's why certification and attestation — whether a HIPAA assessment or a HITRUST certification, which many healthcare buyers now expect — require an independent auditor. No engineering team should tell a client they are "HIPAA-certified" on the strength of their own architecture; that's a conversation for qualified counsel and a third-party assessor.
One related boundary worth flagging: if your AI does more than assist — if it diagnoses or drives clinical decisions autonomously — it may fall under the FDA's Software as a Medical Device (SaMD) framework, an entirely separate regulatory track from HIPAA. Keeping the AI in an assistive, human-reviewed role reduces that exposure and is better clinical practice regardless.
Questions to Ask Your Development Team
"For every model call that touches PHI, is there a signed BAA covering that provider under the plan we're on?" The answer should name the provider, the tier, and confirm data is excluded from training. "We use the standard API" is a red flag.
"Show me the PHI data-flow map." They should be able to trace PHI through ingestion, storage, the model call, the response, and the logs — including the vector store. If free text and metadata aren't accounted for, they haven't mapped it properly.
"What does the AI's service account have access to, and how is it scoped?" You want least privilege and per-context scoping, not a broad or admin-level credential.
"What's logged, and are the logs themselves protected as PHI?" Every model call and PHI access should be logged, immutably and access-controlled.
"Are we de-identifying before the model, and how is that validated?" If they claim de-identification, ask how they test it against free-text leakage.
"Who signs off on the compliance posture — is there an auditor or counsel involved?" The right answer acknowledges that the engineering is necessary but not sufficient, and that attestation needs an independent party.
What Good Looks Like
PHI is mapped end to end. Every hop is accounted for, free text and metadata included, and PHI is encrypted at rest and in transit throughout.
Every PHI-touching vendor has a BAA. The model provider and the cloud provider both, on tiers that exclude your data from training and define retention.
The model sees the minimum. Data is minimised and, where feasible, de-identified before it leaves the trusted boundary — with validation, not blind faith.
Access is least-privilege and audited. Scoped service accounts, per-session isolation, MFA, and immutable logs covering both human and model access to PHI.
The AI stays assistive. A licensed human remains in the decision loop, which keeps both clinical safety and regulatory exposure in check.
Compliance is treated as a program. The architecture is one part; risk assessment, training, incident response, and independent attestation are the rest — and everyone involved knows the difference.
What It Costs and How Long It Takes
Building HIPAA architecture in from the start adds real but bounded effort — the encryption posture, the scoped IAM, the audit logging, the de-identification layer, and the BAA legwork are weeks of work layered alongside core development, not a separate project. The honest caveat is the one from the pillar: healthcare AI has a higher floor than general software, and this is a large part of why. That floor is also the barrier that keeps the work defensible.
The expensive path is the other one. Retrofitting HIPAA architecture onto a working prototype — rescoping data access, rebuilding the logging model, re-routing PHI away from a non-BAA API — typically costs several times what designing it in would have, because these are foundational choices rather than surface features. And beyond the build, certification or attestation (a HIPAA assessment or HITRUST) is its own engagement with an independent auditor, on its own timeline. The same compliance-automation discipline that underpins SOC 2 and ISO programs applies here — HIPAA simply raises the stakes and narrows the acceptable answers.
Related guides
- Healthcare AI development: the full map
- FHIR, HL7 and EHR integration
- Medical LLMs, RAG and human-in-the-loop
- AI medical scribe: speech to SOAP notes to EHR
- AI agent security: risks and mitigations
- AI compliance automation: SOC 2 and ISO
- Our AI development services
We Build Healthcare AI That's HIPAA-Aware From the First Decision
We treat HIPAA architecture as part of scoping, not a review bolted on at the end. That means mapping PHI before we write code, choosing BAA-covered model and cloud tiers, scoping access to least privilege, logging every PHI touch, and de-identifying data before it leaves the trusted boundary where we can. We're also honest about the boundary of our role: we build the technical safeguards, and we'll tell you plainly where you need qualified counsel and an independent auditor to certify the program around them.
If you're building a healthcare AI system and want to get the compliance foundations right before they're expensive to change, that's exactly the conversation worth having early.
Talk to us about your platform — no commitment, just a conversation.
Frequently Asked Questions
Does sending patient data to an LLM like GPT or Claude break HIPAA?
Not automatically, but it requires care. Sending PHI to a third-party model API makes that provider a business associate, so you need a signed Business Associate Agreement with them — and the standard or consumer API tiers of most providers do not include one. You typically need an enterprise or healthcare arrangement that offers a BAA, excludes your data from training, and defines retention. The alternatives are to de-identify the data before it leaves your environment, or to run a model inside infrastructure you already control. What breaks HIPAA is sending PHI to a provider with no BAA in place — the design, not the technology, is the deciding factor.
Can a single tool or platform make my healthcare AI "HIPAA-compliant"?
No, and it's worth being precise about this. "HIPAA-compliant" describes a whole program and posture — administrative, physical, and technical safeguards, risk assessments, training, and incident response — not a property of one tool. A vendor can offer HIPAA-eligible services or sign a BAA, which are necessary building blocks, but assembling them correctly and running the surrounding program is your responsibility. A tool that measures or enforces certain controls helps, but measuring controls is not the same as being compliant. Certification or attestation requires an independent auditor, not a self-declaration.
What exactly counts as PHI in an AI system?
Protected health information is health information tied to an individual through any of eighteen identifier categories — names, dates, contact details, medical record numbers, device identifiers, and more. In an AI system, PHI is broader than the obvious database fields: free-text clinical notes, the retrieval context fed to a model, embeddings in a vector store, and audit logs recording which record was accessed all contain or derive from PHI. The common failure is protecting the marked-sensitive fields while free text and metadata leak elsewhere. Your data-flow map has to trace PHI through every hop, including the model call and the logs.
Do we need a BAA with our cloud provider as well as the model provider?
Yes, if PHI touches either. The major cloud providers offer HIPAA-eligible services and will sign a BAA, but two conditions are load-bearing: you have to actually sign it, and you have to keep PHI on the eligible-service list and configure those services correctly. The same logic applies to the model provider and any other vendor in the PHI path. A practical rule: any vendor that creates, receives, maintains, or transmits PHI on your behalf is a business associate and needs a BAA.
Is de-identifying data before the model enough to avoid HIPAA obligations?
It can be, but only if the de-identification is genuine and validated. HIPAA recognises Safe Harbor (removing the eighteen identifier categories) and Expert Determination (a statistician certifying very low re-identification risk). The difficulty is that de-identification is hard on free text — a name buried in a dictated note, or a rare condition that is itself identifying, can slip past automated redaction. If your de-identification is imperfect or reversible, treat the data as PHI and keep the BAA in place. De-identification is a strong risk-reduction technique, not a blanket exemption you can assume without testing.
How does HIPAA relate to SOC 2, HITRUST, and FDA requirements?
They're different frameworks that often coexist. SOC 2 and ISO 27001 are general security-and-controls attestations — a strong foundation that HIPAA sits on top of, covered in our compliance automation guide. HITRUST is a certifiable framework popular in US healthcare that many buyers now expect, and it incorporates HIPAA requirements. The FDA's Software as a Medical Device (SaMD) framework is separate again — it applies if your AI diagnoses or drives clinical decisions autonomously rather than assisting a human. HIPAA governs how you handle PHI; the others govern security posture and, in the FDA's case, clinical function. Keeping the AI assistive and human-reviewed reduces FDA exposure, while HIPAA and a certification like HITRUST address the data and security side.