HIPAA-Compliant AI Agents: The Architecture Guide for Healthcare CTOs

Healthcare CTOs are being asked to approve AI agents that read clinical notes, listen to patient conversations, summarize discharge records, and answer patient questions. The pressure to say yes is real, and so is the consequence of saying yes to an architecture that cannot pass a HIPAA review. This guide is the architecture we use when we build these systems. It is written for the person who has to sign.

What HIPAA actually asks of an agent

It helps to separate what HIPAA requires from what people assume it requires. The Privacy Rule governs when PHI may be used and disclosed, and it introduces the minimum necessary standard: use only the PHI needed for the purpose. The Security Rule requires administrative, physical, and technical safeguards for electronic PHI, and the technical safeguards are the ones an architect has to design for: access control, audit controls, integrity, authentication, and transmission security.

An AI agent is, from HIPAA's point of view, just another system that uses and discloses PHI. It has to be covered by the same risk analysis, the same policies, and the same safeguards. What makes agents harder is that they do three things ordinary applications do not: they send data to a model that may be operated by a third party, they call tools with side effects based on their own reasoning, and they produce free-text output that may contain PHI the user did not explicitly request. Each of those needs a design decision, and the rest of this guide walks through them.

The BAA chain

A business associate is any vendor that creates, receives, maintains, or transmits PHI on your behalf. For an agent, that chain typically includes the cloud provider, the model provider, the vector database if it is hosted, the speech-to-text service if there is audio, any observability vendor that receives traces, and any email or messaging service that delivers outputs. Every one of them needs a signed BAA before PHI flows.

Two things go wrong here repeatedly. The first is assuming a platform agreement covers everything the platform sells. Model providers publish BAAs that name the covered services explicitly and exclude others, including beta features, consumer tiers, and sometimes specific products. If the product your team is using is not named, it is not covered. The second is forgetting the observability path. A tracing tool that captures full prompts and responses is receiving PHI. It either needs a BAA or it needs to receive redacted data only.

The practical control is a registry: one document listing every vendor in the PHI path, the BAA reference, the covered services, the data residency, and the retention terms. It is the first thing we produce in the assessment phase for any healthcare engagement and the first thing an auditor asks for.

A reference architecture

The following layout is the structure we start from. The boundaries matter more than the specific products.

[Clinician / Patient client]
        |  TLS, authenticated session
        v
[API gateway]  authn, per-user authz, rate limits, request id
        |
        v
[Agent orchestrator]  runs on your cloud, in your VPC
   |         |            |
   |         |            +--> [Audit log]  append-only, no raw PHI
   |         |
   |         +--> [Tool layer]  EHR read/write, scheduling, documents
   |                  scoped credentials per user per call
   |
   +--> [De-identification service]  tokenizes identifiers,
   |         keeps the map in your KMS-backed store
   |
   +--> [Model provider]  under BAA, zero retention, private endpoint
   |
   +--> [Retrieval index]  encrypted, access-filtered, PHI-classified

The key property is that the model provider sits at the edge, behind de-identification and behind a private network path, and everything that can hold PHI (the orchestrator state, the retrieval index, the tool layer, the audit log) lives inside your cloud under your keys. The agent is not a service you call from your EHR. It is a component of your environment that calls a model.

Encryption: in transit, at rest, and in the prompt

Transport encryption and disk encryption are baseline and nobody argues about them. The places that get missed are the ones that did not exist before agents.

Prompts and completions are PHI stores. If the orchestrator persists conversation state, that state contains PHI and needs the same encryption and access control as the EHR record it came from. Keep it in a store you control, encrypted with keys in your KMS, with a retention period that matches the purpose and no longer.

Retrieval indexes are PHI stores. A vector index built from clinical documents contains PHI in embedded form and often in the chunk text stored alongside. Encrypt it, classify it as PHI, filter every query by the requesting user's permissions, and never let the index be the place where access control is enforced for the first time.

Model calls should traverse a private path. Most cloud model endpoints can be reached over a private link from your VPC rather than the public internet. Use it. Combine it with a zero data retention configuration from the provider so prompts are not stored for training or debugging on their side.

Field-level encryption for the identifiers that matter most. For patient identifiers, member numbers, and similar fields, application-level encryption on top of disk encryption means a database read by a compromised service account still does not yield usable identifiers.

Audit logging for agents

HIPAA's audit control requirement predates agents, but its intent is clear: you must be able to reconstruct who accessed which PHI and what they did with it. For an agent, the access is indirect. A clinician asks a question, the agent reads three records and calls two tools, and the clinician sees a summary. The log has to capture the whole chain.

The minimum we log for every agent interaction:

• The authenticated user and the session, with a request identifier that ties everything below together
• Every record or document the agent read, by identifier, and the permission check that allowed it
• Every tool call, with the tool name, the parameters (hashed or redacted where they contain PHI), and the result status
• The model, the model version, and a hash of the prompt template in use
• What the user did with the output: accepted, edited, rejected, or exported

Two rules follow from that list. First, never write raw PHI to the log. Log identifiers, hashes, and metadata, and keep the log queryable by the identifiers so an investigator can join back to the source records. Second, write to an append-only store. A log the application can edit is not evidence. The retention period should match HIPAA's documentation retention requirement of six years and your own policies, whichever is longer.

De-identification before the model

The cleanest way to reduce the PHI exposure of a model call is to not send PHI. HIPAA recognizes two methods for de-identification. Safe Harbor removes a defined list of identifiers, including names, dates more specific than the year, contact details, record numbers, and biometric identifiers. Expert Determination has a qualified expert certify that the risk of re-identification is very small for a given dataset and method.

For agents, the pattern we use is tokenization inside the boundary. The de-identification service replaces identifiers in the input with stable tokens, stores the mapping in a KMS-backed table, and sends the tokenized text to the model. The model's output is passed back through the service, which re-identifies the tokens before the result reaches the user. The model never sees the identifiers, and the mapping never leaves your environment.

This does not work for every task. A scribe summarizing a conversation that names the patient's daughter needs to keep that relationship, and a discharge summary has to state medications and dates. Where full de-identification would break the task, the answer is minimum necessary: send the fields the task requires and nothing else, under the BAA, over the private path. The de-identification service still runs, but in a mode that redacts the fields the task does not need.

Access control and minimum necessary for agents

An agent must act with the identity of the person using it, not with a service account that can read every record. This is where many designs quietly fail. The orchestrator is given broad credentials for convenience, and every user of the agent effectively inherits them.

The control is scoped credentials per tool call. When the agent calls the EHR, it presents a token that represents the user, and the EHR applies the same permission model it would apply if the user were clicking through the interface. When the agent queries the retrieval index, the query carries the user's permission set and the index filters on it. Nothing the agent can reach is reachable by the user through any other path than the one the user is already authorized for.

The same principle applies to writes. An agent that can create orders or update records should require explicit confirmation from the user before the write is committed, and the confirmation should be part of the audit record. A platform shaped like Curengo, where the model generates assessments and discharge summaries, shows the rule: those outputs are drafts for a clinician to sign, and any order is placed by the clinician inside the platform's permission model, not by the model.

Prompt injection and PHI

HIPAA compliance does not make an agent secure, and this is the point where a healthcare CTO has to look past the compliance checklist. An agent that reads documents is reading untrusted input. A referral letter, a scanned form, or a retrieved chunk can contain text designed to change what the agent does: exfiltrate the record it is reading, call a tool it should not, or return fabricated content.

The defenses are architectural. Treat all retrieved and tool-returned content as data, never as instructions, and structure prompts so the boundary is explicit. Restrict tools to the minimum set the task needs, and make destructive tools require confirmation. Test the agent against injection cases as part of the evaluation harness, and have someone attempt to break it before it reaches patients. Our security practice runs this testing for healthcare clients, and it regularly finds paths that a compliance review would never surface.

How the controls map onto three systems we built

FastCare is a mobile-first clinical scribe and wound assessment platform. It converts clinician and patient conversations into structured, coded notes using domain-tuned NLP and measures wound dimensions from mobile images with computer vision and calibrated image analysis. The PHI exposure is high: audio, images, and free-text clinical content. For a system of this shape, the controls we apply are the ones above: capture on the device over an authenticated session, processing inside the platform boundary, integration with the EHR under the clinician's identity, and the scribe output treated as a draft the clinician confirms, with the confirmation in the audit log. FastCare's structured, coded output and its audit-ready documentation workflow are what make that last control practical.

Curengo is a rehabilitation and post-acute care platform that unifies a rehabilitation EMR, AI-generated assessments and discharge summaries, IoT vital monitoring, pharmacy and billing, and patient and consultant applications. Here the architecture question is breadth: many data types, many user roles, and devices streaming vitals. For a system of this shape, the controls are role-based access that separates doctors, therapists, nurses, patients, and referring consultants, applications that see only the scope their role permits, AI features that generate drafts inside the same permission model as the humans reviewing them, and a device ingestion path that authenticates the source and files readings against the patient record under the same access rules as everything else.

Hello Kidney is a conversational system that interprets patient questions about kidney health, retrieves relevant medical information, and answers in plain language. The design choice that matters for a system of this shape is what it holds. The knowledge base should be medical reference content, not patient records, which keeps the retrieval index outside the PHI classification. Anything a patient types should be treated as potential PHI: encrypted, retained only as long as the session requires, and kept out of the knowledge base. Keeping the exposure that narrow is the whole architecture.

A checklist for the CTO

Before approving an agent that touches PHI, confirm the following. Each item maps to a control an auditor will ask about.

• A vendor registry exists listing every system in the PHI path with its BAA, covered services, residency, and retention
• The model provider's BAA names the exact product and tier in use, and zero data retention is configured
• Conversation state, retrieval indexes, and logs are classified as PHI stores and encrypted under your keys
• Model calls traverse a private network path from your VPC
• The audit log captures user, records read, tool calls, model version, and user action, with no raw PHI, in an append-only store
• De-identification runs before the model wherever the task allows, and the re-identification map stays inside your boundary
• The agent acts under the user's identity for every tool call, and writes require explicit confirmation
• Injection and tool-misuse cases are in the evaluation harness and have been tested by someone trying to break the system
• The risk analysis, policies, and training required by the Security Rule have been updated to include the agent

An architecture that satisfies this list is not slower to build than one that does not. It is faster, because the review at the end becomes a formality instead of a redesign. Our application delivery team draws this architecture in the assessment phase for every healthcare engagement, and our healthcare page covers the sector work in more depth. If you have an agent in flight and want the design reviewed against this list, contact us.