How to Build HIPAA-Compliant AI Agents: A Technical Guide

Building AI agents for healthcare is a different discipline than building AI agents for most other industries. The technical fundamentals are the same - retrieval, reasoning, tool use, orchestration - but the compliance requirements change how you architect every layer of the system. Get it wrong, and your organization faces fines up to $1.5 million per violation category per year, along with reputational damage that is hard to recover from.

This guide covers the specific technical decisions you need to make when building AI agents that handle Protected Health Information (PHI) under HIPAA. It is not a general HIPAA primer or an overview of healthcare regulations. It assumes you already know the basics and need to understand how HIPAA requirements map to the architectural choices involved in building and deploying modern AI systems.

• AI agents introduce HIPAA risks that traditional software does not: training data exposure, prompt injection leading to PHI leaks, inference logging, and third-party API calls
• Every service in your AI pipeline that touches PHI must be covered by a Business Associate Agreement (BAA)
• De-identification using Safe Harbor or Expert Determination methods can significantly reduce your compliance surface area
• Architecture patterns like PHI isolation boundaries, encryption in transit and at rest, and tamper-evident audit logging are non-negotiable
• Common mistakes include sending PHI to LLM APIs without BAAs, logging raw prompts, and using patient data for fine-tuning without proper de-identification

Why HIPAA Matters Differently for AI Systems

HIPAA has been around since 1996. The Security Rule was finalized in 2003. Most healthcare IT teams have years of experience building HIPAA-compliant web applications, databases, and APIs. But AI agents introduce a fundamentally new class of risks that traditional application architectures simply do not have to deal with.

The HIPAA Security Rule Mapped to AI Architecture

The HIPAA Security Rule defines three categories of safeguards: Technical, Administrative, and Physical. Here is how each maps to the architectural decisions you will make when building an AI agent.

Technical Safeguards

In practice, this means implementing RBAC at multiple levels:

# Example: Access control policy for AI agent components
roles:
  clinical_user:
    permissions:
      - inference:read
      - patient_context:read
    phi_access: true
    audit_level: full
ml_engineer:
    permissions:
      - model:deploy
      - metrics:read
      - evaluation:read
    phi_access: false
    data_view: de-identified-only
    audit_level: full
system_service:
    permissions:
      - inference:execute
      - vector_db:query
      - ehr_api:read
    phi_access: true
    requires_baa: true
    audit_level: full

Administrative Safeguards

• Model behavior risks (hallucination, memorization, prompt injection)
• Data flow risks (PHI exposure through logging, caching, or third-party services)
• Access control risks (over-permissioned service accounts, shared credentials)
• Incident response risks (how you detect and respond to a PHI breach involving the AI system)

A Business Associate Agreement is a legal contract between a Covered Entity (or another Business Associate) and a Business Associate. It establishes what the Business Associate is permitted to do with PHI and what safeguards they must maintain. For AI workloads, you need BAAs with the following provider types:

• Cloud Infrastructure (AWS, GCP, Azure) - BAAs available, but not all services are in scope. Check the provider's HIPAA-eligible services list.
• LLM API Providers (OpenAI API, Anthropic API, Google Vertex AI) - Enterprise/API tiers typically support BAAs. Consumer tiers do not.
• Vector Databases (Pinecone, Weaviate managed) - Check whether managed cloud versions offer BAAs. Self-hosted avoids this requirement.
• Logging/Monitoring (Datadog, Splunk) - Both offer BAA-eligible tiers. Confirm before sending any PHI to their platforms.
• CI/CD and MLOps (Weights & Biases, MLflow managed) - If experiment tracking includes PHI samples, the platform needs a BAA.

Physical Safeguards

Architecture Patterns for HIPAA-Compliant AI Agents

There is no single correct architecture for a HIPAA-compliant AI agent, but there are patterns that make compliance significantly easier to achieve and maintain.

The PHI Isolation Boundary

The most effective architectural pattern is to establish a clear PHI boundary in your system. Inside the boundary, every component is HIPAA-hardened. Outside the boundary, no PHI ever flows.

The architecture works as follows: EHR/FHIR data flows into the AI Agent Orchestrator, which connects to the clinical user interface, a PHI-aware encrypted vector database, and a tamper-evident audit log. A de-identification service sits at the boundary edge.

The key principle: PHI stays inside the boundary. If data must cross the boundary - for example, to call an external LLM API - it passes through a de-identification service first, or the external service must be covered by a BAA and meet all HIPAA technical safeguard requirements. This boundary-based approach simplifies compliance by concentrating your security controls in a well-defined perimeter rather than trying to harden every individual component independently.

Data Residency and Encryption

PHI must be encrypted at rest using AES-256 and in transit using TLS 1.2+. But encryption alone is not sufficient. You also need to control where data physically resides and ensure that encryption keys are managed properly with appropriate access controls.

For AWS deployments (which is what we typically recommend for healthcare AI workloads), this means:

• S3 buckets with SSE-KMS encryption and bucket policies restricting access to specific VPCs
• RDS or DynamoDB with encryption at rest enabled and VPC-only access
• SageMaker endpoints running in a private VPC with no internet-facing access
• CloudWatch Logs encrypted with KMS keys you control (not AWS-managed keys)

# Example: S3 bucket policy enforcing encryption and VPC-only access
import json
bucket_policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyUnencryptedTransport",
            "Effect": "Deny",
            "Principal": "",
            "Action": "s3:",
            "Resource": [
                "arn:aws:s3:::phi-training-data-bucket",
                "arn:aws:s3:::phi-training-data-bucket/"
            ],
            "Condition": {
                "Bool": {"aws:SecureTransport": "false"}
            }
        },
        {
            "Sid": "RestrictToVPC",
            "Effect": "Deny",
            "Principal": "",
            "Action": "s3:",
            "Resource": [
                "arn:aws:s3:::phi-training-data-bucket",
                "arn:aws:s3:::phi-training-data-bucket/"
            ],
            "Condition": {
                "StringNotEquals": {
                    "aws:sourceVpce": "vpce-1a2b3c4d"
                }
            }
        }
    ]
}

Audit Logging Architecture

HIPAA requires audit logs retained for at least six years. For AI agents, this means logging at a granularity that most engineering teams are not used to.

Here is what a comprehensive audit log entry should look like for an AI agent inference call:

{
  "event_id": "evt_8f3a2b1c",
  "timestamp": "2026-03-05T14:32:18.442Z",
  "event_type": "ai_agent.inference",
  "user": {
    "id": "usr_clinician_4821",
    "role": "physician",
    "ip_address": "10.0.4.52"
  },
  "patient": {
    "mrn_hash": "sha256:a1b2c3d4..."
  },
  "agent": {
    "agent_id": "clinical_summary_agent_v2",
    "model": "gpt-4-turbo-hipaa",
    "model_version": "2026-02-15"
  },
  "data_accessed": [
    "medications_list",
    "lab_results_recent",
    "problem_list"
  ],
  "phi_categories_in_prompt": [
    "patient_name",
    "date_of_birth",
    "diagnosis_codes"
  ],
  "output_destination": "clinical_ui",
  "encryption": {
    "in_transit": "TLS_1_3",
    "at_rest": "AES_256_KMS"
  },
  "baa_verified": true
}

Store these logs in a tamper-evident system. AWS CloudTrail with log file validation enabled is one option. A dedicated SIEM (Security Information and Event Management) system with write-once storage is another. No one - not administrators, not engineers, not the AI agent itself - should be able to modify or delete audit records.

Putting It Together: A Reference Architecture

Here is how a complete HIPAA-compliant AI agent architecture looks in practice when deployed on AWS.

All compute runs inside a private VPC in a HIPAA-eligible region. The AI agent orchestrator, vector database, and all data services sit in private subnets with no public-facing access. The only public-facing component is an Application Load Balancer that terminates TLS and sits behind AWS WAF for additional protection.

Calls to AWS Bedrock or SageMaker use VPC Endpoints (PrivateLink) so they never traverse the public internet. They stay on the AWS backbone network, which is both faster and more secure. Before any data leaves the VPC for inference (if using an external API rather than Bedrock), it passes through a de-identification service. If you use AWS Bedrock within the same account and region covered by your AWS BAA, you can pass PHI directly, but you should still minimize the PHI in each prompt to what is strictly necessary for the task.

Audit logging uses CloudTrail with log file validation, writing to an S3 bucket with object lock enabled (Write Once Read Many). No one can modify or delete these records. Encryption is enforced everywhere: KMS customer-managed keys for data at rest, TLS 1.3 for data in transit, with key rotation on a defined schedule.

Key architectural decisions in this design: the private VPC with no public subnets for compute ensures that the AI agent orchestrator and all data services are fully isolated from the public internet. VPC Endpoints via PrivateLink for AWS AI services ensure calls to Bedrock or SageMaker stay on the AWS backbone network, which is both faster and more secure. The PHI scrubbing service at the boundary ensures that any data leaving the VPC for external services is properly de-identified. And the use of AWS-native services covered by the BAA minimizes the number of additional third-party agreements needed.

Common Mistakes That Break HIPAA Compliance

After working on multiple healthcare AI projects, including our AI Clinical Empowerment Platform and Curengo Rehabilitation Platform, we have seen the same compliance mistakes come up repeatedly. Here are the ones that cause the most trouble and how to avoid them.

Sending PHI to Third-Party LLM APIs Without a BAA

This is the single most common violation we see. A development team builds an AI agent that calls the OpenAI or Anthropic API, passes patient data in the prompt, and assumes that because the API uses HTTPS, they are compliant. They are not.

HTTPS encrypts data in transit, but the API provider is still processing PHI. That makes them a Business Associate, and you need a BAA before any PHI touches their systems. Consumer API tiers (like the standard OpenAI API without an enterprise agreement) do not come with BAAs.

The fix: either sign an enterprise agreement that includes a BAA, self-host the model, or de-identify data before it leaves your PHI boundary.

Logging Prompts That Contain PHI

Debugging AI agents means logging prompts and completions. But if those prompts contain "Summarize the medications for John Smith, DOB 04/15/1962, MRN 482910," you have just written PHI to your logging infrastructure. If that logging infrastructure is Elasticsearch in a shared cluster, or a SaaS platform without a BAA, or a developer's local machine, you have a problem.

The fix: implement a PHI-scrubbing layer between your agent and your logging pipeline. Log structured metadata (what type of data was accessed, which agent handled it, response latency) without logging the raw content.

import re
from typing import Any
class PHIScrubber:
    """Scrub PHI from log entries before they reach the logging pipeline."""
PATTERNS = {
        "ssn": re.compile(r'\b\d{3}-\d{2}-\d{4}\b'),
        "mrn": re.compile(r'\bMRN\s:?\s\d{4,10}\b', re.IGNORECASE),
        "phone": re.compile(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b'),
        "dob": re.compile(
            r'\b(DOB|Date of Birth)\s:?\s\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b',
            re.IGNORECASE
        ),
        "email": re.compile(r'\b[\w.+-]+@[\w-]+\.[\w.-]+\b'),
    }
@classmethod
    def scrub(cls, text: str) -> str:
        for category, pattern in cls.PATTERNS.items():
            text = pattern.sub(f'[REDACTED_{category.upper()}]', text)
        return text
@classmethod
    def safe_log_entry(cls, prompt: str, completion: str,
                       metadata: dict[str, Any]) -> dict[str, Any]:
        return {
            "prompt_length": len(prompt),
            "completion_length": len(completion),
            "prompt_scrubbed": cls.scrub(prompt),
            "phi_detected": any(
                p.search(prompt) for p in cls.PATTERNS.values()
            ),
            metadata
        }

# Example: Clinical text de-identification pipeline
from dataclasses import dataclass
from enum import Enum
class PHIType(Enum):
    PATIENT_NAME = "PATIENT_NAME"
    DATE = "DATE"
    LOCATION = "LOCATION"
    AGE = "AGE"
    CONTACT = "CONTACT"
    ID = "ID"
@dataclass
class PHIEntity:
    text: str
    phi_type: PHIType
    start: int
    end: int
    confidence: float
class ClinicalDeidentifier:
    """
    De-identification pipeline for clinical text.
Uses a combination of:
    1. Fine-tuned NER model for entity detection
    2. Rule-based patterns for structured identifiers (MRN, SSN)
    3. Context-aware replacement to maintain clinical meaning
    """
def __init__(self, ner_model_path: str):
        self.ner_model = self._load_model(ner_model_path)
def deidentify(self, text: str,
                   preserve_dates_as_offsets: bool = False
                   ) -> dict:
        """
        De-identify clinical text while preserving clinical meaning.
Returns:
            dict with 'text' (de-identified), 'entities' (found PHI),
            and 'method' (Safe Harbor or Expert Determination)
        """
        entities = self._detect_phi(text)
        deidentified_text = self._replace_entities(
            text, entities, preserve_dates_as_offsets
        )
return {
            "text": deidentified_text,
            "entities_found": len(entities),
            "entity_types": [e.phi_type.value for e in entities],
            "method": "safe_harbor",
            "confidence_min": min(
                (e.confidence for e in entities), default=1.0
            )
        }
def _detect_phi(self, text: str) -> list[PHIEntity]:
        """Run NER model + rule-based detection."""
        ner_entities = self.ner_model.predict(text)
        rule_entities = self._apply_rules(text)
        return self._merge_entities(ner_entities, rule_entities)

For teams building AI governance programs that extend beyond HIPAA into broader regulatory compliance, our AI governance services cover the full spectrum of requirements. For a broader view of AI governance requirements, see our enterprise AI governance and compliance framework guide. And for organizations that need hands-on help building compliant AI systems, our AI development services include compliance architecture as a core component of every healthcare engagement.

We've built HIPAA-compliant AI systems for healthcare clients including clinical documentation and rehabilitation platforms. See our healthcare case studies or talk to us about your project.