Skip to content

CWE-1426: Improper Validation of Generative AI Output

Overview

This weakness occurs when an application treats the output of a generative AI model - a text response, a tool-call argument, generated code, a generated query - as trusted because a model produced it, and passes that output to a sensitive operation without the validation the same operation would require for any other untrusted input. It shows up wherever an LLM's response reaches code execution, a shell, a database, the filesystem, a rendered page, or another API call.

Relationship to Other CWEs

CWE-1426 is a Base-level entry under the Pillar CWE-707 (Improper Neutralization) in MITRE's research view, and MITRE's mapping guidance for it is Discouraged: the entry "should not be used to map to real-world vulnerabilities", because it is expected to change as the field matures and because it is frequently misinterpreted. MITRE's own comment is the part worth acting on - the entry covers validation of output only, is easily reached for when the real subject is prompt injection, and "analysts should closely investigate the root cause to ensure it is not ultimately due to other well-known weaknesses". So a scanner result carrying this number is a prompt to work out which weakness the finding actually is, and to file that one instead.

Most real-world findings filed under CWE-1426 are actually an existing sink-specific weakness where the untrusted source happens to be a model instead of a user-supplied form field:

When a finding matches one of these patterns, apply that CWE's remediation directly, treating the model as the untrusted source in place of user input - the fix does not change because the input originated from a model. Path traversal is the case the examples here follow: both language pages lead with a tool call whose filename argument is a traversal payload.

What is specific to CWE-1426, and not already covered by those pages, is the mental-model gap of treating "the model produced this" as equivalent to "this is safe" or "a developer authored this," plus two mitigations particular to generative AI output: constraining the shape of generated output with structured or schema-constrained generation, and validating tool-call arguments as untrusted API input rather than as pre-authorized instructions.

This page is the output-side counterpart to CWE-1427 (Improper Neutralization of Input Used for LLM Prompting), which covers untrusted content flowing into a model's context (prompt injection). CWE-1426 covers what happens to untrusted content flowing out of the model. A single incident can involve both: injected content changes what the model outputs, and the application then fails to validate that output before acting on it.

Risk

High: A model can be manipulated - through the original prompt, a poisoned document it retrieves, or a chained tool result - into producing output that looks legitimate but carries an injection payload, an out-of-range value, or a path-traversal filename. If that output reaches a sink unchecked, the result is the same as any other untrusted-input injection: code execution, data exposure, unauthorized data changes, or an unauthorized action taken through a tool call.

Remediation Steps

Core Principle: Validate and constrain model output at every sink exactly as you would validate any other untrusted input reaching that same operation; never treat "the model produced this" as a substitute for a check.

Trace the Data Path

  • Source: The model's text response, a tool-call argument (tool_use.input or equivalent), or a filename/path embedded in generated output or a tool result
  • Sink: Code execution or an interpreter, a shell, a database driver, a filesystem call, an HTML/DOM renderer, or a downstream API call made using the model's output
  • Data Flow / Missing Controls: Free-text parsing that extracts structured values without a schema, tool handlers that execute on input fields without re-validating type, range, or authorization, and filenames or paths taken from output without containment checks

Route Sink-Specific Findings to the Right CWE (Primary Defense)

For each point where model output reaches a sink, identify the sink type and apply the existing, sink-specific guidance, with the model treated as the untrusted source:

  • Code execution or an interpreter -> code injection guidance
  • Shell command construction -> OS command injection guidance
  • SQL or query construction -> SQL injection guidance
  • HTML/DOM rendering -> XSS guidance
  • Filenames or paths in generated output or tool results -> path traversal guidance: canonicalize and confirm containment within an allowlisted directory before any file operation

Validate Tool-Call Arguments as Untrusted Input

  • Confirm there is a server-side validation and authorization step between "the model requested this" and "the action executes" - where that step is missing altogether, the finding is an authorization weakness in its own right: CWE-862 (Missing Authorization) when no check exists, CWE-285 (Improper Authorization) when one exists and decides wrongly. CWE-1427 is not the number for it - MITRE scopes that entry to neutralization during prompt generation, not to what the application does with a tool call afterwards
  • Type-check and range-check every argument the model supplied, the same way an HTTP request body would be validated
  • Re-verify authorization for the specific resource or action using the real authenticated caller, never a value the model included in its own output

Constrain Output Shape (Defense in Depth)

  • Use structured or schema-constrained generation so the model's response is validated against a schema before your code parses it, instead of free-text output parsed with regex or string matching
  • Schema constraints narrow, but do not eliminate, the injection surface for whatever consumes the output downstream - a schema-valid string can still contain a malicious payload for its eventual sink

Do Not Treat Model Self-Reports as Verification

  • A model claiming in its own output that it "verified," "checked," or "confirmed" something is not evidence that verification occurred - a model can be instructed or manipulated into making a false claim
  • Perform independent, code-level verification for anything security-relevant, rather than trusting the model's narrative about its own actions

Test with Malicious or Mocked Model Output

  • Feed a mocked model response containing an out-of-schema value, a path-traversal filename (../../etc/passwd), or an injection payload in a text field, and confirm the validation layer at the sink rejects it independent of what the model intended
  • Re-scan with the security scanner to confirm the finding is resolved once the sink-specific fix is applied

Common Vulnerable Patterns

// VULNERABLE - pseudo-code
modelResponse = callModel(prompt)
toolArgs = parseToolCall(modelResponse)
filename = toolArgs.filename
writeFile(outputDir + "/" + filename, toolArgs.content)
// Attack: model is manipulated (directly or via a poisoned document it read)
// into producing filename = "../../../etc/cron.d/malicious"
// Result: file written outside the intended directory

Why this is vulnerable: modelResponse is untrusted input arriving through a channel that feels internal. Structurally it is no different from a value pulled out of an HTTP request, but it comes back from what looks like a call into your own system, so the boundary where validation belongs is invisible at the call site - and every field of the tool call sits on the far side of it.

The party controlling that input is also not necessarily the person using the application. Anything the model read on the way to producing this response - a retrieved document, a fetched page, an earlier tool result, a file uploaded by someone else - can carry the text that shapes it. "Our users are authenticated and trusted" therefore does not bound the risk, and the standard to hold the value to is the one its sink would demand of any anonymous input: a filename reaching the filesystem needs the same containment check a filename from a web form would need.

Secure Patterns

// SECURE - pseudo-code
modelResponse = callModel(prompt, outputSchema=ToolCallSchema)
toolArgs = validateAgainstSchema(modelResponse, ToolCallSchema)

// Reject a filename carrying any path component rather than rewriting it:
// basename("../../etc/passwd") is "passwd", which writes a different file
// than the call asked for and leaves the containment check below with
// nothing it can catch
safeName = basename(toolArgs.filename)
if safeName != toolArgs.filename:
    reject("filename must not contain a path")

resolvedPath = canonicalize(join(outputDir, safeName))
if not isWithinDirectory(outputDir, resolvedPath):
    reject("path escapes output directory")

writeFile(resolvedPath, toolArgs.content)

Why this works: Schema-constrained generation forces the model's response into an expected shape before it is parsed, and the filename is then held to the same standard a filename from an untrusted user would be - rejected outright if it carries a path, rather than trusted because a model produced it.

Rejecting is the part worth copying, because the obvious alternative silently succeeds. basename() on its own is not a check: with only it in the way, the traversal filename is written under a harmless name rather than refused, and the test step above expects a rejection.

Common Pitfalls

  • Confirming the model's response matches the expected JSON structure, then using the fields directly. A schema checks shape, not whether an amount is in range, a filename escapes a directory, or a string contains an injection payload for its eventual sink.
  • Executing a tool action because the model decided to call it, without a separate server-side authorization check against the real authenticated caller. The model's reasoning is not an access-control decision.
  • Accepting text in the model's response stating that an input was checked or a value was safe, instead of performing that check in code.

Language-Specific Guidance

  • JavaScript/TypeScript - structured outputs with Zod, tool-call argument validation, and filename containment for Node.js LLM applications
  • Python - structured outputs with Pydantic, tool-call argument validation, and filename containment for Python LLM applications

Additional Resources