A complete LLM security audit checklist has ten controls: an asset inventory and threat model, then one check each for prompt injection, system prompt leakage, output handling, agent permissions, data and retrieval access, model supply chain, gateway infrastructure, consumption limits, and adversarial testing. Together they cover all ten risks in the OWASP Top 10 for LLM Applications 2025.
Each control below lists what to check, the evidence an auditor should collect, and the condition that fails it. This is the technical layer of an audit. The program layer (governance, legal, regulatory, training) sits in the OWASP GenAI Security Project’s LLM Applications Cybersecurity and Governance Checklist, version 1.1, dated May 7, 2024. Testing phases and evidence standards for the adversarial work are set out in the AI red team methodology guide.
The 10-point LLM security audit checklist
1. Inventory components and draw trust boundaries
Check: every model endpoint, AI gateway, vector store, embedding pipeline, tool, plugin and MCP server in scope, each with an owner, a version and the data classification it can reach. The UK NCSC’s Guidelines for secure AI system development place threat modelling under secure design, the first of their four lifecycle areas.
Evidence: a component list and a data-flow diagram marking every point where untrusted content (user turns, retrieved documents, tool responses) enters the context window.
Fail if: any tool or data source the model can reach is missing from the inventory.
2. Prompt injection: limit impact, not just inputs (LLM01)
OWASP’s LLM01 guidance states that “it is unclear if there are fool-proof methods of prevention,” so audit the blast radius rather than filter claims.
Check: external content is segregated and labelled; extended functions are handled in code rather than by the model; the application holds its own API tokens; privileged operations require human approval.
Evidence: transcripts of direct payloads and of indirect payloads delivered through retrieved documents and tool output.
Fail if: text planted in a retrieved document triggers a tool call the user never requested. Control options are compared in LLM prompt injection defense: what actually works.
3. System prompt hygiene (LLM07)
OWASP’s LLM07 entry states that “the system prompt should not be considered a secret, nor should it be used as a security control.”
Check: search system prompts and templates for API keys, credentials, database names, user roles and permission structures. Confirm that authorization and privilege separation run deterministically outside the model.
Fail if: an extracted system prompt yields a credential or reveals access logic the application depends on.
4. Output handling (LLM05)
Check: model output is treated as untrusted input to every downstream system, per OWASP’s LLM05 guidance: context-aware encoding, parameterized queries for any database operation, no output passed to a shell, exec or eval, and a strict Content Security Policy on chat interfaces.
Evidence: every code path where output reaches a browser, database, file path, template or interpreter.
Fail if: the chat UI renders model-generated Markdown images or links to arbitrary domains, the channel described in how LLM chatbots leak data through their own rendered output.
5. Agent and tool permissions (LLM06)
OWASP’s LLM06 entry traces Excessive Agency to three root causes: excessive functionality, excessive permissions and excessive autonomy.
Check: test each tool against all three. Does the agent need the tool and every function in it? Does the downstream identity hold more than the minimum, such as write access where read-only would do? Do high-impact actions require approval by a human or by the downstream system? Flag open-ended tools such as general shell or URL fetching, and confirm actions run in the user’s context with minimum OAuth scopes.
Fail if: the model, rather than the downstream system, decides whether an action is authorized. Agent-specific failure modes are mapped in AI agent security risks 2026.
6. Sensitive data and retrieval access (LLM02, LLM08)
Check: vector and embedding stores are permission-aware, with datasets partitioned by user class, per OWASP’s LLM08 guidance. Documents are validated before ingestion, using extraction tools that ignore formatting and detect hidden content.
Evidence: immutable retrieval logs, plus a test in which a low-privilege user queries for content indexed from a higher-privilege source.
Fail if: that query returns the restricted content.
7. Model and dataset supply chain (LLM03, LLM04)
Pickle deserialization executes code at load time. Hugging Face’s pickle scanning documentation identifies the GLOBAL, STACK_GLOBAL and REDUCE opcodes as the threat and describes its own Hub import scanner as “not 100% foolproof.”
Check: weights come from trusted publishers with recorded provenance, safetensors is preferred over pickled checkpoints, and training and fine-tuning datasets have documented sources. Teams that train or fine-tune should compare their pipeline against NIST SP 800-218A, the SSDF community profile for generative AI and dual-use foundation models, finalized July 26, 2024.
Fail if: production loads a pickled checkpoint from an unverified repository.
8. Gateway and serving infrastructure
The most severe flaws in an LLM stack are often conventional ones. CVE-2026-42208 is a pre-authentication SQL injection (CWE-89) in the BerriAI LiteLLM proxy, versions 1.81.16 up to but not including 1.83.7, rated CVSS 3.1 9.8 (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H). NVD records that CISA added it to the Known Exploited Vulnerabilities (KEV) catalog on May 8, 2026.
Check: gateway, inference server and vector database versions against NVD and the KEV catalog; where provider credentials and virtual keys are stored; whether admin routes are reachable from the internet.
Fail if: any component runs a KEV-listed version. Exploitation details are in the LiteLLM SQL injection briefing.
9. Consumption limits (LLM10)
Check: the controls in OWASP’s LLM10 guidance: input size limits, per-source rate limiting, timeouts and throttling, caps on queued and total agent actions, and whether the API exposes logit_bias or logprobs.
Fail if: a single key can issue unbounded requests, or an agent loop has no step ceiling.
10. Adversarial testing, monitoring and retest (LLM09 and all classes)
Check: an automated baseline with NVIDIA’s garak, an Apache-2.0 scanner whose probes cover prompt injection, data leakage, jailbreaks, hallucination and misinformation, including a packagehallucination probe relevant to LLM09 Misinformation. Run deeper campaigns with Microsoft’s MIT-licensed PyRIT framework. Confirm prompts, tool calls and retrievals are logged; the NCSC’s fourth area, secure operation and maintenance, covers logging and monitoring, update management and information sharing.
Fail if: no test results exist for the current model version, system prompt and tool set.
Which frameworks should an LLM security audit map to?
Map findings to three references: the OWASP Top 10 for LLM Applications 2025 for defect classes, MITRE ATLAS for adversary techniques, and the NCSC guidelines for lifecycle coverage. Add NIST SP 800-218A when the organization trains or fine-tunes models, since it extends SSDF version 1.1 with AI-specific practices.
The 2025 OWASP list runs LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation and LLM10 Unbounded Consumption. Each maps to a control above.
MITRE ATLAS is a public knowledge base of adversary tactics, techniques and procedures aimed at AI systems, which lets a finding be reported as an attack sequence rather than a single defect. The NCSC guidelines, published November 27, 2023, split the lifecycle into secure design, secure development, secure deployment, and secure operation and maintenance.
For the program-level audit, the OWASP governance checklist spans 13 areas, as summarized by ReversingLabs: adversarial risk, threat modeling, AI asset inventory, security and privacy training, business cases, governance, legal, regulatory, using or implementing LLM solutions, testing, evaluation, verification and validation (TEVV), model and risk cards, RAG optimization, and AI red teaming.
How often should an LLM security audit run?
Run the full checklist before launch and again on every material change, not on a fixed calendar alone. Model upgrades, fine-tuning runs, system prompt edits, new tool registrations and new retrieval sources can each invalidate a prior result, because each changes what the model can see or do.
Item 8 runs on vulnerability-feed cadence instead. KEV timelines are short: for CVE-2026-42208, NVD lists a CISA action due date of May 11, 2026, three days after the catalog addition. Federal civilian agencies are bound by KEV remediation timelines under BOD 26-04; other organizations can use them as a patch-window benchmark.
Recommended cadence for defenders:
- Check AI gateway and inference server versions against the KEV catalog within 24 hours of each catalog update.
- Gate every model or system prompt release on a passing garak baseline in the same release cycle.
- Re-run the item 5 permission review before any new tool or MCP server reaches production.
- Rotate provider credentials held by the gateway within 24 hours of any indicator of gateway compromise.
Runtime enforcement layers complement the audit but do not replace items 5 through 8, which are architecture decisions. Options are compared in the LLM firewall buying guide and the survey of prompt injection detection tools.
FAQ
what is an llm security audit
An LLM security audit is a structured review of an application built on a large language model, covering its prompts, tools, retrieval data, output handling, model artifacts and serving infrastructure. It combines configuration and code review with adversarial testing, and maps each finding to a framework such as the OWASP Top 10 for LLM Applications.
is the owasp llm top 10 a checklist
No, the OWASP Top 10 for LLM Applications is a risk ranking, not an audit procedure. It names ten defect classes and suggests mitigations for each. OWASP publishes a separate LLM Applications Cybersecurity and Governance Checklist, version 1.1, which covers program-level areas such as governance, legal, regulatory and training alongside technical controls.
can an llm security audit be automated
Partly. Scanners such as garak and frameworks such as PyRIT automate attack generation and repeatable probe runs, which suits regression testing after every model or prompt change. Permission design, data classification, retrieval access control and supply-chain provenance still need human review of code, configuration and architecture, because the prompt interface alone does not expose them.
does an llm security audit cover the model provider
Only indirectly. When the model is consumed through a hosted API, the provider’s weights and training pipeline sit outside the audit; the scope is the integration: prompts, tools, retrieval, output handling, gateway and credentials. Record the provider as a third-party component in the inventory and review its published security documentation and contractual terms.
Related across the network
- OWASP LLM Top 10 2026: Mitigations for Every Risk — aisecreviews.com
- AI Security Review: A Checklist for LLM Systems — aisecreviews.com
