An AI red team methodology guide has to answer a narrower question than “how to jailbreak a chatbot.” It has to say what gets scoped, in what order, against which threat model, and what evidence closes a finding. Four public documents carry most of that weight: the OWASP GenAI Red Teaming Guide for scope decomposition, MITRE ATLAS for adversary tactics, the OWASP Top 10 for LLM Applications 2025 for application-layer risk classes, and NIST AI 600-1 for team composition and the governance hooks findings have to land in.
Scope four layers, not one model
The OWASP GenAI Red Teaming Guide, published 22 January 2025, splits the work into four areas: model evaluation, implementation testing, infrastructure assessment, and runtime behavior analysis. That split is the most useful structural decision in the methodology, because the three non-model layers are where most exploitable defects live.
Model evaluation covers the weights and their alignment behavior. Implementation testing covers the system prompt, tool and function definitions, the retrieval pipeline, and output handling. Infrastructure assessment covers the serving stack, vector store, model registry, and artifacts loaded at startup. Runtime behavior analysis covers what the system does across a live session with memory, state, and multi-step autonomy.
Microsoft’s red team reached the same conclusion operationally in Lessons From Red Teaming 100 Generative AI Products (arXiv:2501.07238, 13 January 2025). Two of its eight lessons are directly methodological: comprehend the system’s capabilities and deployment context before attacking it, and recognize that breaking AI systems does not require gradient computation. A third holds that LLMs both amplify existing vulnerabilities and introduce novel ones, which is the argument for keeping conventional application testing inside the engagement.
Build the threat model from ATLAS and the LLM Top 10
Threat modeling is the second phase, and it should produce a target list. ATLAS is the reference for adversary behavior: release 2026.08 catalogs 16 tactics, 197 techniques and sub-techniques, 72 real-world case studies, and 39 mitigations, per MITRE’s dataset.
The 16 tactics run AI Model Access, AI Attack Adaptation, Reconnaissance, Resource Development, Initial Access, Execution, Persistence, Defense Evasion, Discovery, Collection, Exfiltration, Impact, Privilege Escalation, Credential Access, Command and Control, and Lateral Movement. ATLAS is modeled on ATT&CK and its tactics are complementary to it, so an engagement can express an AI-specific kill chain in notation the blue team already uses.
The OWASP LLM Top 10 supplies the risk classes to test against at the application layer. The 2025 list is LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, and LLM10 Unbounded Consumption.
Use both, not one. ATLAS gives the sequence an adversary follows; the LLM Top 10 gives the defect classes to probe at each step. For the mechanics of individual techniques at the LLM01 and LLM06 end of that list, the writeups at AI Sec go deeper on prompt injection, jailbreak chains, and agent exploitation than a methodology document should.
Execute in the order that preserves signal
Sequence matters: early phases change the attack surface you can see. A workable order:
- Capability and boundary inventory. Enumerate tools, function schemas, retrieval sources, memory scope, authentication context, and what the model may do without a human in the loop. Excessive Agency (LLM06) is a design finding, usually visible here before any payload is sent.
- Single-turn probing. Direct injection, system prompt extraction, and output-handling tests at each entry point. Cheap, parallelizable, and it sets the baseline refusal behavior later phases have to beat.
- Indirect and multi-turn. Payloads delivered through retrieved documents, tool responses, or file uploads rather than the user turn, plus conversational escalation. PyRIT, Microsoft’s MIT-licensed framework, ships multi-turn strategies including Crescendo, TAP, and Skeleton Key, plus converters, scorers, and a memory store for replay.
- Chained and infrastructure. Combine an injection primitive with a real capability: exfiltration via a tool call, retrieval-corpus poisoning, deserialization of an untrusted model artifact, or lateral movement into the serving environment.
- Runtime and stateful. Re-run the successful chains against a live session with accumulated memory. Behavior that fails in a clean session frequently succeeds in a dirty one.
NIST AI 600-1 names the capabilities to assess. Action MS-2.7-007 calls for red-teaming resilience against “Abuse to facilitate attacks on other systems (e.g., malicious code generation, enhanced phishing content),” GAI attacks such as prompt injection, and ML attacks such as adversarial examples. Action MS-2.10-001 covers training-data extraction, model extraction, membership inference, and disclosure of confidential, copyrighted, personal, or proprietary information.
Staff the team deliberately
NIST AI 600-1, published July 2024, defines AI red-teaming as “a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled environment and in collaboration with system developers.” It sets out four team types, and the choice is a methodology decision, not an HR one:
- General Public: non-expert users bringing their own lived experience. More effective with large groups.
- Expert: specialists in the domain of use, such as medicine, biotech, or cybersecurity.
- Combination: experts modify or verify prompts written by general-public participants, which the profile notes “may also expand coverage of the AI risk attack surface.”
- Human / AI: generative AI paired with human teams. The profile states GAI-led red-teaming “can be more cost effective than human red-teamers alone,” and that the two may suit different harm types.
The profile is explicit that results “should be given additional analysis before they are incorporated into organizational governance and decision making,” and that red-teaming can run before or after release. Microsoft’s fourth and fifth lessons agree: automation extends coverage, and human expertise remains indispensable.
Report findings that survive triage
An AI red team finding needs the four things any pentest finding needs, plus one it does not: a reproducible trigger, the ATLAS technique and LLM Top 10 class it maps to, the concrete data or capability reached, a remediation owner, and a success rate. The last matters because model behavior is stochastic. A payload that fires once in forty attempts is a different risk than one that fires consistently, and a single screenshot cannot tell them apart.
Document the prompts and the instructions given to testers. NIST action MS-2.8-002 asks organizations to “document the instructions given to data annotators or AI red-teamers,” which is what makes a retest comparable to the original run.
Retest on change, not on calendar. Model upgrades, fine-tuning runs, system prompt edits, new tool registrations, and changes to a connected data source can each invalidate a prior result, the practical reading of Microsoft’s eighth lesson that securing AI systems is never-ending work. For how these expectations are hardening into regulatory obligation, NeuralWatch tracks the NIST AI RMF and EU AI Act side.
One caution: Microsoft’s third lesson is that red teaming is not safety benchmarking. A benchmark measures known failure modes against a fixed dataset; red teaming searches for unknown ones in a specific deployment. A methodology reporting only benchmark scores has not done the second thing.
Related across the network
- Adversarial ML Testing Tools: ART, ModelScan, Giskard — adversarialml.dev
- AI Risks: A Practitioner’s Taxonomy for Security Teams — ai-alert.org
- LLM Security Risks: A Practitioner’s Field Guide — ai-alert.org
