Home / AI red teaming glossary
AI red teaming glossary
63 terms you meet when buying or running AI red teaming, from adversarial examples to vector store weaknesses. Each has a short definition, framework IDs where they exist and a source where a claim needs one. Every term has its own anchor, so you can link to a single definition.
New to the topic? Start with what AI red teaming is. For buying, see the vendor checklist and pricing. For rules, see AI red teaming requirements. For free tools, see open-source AI red teaming tools. If you meant using AI to pen test, read AI penetration testing.
Terms A to Z
A
- Adversarial example (evasion)
- An input crafted to make a model err, such as an image with imperceptible noise that flips a classifier. The EU AI Act names adversarial examples and model evasion in Article 15(5); ATLAS lists evasion as AML.T0043. Source: EU AI Act Art. 15 · MITRE ATLAS data
- Adversarial machine learning (AML)
- The study of attacks on machine learning systems and defenses against them. NIST's taxonomy, AI 100-2 E2025, groups attacks into evasion, poisoning, privacy and generative AI misuse. Source: NIST AI 100-2 E2025
- Adversarial suffix (GCG)
- An optimized string of tokens appended to a prompt that makes aligned models comply with harmful requests. The Greedy Coordinate Gradient method from Zou et al. (2023) found suffixes that transfer across models. Source: Zou et al., arXiv:2307.15043
- Adversarial testing
- The EU AI Act's term for red teaming. Article 55(1)(a) requires providers of GPAI models with systemic risk to conduct and document adversarial testing to identify and mitigate systemic risks. See requirements. Source: EU AI Act Art. 55
- Agent goal hijack
- Redirecting an agent away from its user's objective, usually through injected instructions. ASI01 in the OWASP Top 10 for Agentic Applications 2026. Source: OWASP Agentic Top 10 2026 · Giskard summary of ASI01 to ASI10
- AI penetration testing
- An ambiguous term. It means either penetration testing of AI systems or penetration testing performed by AI agents against ordinary software. Check which one a vendor sells. See both meanings explained.
- AI red teaming
- NIST defines it as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled environment and in collaboration with system developers." See full explainer. Source: NIST AI 600-1
- AI RMF and GenAI Profile
- NIST's voluntary AI Risk Management Framework 1.0 (January 2023) and its Generative AI Profile, AI 600-1 (July 2024). NIST says the RMF is being revised under the White House AI Action Plan. Source: NIST AI RMF · NIST AI 600-1
- Attack success rate (ASR)
- The share of attack attempts that produce a policy violation. Useful only with the number of attempts stated. Microsoft warns that red team examples are not prevalence metrics. Source: Microsoft Learn · arXiv:2507.20526
- Automated red teaming
- Using models or tools to generate and run attacks at scale. The approach was set out by Perez et al. (2022) and is packaged in tools such as garak, PyRIT and Promptfoo. See open-source tools. Source: Perez et al., arXiv:2202.03286
B
- Backdoor
- A hidden trigger planted through poisoning that makes a model behave in a targeted way when the trigger appears. Research from 2025 found such poisoning needs a near-constant number of samples regardless of model size. Source: arXiv:2510.07192
- Best-of-N jailbreaking
- Repeatedly sampling small random variations of a prompt until one gets past the model's safeguards. It needs only black-box access to the model. Source: arXiv:2412.03556
C
- CAISI
- NIST's Center for AI Standards and Innovation, renamed from the US AI Safety Institute in June 2025. It evaluates frontier models and ran a large agent red teaming competition with the UK AI Security Institute. Source: Commerce, Jun 2025 · CAISI, Mar 2026
- Cascading failures
- An error or compromise in one agent spreading through chained agents and tools. ASI08 in the OWASP Agentic Top 10. Source: OWASP Agentic Top 10 2026 · Giskard summary of ASI01 to ASI10
- CBRN uplift
- A meaningful increase in someone's ability to cause chemical, biological, radiological or nuclear harm because of model access. One of NIST AI 600-1's 12 generative AI risks and a core frontier lab test area. Source: NIST AI 600-1 · GPT-5 system card
- Confabulation
- NIST's term for confidently stated false or fabricated content, often called hallucination. One of the 12 risks in AI 600-1. Source: NIST AI 600-1
- Constitutional classifiers
- Anthropic's classifier-based safeguards designed to block universal jailbreaks, which it tested through a bug bounty program in 2025. Source: arXiv:2501.18837 · Anthropic, 2025
- Continuous red teaming
- Ongoing automated testing, usually on a platform subscription or in CI, so every model, prompt or tool change gets retested. Complements periodic expert engagements. Source: Repello (vendor blog)
- Crowdsourced red teaming
- Testing by many outside participants, through competitions, bug bounties or community events. NIST calls this the "General Public" team type. Source: NIST AI 600-1 · Anthropic, Jun 2024
D
- Data exfiltration (AI)
- Getting data out through a model: via its inference API, or by making it render a link or image that carries data to an attacker. ATLAS v2026.10 added AML.T0136, Exfiltration via AI User Interface. Source: MITRE ATLAS CHANGELOG
- Data poisoning
- Manipulating training, fine-tuning or retrieval data to change model behavior. Named in EU AI Act Article 15(5), OWASP LLM04:2025 and ATLAS AML.T0020. Source: EU AI Act Art. 15 · OWASP LLM Top 10 2025
- Digital Omnibus on AI
- The 2026 EU law that amended AI Act dates. In force since 27 July 2026, it moved Annex III high-risk duties to 2 December 2027 and Annex I duties to 2 August 2028. See what changed. Source: European Commission
- Direct prompt injection
- A user typing instructions that override the system prompt or developer intent. ATLAS AML.T0051.000; part of OWASP LLM01. Source: OWASP LLM Top 10 2025 · MITRE ATLAS data
E
- Eval (evaluation)
- A repeatable test set that measures how often a model shows a known behavior. Microsoft's AI Red Team states that "AI red teaming is not safety benchmarking": evals measure, red teams search. Source: Microsoft AI Red Team, arXiv:2501.07238
- Excessive agency
- An LLM system given more functions, permissions or autonomy than its task needs, so a manipulated model can do real damage. OWASP LLM06:2025. Source: OWASP LLM Top 10 2025
- Expert red teaming
- Testing by specialists in a domain such as biosecurity, cyber or medicine. NIST names "Expert" as its own team type; OpenAI used over 400 external testers and experts for GPT-5. Source: NIST AI 600-1 · GPT-5 system card
F
- Frontier model
- Informally, the most capable general models. California SB 53 defines frontier models as those trained with more than 10^26 operations. Source: The Register, Sep 2025
G
- GPAI Code of Practice
- A voluntary code, published by the European Commission on 10 July 2025, that GPAI providers can use to show AI Act compliance. Its Safety and Security chapter covers systemic-risk models. Source: GPAI Code of Practice
- GPAI model with systemic risk
- An EU AI Act category for the most capable general-purpose models, presumed when training compute exceeds 10^25 FLOPs. These providers carry the Article 55 adversarial testing duty. Source: EU AI Act Art. 51 · EU AI Act Art. 55
- Guardrails (AI firewall)
- Runtime filters that inspect model inputs and outputs and block attacks or policy violations. Several red teaming vendors also sell them, which matters for neutrality. See who owns whom. Source: SecurityWeek on Lakera
H
- High-risk AI system
- An EU AI Act category covering systems in Annex III areas or embedded in Annex I regulated products. Article 15 requires measures against poisoning and adversarial attacks for them. Source: European Commission · EU AI Act Art. 15
- Human-led red teaming
- Testing driven by people who explore a system, form hypotheses and chain weaknesses. Microsoft recommends manual rounds before automated measurement and says humans remain essential. Source: Microsoft Learn · Microsoft AI Red Team, arXiv:2501.07238
I
- Improper output handling
- Passing model output to browsers, shells, databases or other systems without validation, which turns prompt injection into XSS, SQL injection or code execution. OWASP LLM05:2025. Source: OWASP LLM Top 10 2025
- Indirect prompt injection
- Instructions hidden in content the model retrieves, such as a web page, email, document or tool result, rather than typed by the user. Described by Greshake et al. (2023); ATLAS AML.T0051.001. Source: Greshake et al., arXiv:2302.12173 · MITRE ATLAS data
- ISO/IEC 42001
- The certifiable international standard for AI management systems, published December 2023. It covers governance and risk controls; red team reports can serve as testing evidence. Source: ISO
J
- Jailbreak
- A prompt or technique that gets a model past its safety training or filters. ATLAS AML.T0054, LLM Jailbreak. Source: MITRE ATLAS data
M
- MCP (Model Context Protocol)
- An open protocol, introduced by Anthropic in November 2024, for connecting models to tools and data. OWASP's vendor criteria list MCP architectures among the advanced systems that need broader testing. Source: Anthropic, Nov 2024 · OWASP vendor criteria, Feb 2026
- Membership inference
- Working out whether a specific record was in a model's training data. A privacy attack listed under ATLAS AML.T0137 in v2026.10. Source: MITRE ATLAS CHANGELOG
- Memory and context poisoning
- Corrupting an agent's persistent memory or working context so that later sessions act on false or malicious information. ASI06 in the OWASP Agentic Top 10. Source: OWASP Agentic Top 10 2026 · Giskard summary of ASI01 to ASI10
- MITRE ATLAS
- MITRE's knowledge base of adversary tactics, techniques and case studies against AI systems. Release v2026.10 (28 October 2026) has 16 tactics, 129 techniques, 92 sub-techniques, 44 mitigations and 77 case studies. Source: MITRE ATLAS CHANGELOG
- Model extraction (model stealing)
- Recovering a model's parameters or replicating its behavior through queries. Carlini et al. (2024) recovered part of a production model. ATLAS AML.T0005, Replicate AI Model Behavior. Source: Carlini et al., arXiv:2403.06634 · MITRE ATLAS CHANGELOG
- Model inversion
- Reconstructing training data, or features of it, from a model's outputs. Listed under ATLAS AML.T0137 in v2026.10. Source: MITRE ATLAS CHANGELOG
- Model poisoning
- Tampering with pre-trained models or components before they reach you. Named alongside data poisoning in EU AI Act Article 15(5). Source: EU AI Act Art. 15
O
- OWASP Top 10 for Agentic Applications
- OWASP's list of the top risks for agentic AI, released 9 December 2025, from ASI01 Agent Goal Hijack to ASI10 Rogue Agents. Source: OWASP Agentic Top 10 2026 · Giskard summary of ASI01 to ASI10
- OWASP Top 10 for LLM Applications
- OWASP's list of the top risks for LLM applications. The 2025 edition runs from LLM01 Prompt Injection to LLM10 Unbounded Consumption and is the default mapping for findings. Source: OWASP LLM Top 10 2025
P
- Penetration testing
- Time-boxed security testing of software, networks and APIs by simulated attack. OWASP warns against vendors that relabel it as AI red teaming without AI-specific method. Source: OWASP vendor criteria, Feb 2026 · Microsoft Learn
- Prompt injection
- Input that changes an LLM's behavior against its developer's intent, either typed directly or hidden in retrieved content. OWASP LLM01:2025 and ATLAS AML.T0051. Source: OWASP LLM Top 10 2025 · MITRE ATLAS data
R
- RAG (retrieval-augmented generation)
- A design where the application retrieves documents and adds them to the model's context. Every retrieval source is a channel for indirect prompt injection and for OWASP LLM08 vector and embedding weaknesses. Source: OWASP LLM Top 10 2025 · Greshake et al., arXiv:2302.12173
- Red team as a service (RTaaS)
- Per-engagement external red teaming sold at a fixed scope and fee, one of five pricing models one vendor describes. See pricing. Source: Repello (vendor blog)
- Red team, blue team, purple team
- Attackers, defenders, and a joint exercise where both work together. In AI, Microsoft notes, red teaming extends beyond security bugs to harms such as hateful or violent content. Source: Microsoft Learn
- Retest
- Rerunning findings after mitigations to confirm they are fixed. Microsoft recommends testing with and without mitigations and on the production endpoint. Source: Microsoft Learn
- Rogue agent
- An agent acting outside its intended scope, whether through compromise or misalignment. ASI10 in the OWASP Agentic Top 10. Source: OWASP Agentic Top 10 2026 · Giskard summary of ASI01 to ASI10
- Rules of engagement
- The written scope, permissions, limits and contacts for a test, agreed before testing starts. They protect both the buyer and the tester. See how an engagement runs.
S
- Sensitive information disclosure
- A model revealing personal data, secrets, proprietary data or other users' information. OWASP LLM02:2025. Source: OWASP LLM Top 10 2025
- Supply chain compromise (AI)
- Attacks through third-party models, datasets, plug-ins or MCP tools. OWASP LLM03:2025 and ATLAS AML.T0010, renamed Supply Chain Compromise in v2026.10. Source: OWASP LLM Top 10 2025 · MITRE ATLAS CHANGELOG
- System prompt leakage
- Extraction of the hidden instructions that configure an LLM app. OWASP LLM07:2025. In ATLAS v2026.10 the technique moved from AML.T0056 to AML.T0085.003. Source: OWASP LLM Top 10 2025 · MITRE ATLAS CHANGELOG
T
- Threat model
- A structured description of who might attack a system, what they want and which paths reach it. It sets the scope and depth of a red team engagement. Source: OWASP vendor criteria, Feb 2026
- Tool misuse
- An agent using its legitimate tools in harmful or unintended ways, for example after injected instructions. ASI02 in the OWASP Agentic Top 10; ATLAS AML.T0053, AI Agent Tool Invocation. Source: OWASP Agentic Top 10 2026 · MITRE ATLAS data
U
- UK AI Security Institute (AISI)
- The UK government body for frontier AI evaluation, renamed from the AI Safety Institute on 14 February 2025. It focuses on chemical and biological misuse, cyber attacks and crime. Source: UK government, Feb 2025
- Unbounded consumption
- Prompts or usage patterns that run up compute cost or deny service. OWASP LLM10:2025. Source: OWASP LLM Top 10 2025
- Universal jailbreak
- A single prompting strategy that defeats a model's safeguards across many questions, not just one. The main target of Anthropic and OpenAI safety bounties. Source: Anthropic, Aug 2024 · OpenAI bio bug bounty
- Uplift study
- A controlled experiment comparing how well people complete a harmful task with and without model access. Frontier labs report them in system cards for CBRN risk. Source: Anthropic system card
V
- Vector and embedding weaknesses
- Flaws in how a RAG system stores and retrieves embeddings, such as poisoned entries or retrieval across access boundaries. OWASP LLM08:2025. Source: OWASP LLM Top 10 2025
Common questions
What is the difference between direct and indirect prompt injection?
Direct prompt injection is typed by the user. Indirect prompt injection is hidden in content the model reads, such as a web page, email, document or tool result. Indirect injection is the bigger risk for agents because the attacker never needs to talk to your system.
What is the difference between a jailbreak and prompt injection?
A jailbreak targets the model's safety training to get content it should refuse. Prompt injection targets the application's instructions to change what the system does. The techniques overlap, and OWASP groups jailbreaking under LLM01.
What is excessive agency?
OWASP's term, LLM06:2025, for an LLM system that has more functions, permissions or autonomy than its task needs. When such a system is manipulated, the damage is real actions, not just bad text.
What is MITRE ATLAS?
MITRE's knowledge base of adversary tactics and techniques against AI systems. The current release, v2026.10 from 28 October 2026, lists 16 tactics and 129 techniques.
What does adversarial testing mean in the EU AI Act?
It is the Act's term for red teaming. Article 55 requires providers of GPAI models with systemic risk to conduct and document adversarial testing to identify and mitigate systemic risks.
What is a universal jailbreak?
A single strategy that bypasses a model's safeguards across many questions. Anthropic and OpenAI have paid bounties of $15,000 to $25,000 for them.
Can we link to a single definition?
Yes. Every term has an anchor. Click the term name to get a link such as /glossary/#prompt-injection.
Sources
- NIST AI 600-1: Generative AI Profile (26 Jul 2024)
- NIST AI 100-2 E2025: Adversarial machine learning taxonomy
- NIST AI Risk Management Framework
- Microsoft Learn: Planning red teaming for LLMs and their applications
- Microsoft AI Red Team: Lessons from red teaming 100 generative AI products, arXiv:2501.07238
- OWASP Top 10 for LLM Applications 2025
- OWASP Top 10 for Agentic Applications 2026 (9 Dec 2025)
- Giskard: summary of the OWASP Top 10 for Agentic Applications 2026
- OWASP Vendor Evaluation Criteria for AI Red Teaming v1.0 (4 Feb 2026)
- MITRE ATLAS data and CHANGELOG (v2026.10)
- EU AI Act Articles 15, 51 and 55 (via artificialintelligenceact.eu)
- European Commission: AI Act regulatory framework
- European Commission: GPAI Code of Practice
- Greshake et al.: Indirect prompt injection, arXiv:2302.12173
- Perez et al.: Red teaming language models with language models, arXiv:2202.03286
- Zou et al.: GCG adversarial suffixes, arXiv:2307.15043
- Carlini et al.: Stealing part of a production language model, arXiv:2403.06634
- Anthropic: Model Context Protocol (Nov 2024)
- OpenAI: GPT-5 system card (13 Aug 2025)