AI red teaming: what it is and how it works

The definition

NIST's Generative AI Profile defines AI red-teaming as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled environment and in collaboration with system developers" (NIST AI 600-1, July 2024). The EU AI Act uses the term "adversarial testing" for the same idea when it sets duties for the largest general-purpose models. We cover that in AI red teaming requirements.

LLM red teaming is the same practice applied to systems built on large language models. In 2026 that is most of the market: support chatbots, RAG assistants over internal documents, coding copilots and agents that call APIs.

Two points from NIST shape how buyers should think about it. Red teaming can happen before or after release. And the quality of the output "is related to the background and expertise of the AI red team itself." The team matters more than the tool.

How it differs from pen testing and from evals

Microsoft notes that "red teaming" used to mean adversarial attacks to test security. With LLMs it also covers harms like hate speech and violent or sexual content, because a benign user can trigger them by accident (Microsoft Learn). Its AI Red Team puts the line with evals bluntly: "AI red teaming is not safety benchmarking" (arXiv:2501.07238).

A good AI red team still runs classic appsec checks on the surrounding application: auth, session handling, the APIs your tools call. What it adds is AI-specific method. OWASP lists as a red flag any vendor that offers AI red teaming as an extension of existing appsec testing with no AI-specific approach, or that cannot explain how testing an LLM endpoint differs from testing a normal API (OWASP vendor criteria, Feb 2026). If you are unsure which kind of test you are being sold, read our guide to AI penetration testing.

Model, application and agent testing

What you test changes who you hire. Microsoft separates two layers, and the OWASP agentic work adds a third.

• Model level. The base model with its safety system, usually through an API endpoint. This is where frontier labs spend most of their effort: jailbreaks, dangerous capability uplift, multilingual gaps (Microsoft Learn).
• Application level. The product your users touch, ideally tested through the production UI, before and after mitigations. Microsoft says reports should name the endpoint that was tested. This layer includes system prompts, retrieval, output handling and guardrails.
• Agent level. Systems that plan, keep memory and call tools. The OWASP Top 10 for Agentic Applications 2026 (released 9 December 2025) covers risks such as agent goal hijack, tool misuse, identity and privilege abuse, memory and context poisoning, and cascading failures.

OWASP's GenAI Red Teaming Guide (January 2025) frames the work as four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. Most enterprise buyers need the second and fourth. Few need the first unless they train or fine-tune models.

The attack surface of an LLM app with tools

Every place untrusted text can reach the model is an entry point. Every place model output can act is a blast radius. The diagram shows the four zones a tester works through.

Attack classes a red team should cover

These are the classes we expect in any serious scope, mapped to the OWASP Top 10 for LLM Applications 2025 and to MITRE ATLAS (release v2026.10, 28 October 2026). Ask vendors to map findings to the same IDs so you can compare reports.

Prompt injection is first on the OWASP list for a reason. HackerOne reported in October 2025 that prompt injection reports on its platform rose 540% year over year, and valid AI vulnerability reports rose 210% (HackerOne). Definitions for every term here are in our AI red teaming glossary.

Human-led vs automated red teaming

Automated red teaming uses one model to generate attacks against another, an approach set out in Perez et al., 2022. Open-source tools such as garak, PyRIT and Promptfoo package it. They are good at breadth: thousands of known attack patterns, every release, in CI. We compare them in open-source AI red teaming tools.

Humans are good at depth. They read your product, guess what an attacker would want and chain small weaknesses into a real impact. Microsoft's AI Red Team says automation extends coverage but humans remain essential, and that attacking an AI system does not require gradient access (arXiv:2501.07238). Microsoft's planning guide recommends a first round of manual testing before systematic measurement (Microsoft Learn).

NIST describes four team types: general public, expert, combination, and human/AI. It also notes that AI-led red teaming "can be more cost effective than human red-teamers alone" (NIST AI 600-1). Anthropic lists domain-expert, automated, multimodal, crowdsourced and community methods, and says the lack of standard practice makes it hard to compare systems (Anthropic, June 2024). In practice, buy a mix. OWASP lists automation-only testing with no manual adversarial work as a red flag.

How an engagement runs

Most engagements follow the same five steps. The structure below follows Microsoft's planning guide, with OWASP for threat modeling.

1. Scoping. Agree what is in scope: base model via API, the app via its UI, the retrieval pipeline, each tool and its permissions. Decide which harms matter, who tests (domain experts plus people with adversarial and benign mindsets), the rules of engagement and how results are recorded. Microsoft suggests logging a date, a unique ID, the prompt and the output for every finding.
2. Threat model. List who would attack the system, what they want and which paths reach it. Map those paths to the OWASP LLM and Agentic Top 10 and to MITRE ATLAS tactics. OWASP's vendor criteria stress realistic threat models and matching depth to complexity: a chatbot is not a RAG system, and neither is a multi-agent workflow over MCP.
3. Testing. Start with open-ended manual rounds, build a list of harms and weaknesses, then run guided rounds and automated attacks at scale. Test with and without mitigations so you can see what the guardrails actually stop.
4. Reporting. Microsoft's recommended report lists the top issues, links to raw data, previews the next round and states that examples are not prevalence metrics. Each finding should be reproducible, mapped to a framework ID, rated for severity and paired with a fix.
5. Retest. After fixes, rerun the findings on the production endpoint. A finding is not closed until it fails on the system users actually reach.

On timing, one vendor publishes estimates of about 5 days for a simple chatbot, 10 to 15 days for RAG or a small agent, 15 to 25 days for complex agentic systems and 20 to 30 days when compliance documentation is included (SecurityWall, vendor blog, unaudited). Budget figures are in our AI red teaming cost guide.

The evidence that agents break easily

If you are deploying agents, assume they can be hijacked until a test shows otherwise. The public data is consistent.

• A public competition co-run by the UK AI Security Institute and Gray Swan between March and April 2025 (Gray Swan) collected 1.8 million prompt injection attacks against 22 frontier AI agents in 44 scenarios. It produced more than 60,000 policy violations. The authors write: "Nearly all agents exhibit policy violations for most behaviors within 10-100 queries" (arXiv:2507.20526).
• NIST's Center for AI Standards and Innovation (CAISI) reported in March 2026 on a later competition: about 250,000 attacks from more than 400 participants against 13 frontier models. "At least one successful attack was found against all of the target frontier models." Attacks that worked on the most resistant models tended to transfer to weaker ones, rarely the reverse (CAISI).
• OpenAI's GPT-5 system card reports more than 5,000 hours of external red teaming from over 400 testers. Bio red teamers filed 46 potential jailbreaks in about 380 hours, and two external groups ran a two-week prompt injection assessment of connectors (GPT-5 system card, August 2025).

The takeaway for buyers: the best-funded labs in the world still find breaks with sustained effort. Your vendor should not promise you zero findings.

What red teaming cannot do

A red team report is a sample of failures, not a safety certificate. Microsoft calls the work never finished. An academic review found red-teaming practice diverges on purpose, artifact and setting, and warned that without structure it "could verge on security theater" (Feffer et al., 2024). Treat any vendor claim of "100% coverage" or "certified safe" as a reason to keep looking. Our vendor selection checklist lists the other red flags.

Where to go next

• How to choose an AI red teaming vendor: the checklist and the questions to ask.
• AI red teaming cost: published price ranges and a worked budget.
• AI red teaming requirements: EU AI Act, NIST and US policy.
• Open-source AI red teaming tools: what you can run yourself.
• Compare AI red teaming companies.

This guide is general information written by our editors. It is not security advice for your specific system.