AI red teaming is a structured effort to make an AI system fail before a real attacker, user, or unexpected situation does it for you.
NIST defines it as structured testing to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers. In plain English: give capable testers permission to challenge the system, observe how it breaks, and turn those findings into fixes and better tests.
It is more than trying a clever prompt
For a basic chatbot, a red team may test whether carefully designed prompts can bypass safeguards, expose sensitive information, generate prohibited content, or reveal hidden instructions.
For an AI agent, the target is larger. Testers may examine:
- the model’s responses;
- tools and connectors;
- identity and permissions;
- data retrieval and memory;
- approval gates;
- logging and alerting;
- actions taken in external systems; and
- whether the agent can be stopped safely.
That is why agent red teaming can look more like a security assessment of an entire workflow than a prompt contest.
Red teaming, evaluation, and penetration testing
These practices overlap, but they answer different questions.
Evaluation measures performance against defined tests: accuracy, refusal behavior, robustness, or task completion.
Penetration testing looks for exploitable weaknesses in applications, infrastructure, and security controls.
AI red teaming uses adversarial thinking to discover harmful or unexpected behavior across the model and the surrounding system. It may include both evaluation and traditional security testing, but it also explores failures that were not fully specified in advance.
The label matters less than the scope. A vendor saying “we red team our model” does not tell you whether your prompt, data, connectors, permissions, and deployment configuration were tested.
What testers actually do
- Map the system. Identify the model, interfaces, data sources, tools, permissions, users, and consequential actions.
- Define failure modes. Decide what counts as harm: data exposure, policy bypass, prompt injection, unsafe advice, unauthorized action, biased outcomes, or loss of availability.
- Create adversarial scenarios. Test realistic misuse, malicious inputs, indirect prompt injection, tool abuse, permission escalation, and unusual multi-step interactions.
- Record reproducible evidence. Preserve the input, configuration, model/version, action trace, result, and severity rationale.
- Remediate and retest. Fix the weakness, rerun the attack, and add a regression test so the same failure is checked after future changes.
A one-time exercise can find useful problems. It cannot prove that a changing AI system will remain safe after model, prompt, connector, permission, or data changes.
What a red-team report should tell you
- What exact system and version were tested?
- Which data, tools, and permissions were in scope?
- Which threat scenarios were excluded?
- Who performed the testing, and how independent were they?
- Which attacks succeeded?
- How was severity determined?
- What was fixed, accepted, or deferred?
- Was remediation retested?
- Which tests will run again after changes?
NIST’s 2026 analysis of a large public agent-security competition illustrates why broad testing matters. More than 400 participants made over 250,000 attack attempts against 13 frontier models, and at least one successful attack was found against every target model. That does not mean every deployment is equally vulnerable; it shows that model capability alone is not a reliable substitute for adversarial testing.
What AI red teaming cannot prove
A passed red-team exercise is not a safety certificate. Testers can only explore the scenarios, access, time, and system version in scope. Unknown attacks, configuration changes, new tools, and real-world misuse can still produce failures.
The useful outcome is not “nothing bad can happen.” It is a clearer map of failure modes, tested mitigations, and evidence for what should be monitored next.
From the field
I start with what I want the AI to accomplish, then ask the vendor to show which guardrails were tested and how thoroughly. If the evidence exposes gaps, the next step is not to treat the report as the finish line. Add the missing guardrails, improve or retrain the system where appropriate, and test it again. Red teaming should be a cycle of testing, remediation, and retesting—not a one-time badge.
The practical takeaway
Treat “red-teamed” as the beginning of a procurement or governance question, not the answer. Ask what was tested, against which threats, with what access, what failed, what changed, and whether the fixes were retested.