What it is
AI red teaming deliberately attacks an AI agent or model with adversarial inputs—prompt injection, jailbreaks, data exfiltration attempts—to find weaknesses before deployment. It borrows from traditional cybersecurity red teaming but targets AI-specific risks like hallucination triggers and guardrail bypasses. Results feed back into guardrails, instructions, and access controls.
Why it matters
As agents gain access to real business data and actions, an unpatched vulnerability can lead to data leaks or unauthorized transactions. Red teaming turns theoretical risk into a fixable, documented list before go-live.
Key components
- Adversarial prompt testing
- Jailbreak simulation
- Guardrail validation
- Vulnerability reporting
How it connects
Agentforce deployments benefit from red teaming against topics, actions, and guardrails before granting agents production data access.
Good to know
Run red team exercises every major release, not just once at launch—new topics and actions introduce new attack surfaces.
Related terms
Guardrails
The boundaries you set on AI agents to control what they can discuss, what actions they can take, and when they must escalate to a human.
Prompt Injection
An attack where malicious text hidden in data an AI reads hijacks its instructions, making it ignore its real task and do the attacker's bidding.
STRIDE
A six-category threat-modeling framework — Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege — adapted from classic software security to AI agents.
Threat Modeling (AI Agents)
A structured way to identify what could go wrong with an AI agent before it does — mapping attackers, attack surfaces, and impacts so you can build defenses in.
