Red-Teaming

Red-teaming (from military/security practice) is the process of deliberately attacking or stress-testing an AI system to surface failures — before real users or adversaries do.

What red teams look for in AI:

• Jailbreaks — prompts that bypass safety guidelines
• Harmful content — can the model be induced to produce misinformation, malware, dangerous instructions?
• Bias and fairness failures — differential treatment of demographic groups
• Prompt injection — can adversarial content in external data control the agent?
• Capability elicitation — can the model do things it shouldn't, even if not normally willing?
• Specification gaming — does the model find unexpected ways to satisfy the letter but not spirit of instructions?

Methods:

• Manual red-teaming — human experts crafting adversarial prompts (most creative, most expensive)
• Automated red-teaming — use a separate LLM to generate adversarial prompts at scale
• Structured evaluation sets — standardised benchmarks like HarmBench, StrongREJECT

Why it matters: All major AI labs run extensive red-teaming before releases. The US AI Safety Institute (AISI) and UK AISI conduct independent red-teaming of frontier models under safety commitments.