Red teaming
Definition
Deliberately trying to make an AI system fail, behave unsafely or operate outside its intended boundaries.
Put simply
Don't just demonstrate that it works. Actively try to break it.
What boards and leaders should understand
Normal testing tends to show what happens when users behave as expected.
Red teaming asks what happens when people are forceful, manipulative, malicious or simply unusual.
For higher-consequence systems, assurance should include scenarios designed to uncover failures rather than only scenarios designed to demonstrate success.
Questions to consider
- For our higher-consequence AI systems, who has been asked to challenge the system rather than prove that it works?
