Automation bias

Definition

The tendency for people to accept or over-rely on automated recommendations even when there is reason to question them.

Put simply

People can become less critical simply because the system looks authoritative.

What boards and leaders should understand

Putting a person into an AI process doesn’t automatically create effective oversight.

If an AI produces persuasive outputs quickly and frequently, the human reviewer may gradually become a rubber stamp, particularly where they lack the information, expertise or time required to challenge it.

The effectiveness of human oversight therefore depends on the design of the human role, not simply the presence of a person.

Questions to consider

  • Where are we relying on humans to challenge AI recommendations which they might not be equipped or motivated to challenge?