LLM-as-a-judge
Definition
Using one large language model to assess or score the output or behaviour of another AI system.
Put simply
AI helps test AI.
What boards and leaders should understand
This can make large-scale evaluation faster and cheaper than relying entirely on human reviewers.
But the judge is itself an AI model and may introduce its own biases, blind spots or inconsistent judgements.
Automated evaluation can therefore provide useful evidence without necessarily providing independent assurance.
Questions to consider
- Where AI is evaluating AI, how are we checking the quality and independence of the evaluator?
