Safety20h ago
Decoding Why AI Annotators Disagree
A new arXiv paper analyzes whether annotator disagreement stems from operational failure, policy ambiguity, or value pluralism. The researchers use interpretability methods to distinguish these sources of conflict. This allows developers to target specific fixes, such as clarifying policy wording versus improving quality control, rather than treating all errors as noise.