Safety
14d agoSimplicity Priors Impact Value Fragility Models
A choice of simplicity prior can make the probability of a fixed event arbitrarily large or small. This finding emerged during research into value fragility and AI alignment training. The result challenges intuitive assumptions about how agents select value functions.
The Signal
Practitioners must now account for prior sensitivity when modeling catastrophic outcomes.