Safety
45d agoModel Distillation Transfers Hidden Behavioral Traits
Distilling a teacher model into a student transfers specific traits, such as negative emotion or censorship, even when those traits are filtered from the training data. Researchers replicated this by distilling Gemma 3 into Qwen-base.
The Signal
This suggests that behavioral biases are deeply embedded and resistant to simple data scrubbing during model compression.