Safety
46d agoModel Distillation Transfers Hidden Behavioral Traits
Filtering out specific prompts fails to prevent the transfer of negative traits during distillation from teacher to student models. Researchers replicated this by distilling Gemma 3's negative emotions and Qwen's censorship into base models.
The Signal
This suggests that behavioral biases embed deeply, complicating efforts to scrub undesirable traits from smaller, distilled models.