dailyai.report

Researchers hand-coded weights for one-layer MLPs to memorize labels for two-token sequences. These models achieve 90% accuracy, scaling linearly with parameter count similar to trained models. However, the hand-coded prefactor remains significantly lower than that of trained counterparts. This provides a baseline for understanding how mechanistic interpretability maps to memory efficiency.