dailyai.report

Researchers hand-coded weights for one-layer MLPs to memorize labels for two-token sequences. These models achieve 90% accuracy with a memory capacity that scales linearly with parameter count. While they mimic trained model scaling, the prefactor remains significantly lower. This experiment provides a baseline for understanding mechanistic interpretability and weight initialization.