Research1d ago
Data Pruning Boosts LLM Fact Memorization
Information-theoretic analysis reveals that LLMs fail to memorize facts when training data exceeds model capacity. Researchers at Apple found that pruning training data actually improves factual accuracy. This suggests that less, higher-quality data prevents the capacity bottlenecks that trigger hallucinations. Practitioners should prioritize data curation over raw volume to maximize knowledge retention.