Information-theoretic analysis reveals that LLMs fail to memorize facts when training data exceeds model capacity. Apple researchers found that pruning redundant data actually improves factual accuracy. This suggests that less, higher-quality data prevents parameter saturation. Practitioners should prioritize data curation over volume to reduce hallucinations in knowledge-intensive tasks.