Research5h ago
LLM Benchmarks Fail to Keep Pace
Canonical benchmarks like MMLU and HumanEval no longer reliably measure LLM capabilities. Data contamination and outpaced performance force researchers to rely on subjective "vibe checks" for assessment. This erosion of objectivity leaves practitioners without a concrete standard for model comparison. The field now lacks a dependable telescope to gauge actual intelligence.