Research
51d agoOpenAI Challenges SWE-Bench Pro Reliability
Analysis from OpenAI reveals critical flaws in SWE-Bench Pro, a benchmark used to measure AI coding capabilities. The study identifies noise and inaccuracies that skew performance results. These findings suggest current coding evaluations lack the precision needed for rigorous model comparison.
The Signal
Practitioners should treat these specific benchmark scores with caution.