Research1h ago
Research
OpenAI Challenges SWE-Bench Pro Reliability
Analysis from OpenAI reveals critical flaws in SWE-Bench Pro, a benchmark used to measure AI coding capabilities. The study identifies noise and inaccuracies that skew performance results. These findings suggest current coding evaluations lack the precision needed for rigorous model comparison. Practitioners should treat these specific benchmark scores with caution.