Research3h ago
Research
OpenAI Questions SWE-Bench Pro Reliability
OpenAI identified critical flaws in SWE-Bench Pro, a benchmark used to measure AI coding capabilities. The analysis reveals that noise and inaccuracies in the test set compromise model evaluation. This undermines current leaderboards for software engineering agents. Practitioners must now seek more rigorous, verified datasets to accurately measure real-world coding performance.