Research
51d agoOpenAI Questions SWE-Bench Pro Reliability
OpenAI identified critical flaws in SWE-Bench Pro, a benchmark used to measure AI coding capabilities. The analysis reveals that noise and inaccuracies in the test set compromise model evaluation. This undermines current leaderboards for software engineering agents.
The Signal
Practitioners must now seek more rigorous, verified datasets to accurately measure real-world coding performance.