What is SWE-bench Multimodal v2?

AI systems can now fix bugs and write new features in software on their own. The standard test for this, SWE-bench, uses only Python code and mostly text-based problem descriptions, which leaves other kinds of software uncovered. SWE-bench Multimodal is a test set that checks whether an AI can fix bugs in visual, user-facing JavaScript software, where each task includes at least one image, such as a screenshot, in its problem statement or tests. Version 2, dated September 1, 2026, keeps 480 tasks chosen for reproducible evaluation, with flaky or ungradeable tests removed. The original was described in a paper first submitted in October 2024 by John Yang, Carlos E. Jimenez and colleagues.

Written from primary sources and reference pages, and checked against them by an independent AI fact-checker. Drafted with AI assistance.