Hugging Face reported on the release of ScarfBench, an open benchmark built to test AI coding agents on cross-framework migrations across Enterprise Java systems. Srikanth Govindaraj Tamilselvam published the benchmark on June 30, 2026, targeting migrations between Spring, Jakarta EE, and Quarkus.
Unlike traditional software engineering benchmarks that evaluate generated code against reference implementations, ScarfBench measures operational outcomes. Test applications must successfully build, deploy correctly, and pass behavioral validation to meet the benchmark criteria.
Performance findings
Evaluations of state-of-the-art coding agents showed that even the top-performing models achieve under 10% behavioral success on framework migrations. Hugging Face noted that build success rates consistently exceed deployment success rates, which in turn exceed behavioral success rates. Build execution alone does not accurately reflect whether a modernized application works.
Target framework choice significantly affects overall difficulty, with migrations targeting Jakarta EE presenting the greatest challenge. Agents also showed clear self-assessment errors during whole-application testing. Claude Code reported successful builds for 29 out of 30 whole applications, but independent verification showed only 22 actually built successfully. The model also misclassified a successful build as a failure in one instance.
Migration effort
Tracking agent navigation across application layers showed that framework migration operates as an iterative dependency-resolution process rather than a direct source code translation. Agents spent most of their time repeatedly revisiting configuration files while resolving dependency injection, database queries, and framework semantics.
Operational and environmental issues created additional bottlenecks during testing. Coding agents regularly struggled with Docker cache inconsistencies, port connectivity failures, and Maven wrapper build tooling errors, which stalled validation even after code edits were complete.
ScarfBench maintains an open evaluation infrastructure, public datasets on Hugging Face, code repositories on GitHub, and a live leaderboard at scarfbench.info. Research detailing the benchmark is published on arXiv under reference 2605.06754.
