AI Coding Agents on a Legacy Java Monolith: Scope, Rework and CI Pass Rates

Most agent comparisons run on greenfield toy problems. Legacy migration is where the difficulty actually lives: implicit coupling, undocumented behaviour, and tests that pass for the wrong reasons.

This piece frames a controlled trial of three coding agents against the same legacy Java monolith, across a 15-module migration scope.

Read this first. Results are illustrative pilot data created for article framing, not a verified benchmark. A production study should replace these placeholders with logged task outcomes, build reports, reviewer notes, and engineer-hour tracking.

---

Scope of the trial

Each agent received the same 15-module migration backlog against the same codebase snapshot: a Spring upgrade, a database migration, batch-job conversion, domain-service extraction, performance work and test expansion. Every module went through the same review gate — CI, code review, and where relevant a DBA or architecture review.

Four things were tracked per module: how much of the scope the agent completed without human help, whether the module was accepted on first review, engineer-hours saved against the manual baseline, and the median human rework hours needed afterwards.

---

Headline results

Agent	Scope completed unaided	Modules accepted first pass	Engineer-hours saved per module	Median rework hours per module	CI pass rate
Opus 5	68%	11/15	14.2 h	3.1 h	73%
GPT-5.3-Codex-Spark	74%	12/15	15.6 h	2.4 h	80%
Muse Code	57%	8/15	10.8 h	4.7 h	60%

Figure 1. In this controlled migration trial on a legacy Java monolith, GPT-5.3-Codex-Spark completed the highest share of work unaided, while Opus 5 delivered the most balanced results across code quality and repair effort; Muse Code required the most human correction.

Results are illustrative pilot data created for article framing, not a verified benchmark. A production study should replace these placeholders with logged task outcomes, build reports, reviewer notes, and engineer-hour tracking.

---

What broke

Aggregate scores hide the failures that matter. Every one of these defects passed the agent's own checks and was caught downstream.

Agent	Module	What broke	Severity	How detected	Intervention	Final status
Opus 5	Spring upgrade	Broke legacy authentication flow through outdated filter ordering	High	Integration tests and manual QA	5.5 h to rewire security chain	Recovered
Opus 5	Database migration	Correct schema change, but missed rollback steps	Medium	DBA review	2.0 h to add rollback script	Accepted with rework
GPT-5.3-Codex-Spark	Batch-job migration	Logic converted correctly but introduced duplicate processing on retries	High	Replay test and fixture comparison	3.8 h to add idempotency guard	Recovered
GPT-5.3-Codex-Spark	Performance fix	Improved code structure but caused a query plan regression	High	Load test	4.2 h to optimise indexing	Recovered
Muse Code	Domain-service extraction	Over-extracted shared classes, creating circular dependencies	High	Architecture review	6.0 h to simplify boundaries	Rejected then reworked
Muse Code	Test expansion	Added shallow tests that passed but missed a seeded defect	Medium	Mutation test and defect replay	2.9 h to strengthen assertions	Accepted with rework

---

Reading the results honestly

Completion share is not delivery speed. The agent with the highest unaided completion also produced two high-severity defects — a retry-path duplication and a query plan regression. Neither was visible to unit tests. Both were caught only because the review gate included replay and load testing.

The failures cluster in the same places. Filter ordering, rollback steps, retry semantics, query plans, module boundaries, assertion depth. These are all properties of a system rather than properties of a file, and they are exactly what a model working from local context tends to miss.

Shallow tests are the quiet risk. The test-expansion failure is the one to worry about: tests that pass, look like coverage, and catch nothing. Without mutation testing that defect ships.

---

What this implies for your pipeline

If you are putting agents on legacy work, the review gate matters more than the model choice. Three gates caught every defect above: replay testing for anything touching retries or queues, load testing for anything touching queries, and mutation testing for anything that claims to add coverage.

Budget for rework rather than assuming savings. On these figures the spread between best and worst median rework was 2.3 hours per module — across a 15-module scope that is most of an engineer-week.

---

Planning a legacy modernisation and want the migration gates designed before the agents start writing code? Ortem Technologies' AI and ML solutions practice builds these pipelines for production systems. See our AI-powered application modernization guide for the broader approach, or book an architecture review →.