AI Coding Agents on a Legacy Java Monolith: What Actually Worked

Most AI coding-agent comparisons measure how quickly a model can generate a feature in a clean repository. Enterprise engineering teams face a different problem: upgrading and refactoring large, interdependent applications where an apparently correct patch can fail CI, break an undocumented integration, introduce a data risk, or make future maintenance harder.

Ortem Technologies LLC ran an illustrative controlled evaluation of Opus 5, GPT-5.3-Codex-Spark, and Muse Code against the same legacy Java monolith. The goal was not to determine a universal "best AI coding tool." It was to measure the work that matters in a real modernisation engagement: how much migration could be accepted without engineers materially rewriting the agent's production code.

Read this first. Results are illustrative pilot data created for article framing, not a verified benchmark. A production study should replace these placeholders with logged task outcomes, build reports, reviewer notes, and engineer-hour tracking.

Short answer: In this illustrative trial, GPT-5.3-Codex-Spark completed the largest share of the migration backlog unaided, at 74%. Opus 5 delivered a close, more balanced result for complex refactoring. Muse Code was economical for contained changes but needed more architect and engineer intervention on cross-module work.

---

Benchmark methodology

Every agent received the same conditions:

• The same anonymised legacy Java codebase and baseline commit
• The same 15-task modernisation backlog
• The same CI environment, build commands, time budget, and test suite
• The same limited architecture notes and clarification policy
• The same review gates: build, automated tests, security scan, architecture review, and regression checks

A task was marked unaided accepted only when the agent-produced patch passed every defined gate without an engineer modifying production code. A patch that compiled but failed contract testing, required business-logic correction, or created an unsafe migration did not receive full credit.

---

Headline results

Agent	Migration completed unaided	Modules accepted first pass	Engineer-hours saved per accepted module	Median human rework per module	CI pass rate
GPT-5.3-Codex-Spark	74%	12 of 15	15.6 hours	2.4 hours	80%
Opus 5	68%	11 of 15	14.2 hours	3.1 hours	73%
Muse Code	57%	8 of 15	10.8 hours	4.7 hours	60%

Figure 1. Illustrative benchmark results: GPT-5.3-Codex-Spark achieved the highest unaided migration completion rate, while Opus 5 delivered comparable value across legacy-code refactoring tasks.

Results are illustrative pilot data created for article framing, not a verified benchmark. A production study should replace these placeholders with logged task outcomes, build reports, reviewer notes, and engineer-hour tracking.

---

Which tasks were evaluated?

The backlog combined predictable upgrade work with high-risk modernisation tasks:

Task category	Examples
Runtime and dependency upgrades	Java LTS migration, Maven and Gradle modernisation, CVE remediation
Framework work	Spring upgrade, security configuration updates, API compatibility checks
Refactoring	Domain-service extraction, controller and service decoupling, shared-model cleanup
Data and jobs	Schema migration, rollback plan, legacy batch-job conversion
Quality and operations	Test coverage, observability, Docker, CI/CD hardening, performance remediation

This mix matters because AI agents generally perform better when requirements are explicit and the change is local. Their reliability falls when they must infer hidden business rules, reason about database state, preserve undocumented interfaces, or make architectural trade-offs.

---

Where the agents broke

Agent	Failure point	What happened	Required intervention
Opus 5	Spring-security upgrade	Changed legacy filter sequencing and interrupted an existing authentication flow	5.5 hours
Opus 5	Database migration	Produced a correct forward migration but omitted a robust rollback path	2.0 hours
GPT-5.3-Codex-Spark	Batch-job modernisation	Preserved main output but allowed duplicate processing after retries	3.8 hours
GPT-5.3-Codex-Spark	Performance task	Refactor passed tests but created a database query-plan regression under load	4.2 hours
Muse Code	Domain-service extraction	Created circular dependencies by over-extracting shared classes	6.0 hours
Muse Code	Test generation	Added tests with weak assertions that did not catch a seeded regression	2.9 hours

Every one of these defects passed the agent's own checks and was caught downstream — by integration testing, DBA review, replay testing, load testing, architecture review, or mutation testing. The failures cluster in the same places: filter ordering, rollback steps, retry semantics, query plans, module boundaries, and assertion depth. These are all properties of a system rather than properties of a file, which is exactly what a model working from local context tends to miss.

---

What engineering leaders should take from this

AI coding agents can reduce the cost and elapsed time of a Java modernisation programme, but "generated" does not mean "ready to merge." The most reliable operating model is agent-assisted delivery with human acceptance gates:

• Use agents for repository discovery, upgrade planning, dependency changes, test drafting, repetitive refactoring, and documentation.
• Require senior engineers for security boundaries, data migrations, performance-sensitive paths, and architecture decisions.
• Measure net hours saved after reviews and remediation — not code volume, token count, or the speed of an initial patch.
• Preserve a complete audit trail: task prompt, branch, code diff, build logs, scan output, reviewer findings, and final outcome.

---

Modernising a complex platform and want the review gates designed before the agents start writing code? Ortem Technologies combines AI-enabled development with hands-on architecture, cloud and DevOps delivery through its AI and ML solutions practice and application modernization service. See our AI-powered application modernization guide, or book an architecture review →.