--- title: "Testing AI generated code: fast to ship, hard to test" description: "Testing AI generated code gets harder once an agent writes it fast. What a Harvard study shows about review, and how to test agent code before it merges." author: "Usman Masood, Software Developer" published: 2026-10-28 updated: 2026-10-28 url: https://www.mantaq.co/blogs/testing-ai-generated-code --- # Testing AI generated code: fast to ship, hard to test Testing AI generated code gets harder once an agent writes it fast. What a Harvard study shows about review, and how to test agent code before it merges. An AI coding agent helped me ship features fast. Testing what it wrote got harder, and that is the part people underestimate when they bring an agent into their work. Testing AI generated code is harder because the writing step speeds up while the checking step stays the same size, or grows. ## Why an agent makes shipping feel fast You give the agent a task and it comes back with a change. For me that meant features reaching the finish line sooner than they used to. That part is real, and nobody who has tried it needs convincing. But writing was never the whole job. In their [Harvard working paper on software production](https://fion.ac/jellyfish.pdf), economists Fiona Chen and James Stratton estimate that engineers spend 30 to 40 percent of their time writing code. The rest goes on planning, review and testing. Speed up one step of a pipeline and the next step becomes the one you wait on. ## Where the time goes after the code is written The paper studies 718 software firms and finds that agents did what everyone expects. After adoption, lines of code went up 30 percent, commits 20 percent and pull requests 23 percent. The number people tend to skip comes next: the rate at which tracked issues and features were finished did not change in a way the authors could call statistically significant. The review queue explains the gap. Time from opening a pull request to merging it rose 49 percent, from a baseline of about seven days. The share of pull requests with changes requested nearly doubled, comments per pull request rose 35 percent, and more of the team ended up reviewing code. ![More code went in, and merging it took longer: lines of code up 30 percent, pull requests up 23 percent, review comments up 35 percent, time to merge up 49 percent.](https://images.ctfassets.net/he34i99xhodg/4rIF2UcE8w5ELUVTYBdQy8/1d83cf86e79602d743249bd2c678a79a/review-bottleneck.png) [Ars Technica's write-up](https://arstechnica.com/ai/2026/10/ai-coding-agents-generate-more-code-but-not-more-software/) adds that by March 2026 most of these firms already used AI to help review, yet it wrote under a quarter of the review comments. People were still doing most of the checking. ## Why testing gets harder with agent code The study measures review, not testing. But the two sit next to each other, and the reasons review slows down are the reasons testing does. Agent code arrives polished. SitePoint's [guide to testing AI-generated code](https://www.sitepoint.com/testing-ai-generated-code) points out that it is well formatted and idiomatic, so the usual signal is gone: a messy pull request used to hint at a messy idea. The failures it lists are quiet ones. A method that never existed in the library version you use. A loop that is off by one. Logic that handles the happy path and breaks on an empty string. An API pattern that was current two years ago. Then there is the agent's own test suite. Momentic's [testing guide](https://momentic.ai/blog/testing-ai-generated-code) warns that a generated test can share the assumptions of the implementation it checks. If the agent misread the requirement, it misread it twice, once in the code and once in the test, and everything comes back green. Green is not proven. A passing build does not show that checkout, login or a saved preference works, and more output means more changes waiting for that proof. ## What the study measured, and what it does not It is a working paper, and nothing in the version I read shows that peer review is complete. The data is observational and ends in March 2026, so it says nothing about agents released since. It describes a pattern across hundreds of firms, not what will happen on your team. It also does not claim that agents are a bad trade. Chen and Stratton say the review bottleneck is where the extra speed gets absorbed. Ars Technica notes that teams are still learning when to point an agent at a problem and when not to, so the balance may shift. What the paper does give you is a place to look: if shipping feels fast and finished work does not, look at what happens after the code is written. ## How to test agent code: tests first, then diff review, then a real run The common thread in the vendor guides is to decide what correct means before the agent writes anything, and keep that definition out of the agent's hands. In order: 1. Write the expected Behavior as tests or acceptance criteria first, including the error paths. 2. Give the agent a constrained task and the files it may change. 3. Run lint, type checks and the existing tests. Run them in the order that fails fastest. 4. Read the diff as a person. Look for hard-coded values, deleted tests and edits outside the task. 5. Run an end-to-end check against the built app, and link the result in the pull request. The agent can still write tests. Have it draft them, then check that each one asserts the Behavior you need and not a detail of how the code happens to work. ### Where a check should block We hold document systems to the same standard. In [Stay Funded 360](https://www.mantaq.co/case-studies/stay-funded-360), the grant packet download is blocked while a receipt is missing, and the system names the record that is missing. Every value AI reads off a receipt is confirmed by a person before it is recorded. Extraction saves the typing; it does not get to decide what a packet says. Code from an agent deserves the same treatment: it drafts, a check blocks, a person confirms. ### What more checking costs The limit is cost. A bigger suite takes time to run, maintain and triage, and a heavy end-to-end layer on every change can slow you down all over again. Start with the one flow a change can break and add from there. So the speed is real, and it counts once checking keeps pace with writing. If you build document workflows and want a second view on where a check should block and where a person should confirm, our [services page](https://www.mantaq.co/services) shows how an audit, a build and ongoing support fit together. ## Frequently asked questions ### How do you test AI-generated code? Define the expected Behavior before the agent writes, run the existing tests, read the diff yourself and run an end-to-end check against the built app. Treat it like code from a colleague you have not worked with yet. ### Why does AI-generated code take longer to review? In the Harvard study, review time from opening a pull request to merging it rose 49 percent after agents arrived, and changes were requested far more often. The code looks clean, so problems hide in logic and edge cases. ### Can AI write the tests for its own code? It can draft them, but a generated test can share the code's wrong assumptions. Check each test against the requirement, and keep the expected Behavior independent of the implementation. ### Do AI coding agents really make developers faster? They write more code faster: the study found 30 percent more lines of code. It did not find a statistically significant rise in finished work, and its data ends in March 2026. ### What should be checked before an agent's pull request merges? A passing test run, a person reading the diff, a passing end-to-end run against the built app, and a link to that run in the pull request.