RDEL #156: How does AI-assisted coding impact code comprehension?
AI assistance cut the odds of correctly explaining code nearly in half, and the loss was largest on the tasks where AI helped most.
Welcome back to Research-Driven Engineering Leadership. Each week, we pose an interesting topic in engineering leadership and apply the latest research in the field to drive to an answer.
A large pull request is opened, with all tests passing. But when a reviewer asks why a function was written the way it was, the author isn’t sure. Increasingly, this is a common experience as developers rely on AI assistants or agents to generate code. This week we ask: how does AI-assisted coding impact code comprehension?
The context
Code ownership has always been an important part of the review process. It’s the reason a developer can defend a design choice in review, find the cause of a failing test quickly, or make a change six months later without breaking three things. It also generally isn’t well documented, and rather lives in the head of whoever wrote the code. Teams have historically been able to assume it comes bundled with authorship.
AI assistants break that assumption. When a developer accepts a suggestion, they get working code without necessarily having reasoned through the tradeoffs behind it. Most of the evidence so far tells us AI tools make developers faster and let them close more tasks. But speed and comprehension are separate things, and almost no one has measured them in the same experiment.
The research
Researchers ran a controlled experiment with 69 participants, each completing two coding tasks - one with AI tools allowed and one with them blocked. Afterward, they answered technical questions about the code they had just written, with no AI and no internet, and the share they got right measured code ownership.
Here were their findings:
AI more than doubled how much of the task got done. Median task completeness was 84 out of 100 with AI, compared to 35 without. The model estimated a gain of about 43 points, holding experience, education, and task constant, and the effect held across every category of participant.
Completeness of task with versus without AI (and broken up by seniority) Ownership dropped when AI was in the loop. The median share of correctly answered questions fell from 100% without AI to 87.5% with it, and in the model AI assistance cut the odds of a correct answer nearly in half (odds ratio 0.58). The direction held among the professionals and researchers on their own, though there were few enough that that result did not reach statistical significance.
The comprehension loss concentrated in the questions that matter most. When researchers split questions into closed ones (where does the code do X?) and open ones (why did you implement it this way?), the effect was significant only for the open questions.
The tasks where AI helped most were the tasks where understanding suffered most. On the task with the largest AI productivity boost, writing a JUnit test suite from scratch, median completeness went from 42.5 without AI to 100 with it. Ownership on that same task fell from a median of 100% to 67%.
The speed gain didn’t hold up. Participants finished in a median of 43 minutes with AI versus 63 without, but everyone did their two tasks back to back, and once the researchers accounted for people warming up on the second one, that gap disappeared.
Note: The completeness gain and the ownership loss both survived the same check.
Implementation time with versus without AI (and broken up by seniority)
The application
AI raised output across every group in the study, from bachelor’s students to working professionals. The cost landed somewhere else,though - developers struggled to explain code they had written twenty minutes earlier, and that gap was widest on the tasks where the tool did the most work.
Put the reasoning in the pull request description. Ask authors to name the design decisions they made and the alternatives they ruled out. That moves comprehension to before the merge, and it leaves reviewers and future maintainers important context to support maintenance.
Track quality alongside throughput. These costs surface downstream, in debugging, on-call, and change requests. When ticket volume climbs while time-to-diagnose climbs with it, thin comprehension of recently shipped code becomes a plausible cause worth investigating.
Apply extra scrutiny where AI performs best. Test suites, boilerplate, and CRUD endpoints are the work these tools handle well, and the study suggests comprehension erodes fastest there. Those are strong candidates for a human walkthrough before merge, especially for code someone else will maintain.
Wishing you a week of code you can explain.
Happy Research Tuesday!
Lizzie