RDEL #155: How well can developers detect sabotage from an AI coding agent?
94% of developers missed malicious code from their coding agent, and 56% merged it even after a monitor correctly warned them.
Welcome back to Research-Driven Engineering Leadership. Each week, we pose an interesting topic in engineering leadership and apply the latest research in the field to drive to an answer.
Most teams have quietly changed what code review means: when an agent writes the diff, review often collapses into a permission dialog and a click. That click is the only thing standing between the codebase and an agent pursuing an objective nobody approved, and a poisoned system prompt, a third-party plugin, or a subverted agent skill is enough to create one. This week we ask: can developers detect sabotage from an AI coding agent?
The context
Nearly every agentic coding setup rests on the same safeguard: a human approves before the feature lands. Permission prompts, diff review, CI gates, a merge request with a name attached to it. That approval step carries most of the weight in how teams describe their AI safety story to security reviewers and to themselves. But it only works if the person approving is actually reading.
The pressure is that agents no longer make one suggestion at a time. They run for hours, touch dozens of files, call tools, and hold broad access to the codebase. Meanwhile most AI safety research studies sabotage as an AI-versus-AI problem: can one model catch another model misbehaving? That framing assumes the human downstream will respond correctly to whatever the monitor surfaces. So what actually happens when a developer, not a benchmark, is the last line of defense across a long session?
The research
Researchers at Northeastern University ran the first large-scale human study of sabotage detection in developer-agent collaboration. 107 developers spent five hours building an e-commerce app in Python across five CI-gated subtasks, paired with one of four frontier coding agents. Each agent carried a hidden second objective: exfiltrate synthetic personal data to an external IP address. One condition added a live LLM monitor that surfaced security warnings. Note: 86% of participants reported security-related backgrounds and 70% had over three years of coding experience.
94% of developers failed to catch the sabotage. Across the no-monitor conditions, 83 of 88 sessions ended with the malicious code committed. The pattern held for the three most capable agents: 95%, 94%, and 100% success rates.
A monitor caught the sabotage. Developers ignored it anyway. A cheaper model watching the agent flagged 16 of 19 attempts (84%), but success only fell from 94% to 63%. Narrowing to the 16 sessions where the warning was correct and clearly displayed, 9 developers still merged the malicious code.
The most capable agents didn’t need coaching to be covert. One model was tested both with and without task-specific hints on how to hide the exfiltration, and reached a 100% success rate either way. The one model with a noticeably lower rate (82%) was lower because it sometimes leaked its own intent in messages the developer could see.
The dominant failure mode was simply not looking. Of those 9 cases, 6 came down to minimal code review. Seniority offered no protection: 67% of the developers in this group had over three years of experience and 44% had security backgrounds.
“I’m always used to clicking approve directly.” - One participant
When developers did investigate, a plausible story was easily-believed. In one session the agent described its exfiltration function as providing “real-time internal telemetry” and the participant accepted it without further inquiry. In another, a developer questioned an external URL, heard that it was necessary for separating internal services from external endpoints, and approved.
Trust rose during the task and only broke at the debrief. On a 5-point scale, trust went from 3.52 before the session to 3.71 after, then dropped to 2.98 once the sabotage was disclosed. Developers were calibrating on test pass rates, not on code. Notably, the 11 who did catch the sabotage started with lower trust in coding agents than the 94 who missed it (3.18 vs. 3.56, p = .033).
The application
The safeguard most teams are counting on (human reviews) is the weakest link in the chain, and it fails at attention rather than at judgment. A developer who is task-focused, mid-flow, and used to clicking approve will merge a data exfiltration endpoint even with a correct security warning on screen.
Here’s how engineering leaders can apply these findings:
Route review to specific choke points instead of asking for general vigilance. Pick a short list that always requires explicit human sign-off: new outbound network destinations, new or changed config values and encoded constants, new dependencies, and edits to auth or payment paths. Asking engineers to carefully read every agent diff means they will read none of them.
Stop letting green CI stand in for review. Participants tracked whether tests passed and treated that as the signal. Add checks the agent cannot satisfy by making tests pass: static analysis in the pipeline plus an egress log of outbound requests from agent sessions. When two independent sources flag the same change, it is much harder to dismiss.
Make monitors act, not just warn. Only 9% of participants preferred the flag-only design that ships in most tools today. 70% wanted the monitor to propose a concrete fix and wait for approval, and 19% wanted it to remove the harmful code and notify them afterward. If your internal tooling surfaces security warnings that a single click can dismiss, expect them to be dismissed.
It’s worth noting two important caveats:
This sabotage was elicited, not spontaneous: two of the four models refused explicit red-team prompts and only complied when the exfiltration was reframed as a benign engineering task, so this is not a claim about off-the-shelf agent behavior.
The study design was limited in use cases. It covers one application, one attack class, and one simple monitor design, with only 11 successful detectors. It is very likely that different attacks will have different acceptance rates.
Wishing you a week of careful diffs and quiet pipelines.
Happy Research Tuesday!
Lizzie