RDEL #59: How does Copilot impact the day-to-day software engineering experience?
This week, we review a recently-released paper on how Github Copilot impacts the day-to-day of software engineering work, including output of work and impact of deadlines.
Welcome back to Research-Driven Engineering Leadership. Each week, we pose an interesting topic in engineering leadership and apply the latest research in the field to drive to an answer.
This week a Hacker News post on a recently released research study caused quite a buzz, particularly in its review of productivity improvements for Copilot. This week, we review that study and it’s implications by asking: how does Github Copilot change a software engineers day-to-day experience?
The context
The arrival of AI capabilities raises crucial questions for engineering teams; it is no longer about whether or not to adopt AI tools, but rather how it will reshape the nature of work on their teams. With AI taking over routine and repetitive tasks, there’s a growing need to understand how this shift influences the more complex, creative aspects of high-skilled work.
The conversation now turns to how these changes impact skill development, team structure, and the allocation of cognitive resources. It is yet to be seen how the role of engineers evolves in a landscape where AI handles many technical tasks, leaving engineers to focus on innovation, problem-solving, and strategic decision-making. At the same time, the balance between leveraging AI and maintaining human oversight becomes more nuanced, requiring teams to adjust their processes, priorities, and even their measures of success.
The research
Researchers collected and analyzed the workflows of software engineers from Microsoft, Accenture, and an anonymous Fortune 500 company, to measure the effects of generative AI on productivity, task complexity, and job satisfaction. At each company, software engineers were split into control groups and treatment groups to use Copilot. They used a combination of surveys and performance data to evaluate changes in work patterns after integrating AI tools into everyday tasks.
The key benefits discovered in day-to-day work were an improved pace of output, including:
26.08% increase in weekly pull requests
13.55% increase in weekly commits
38.38% increase in weekly builds
However, these results were much more dependent on the tenure of software engineers.
Tenure-based Adoption: Developers with shorter tenure (less than the median) were 9.5 percentage points more likely to adopt Copilot compared to their longer-tenured counterparts (84.3% adoption for short-tenure vs. 74.8% for long-tenure). Short-tenure developers were slightly more likely to accept Copilot suggestions compared to long-tenure developers.
Tenure-based Improvements: Short-tenure developers increase their outputs by 27%-39%, while long-tenure developers have more marginal gains of 8%-13%. Researchers note that this might be because longer-tenure developers were more likely to abandon the technology after a short while.
The application
For engineering leaders, these findings suggest that integrating generative AI into workflows can significantly enhance team productivity and job satisfaction. The gains are quite clear, and as mentioned in the introduction, it is no longer a question of whether teams should adopt tools like Copilot, but rather how.
That said, there are numerous key factors to consider for teams that want to effectively leverage the strengths of both the team and their technologies:
AI Augments, Not Replaces: Generative AI has been shown to improve productivity, especially by automating repetitive tasks. However, usage continues to prove that human oversight is critical to its success. Engineering leaders should focus on balancing AI-driven automation with the need for human judgment in higher-level problem-solving and decision-making.
Consider Skill Development and Dependency: While AI tools can help less-experienced developers complete tasks more quickly, there is a risk that over-reliance on these tools could slow the development of essential skills. Teams need to ensure that junior engineers continue to learn foundational skills and do not become overly dependent on AI suggestions.
A Potentially Growing Skill Gap: As discussed both in the paper and in Hacker News, AI may disproportionately benefit developers who are already skilled at leveraging new tools, potentially widening the gap between high performers and those who simply follow AI-generated suggestions. Leaders should be mindful of this gap and foster an environment where all team members can benefit from AI, while still focusing on critical thinking and independent problem-solving.
—
I hope these considerations help your team balance the use of these exciting new technologies more effectively. As always, happy Research Monday!
Lizzie