RDEL #66: What is the impact of AI adoption on engineering organizations? (2024 DORA Report)

In the first of our three-part review of the 2024 DORA report, we'll look at the (good and bad) ways that AI has impacted the performance of engineering teams this year.

Welcome back to Research-Driven Engineering Leadership. Each week, we pose an interesting topic in engineering leadership and apply the latest research in the field to drive to an answer.

The 2024 DORA report is out! This year, we’ll be doing a three-part review on the top findings and their application on software engineering teams. To start, we look at the deepest analysis of engineers using AI and ask: how has AI impacted developer productivity in 2024?

Note: Our team is running a quick, 5-question survey on technical and social drivers of productivity on engineering teams. If you want to contribute to our dataset and get a view of what engineering leaders are measuring today, please fill this out.

The context

The DevOps Research and Assessment (DORA) group’s annual report marks a decade of research into the practices that drive high-performing software engineering teams. This year’s report surveyed more than 39,000 technical professionals across diverse industries and roles, providing a robust data set on key performance metrics like software delivery, productivity, and team dynamics. To deepen understanding, the team also conducted select in-depth interviews to get deeper perspectives on emerging themes like AI adoption. By applying rigorous statistical analysis, the report identifies the practices, organizational structures, and cultural factors that correlate with improved team and organizational outcomes.

Over 75% of respondents are relying on AI in one or more of their professional experiences. Now that there’s been such widespread adoption of AI on engineering teams, researchers are able to see the downstream impacts on engineering productivity, both good and not so good. For this week’s RDEL, we review what those downstream impacts are.

The research

Researchers dedicated a portion of the study to the impact of AI adoption across the spectrum (from engineers to broad organizations). Using the data from their broad organizational study, researchers analyzed and modeled the impact of increased AI adoption on how engineering teams work.

First, they looked at the data on how adopting AI improves an individual’s success and well-being. They found that the greatest improvements come from flow, job satisfaction, and productivity.

Surprisingly, the data also suggests that more AI adoption is actually decreasing time spend doing valuable work. Researchers hypothesized a possible explanation for this: by increasing productivity and flow, AI is helping people work more efficiently, and that efficiency is helping people finish up work they consider valuable faster. AI is not stealing the valuable work, but expedites the time to realize that full value. (They call this the “vacuum hypothesis”). Unfortunately, AI hasn’t yet cracked the “toilsome” work which leaves huge opportunities for improvement in developer experience.

A representation of the hypothesis that AI is helping with valuable work, but not toilsome work

Next, researchers looked at how AI is impacting team processes and workflows. They found that AI is most significantly associated with improvements in documentation, code quality, and code review speed.

Researchers also reviewed how AI impacted delivery performance and stability. Interestingly, they found that increasing AI adoption has a negative impact on delivery performance and delivery stability.

Researchers hypothesis that this may be caused by a well-known and easy-to-forget principle of development quality: batch sizes. In a world of increased code generation, respondents can product a much greater amount of code in the same amount of time, which is likely creating larger changelists. Prior research by the DORA team has consistently shown that larger changes are slower and more prone to instability.

Finally, the DORA report wouldn’t be complete without the most downstream impacts - organizational, team, and product performance. While organization and team performance improve with increasing AI adoption, product performance does not.

Researchers hypothesize that organization and team performance improve because they rely heavily on communication and knowledge sharing (where AI is alleviating some bottlenecks). Product success, however, involves numerous additional factors including human intuition and expertise (i.e UX design, broader creativity) that AI has not impacted as much.

The application

This study does a great job of analyzing on how a year of AI adoption has actually impacted the engineering organization at the highest and lowest levels. There’s so much to take away, but for this weeks application we’ll highlight some of the researchers’ top takeaways on AI adoption:

1. Have a clear vision for how AI will empower your organization. When you have a clear vision and the right tools to realize that vision, teams will view AI as an accelerant to their efforts.

2. Build a culture of continuous iteration and experimentation. AI tools are ever-evolving, which means in order to capture the full value of AI, teams need to be flexible to trying new things. Teams with high autonomy will feel more comfortable testing out new use cases of AI to improve their development efforts. Be careful to make sure you are maintaining a high bar of quality as you do so, and consider an evaluation framework to see what tools are worth continued use.

3. Know there are trade-offs. AI has many advantages, but it’s not perfect. Be mindful of the areas where AI can introduce more team challenges, as in decreasing delivery stability and reducing time spent on valuable work. As AI advances those challenges will evolve, so it’s important for teams to constantly evaluate how the team’s developer experience is evolving with it.

—

This has been an extremely exciting year for technology adoption in engineering, and we hope this week’s RDEL gave a taste of what the benefits (and costs) of that adoption have been.

Have a wonderful week,

Lizzie