RDEL #36: What are the most common conversations developers have with ChatGPT?
This week we review the most common types of conversations developers have with ChatGPT, and how engineers share conversations with their team.
Welcome back to Research-Driven Engineering Leadership. Each week, we pose an interesting topic in engineering leadership, and apply the latest research in the field to drive to an answer.
It’s no secret that software engineers frequently use search tools for help with software design and implementation (like Google, Stack Overflow, or even Reddit). Since launching in 2022, ChatGPT has quickly become a valued tool to add to the toolkit. But the chat model is newer for this type of search work, and has rapidly evolved as ChatGPT has become more intelligent. So this week we ask, what are the most common conversations developers have with ChatGPT?
The context
Foundation models like ChatGPT have revolutionized software engineering tasks, including coding, testing, and debugging. One of the most valued uses of ChatGPT for software engineers is to get support with software design and implementation. It’s popularity is attributed to its ability to engage in complex, multi-turn conversations, offering detailed guidance and solutions across various aspects of project development.
Since it’s still relatively new in production (since 2022), engineering teams are still defining what use cases make the most sense for ChatGPT, and how best to use tools of this nature to improve productivity. To analyze that further, this paper dives into the most common use-cases today for engineers working with ChatGPT.
The research
Huizi Hao and a team of researchers analyzed 580 shared ChatGPT conversations from GitHub, categorizing the types of inquiries made by developers and the nature of these interactions. The researchers then analyzed the inquiries into distinct categories of interaction, as well as unique sharing behaviors of those types of interactions.
The researchers found that:
Engineers’ conversations with ChatGPT can be summarized in 16 different categories, with the most common categories being code generation, conceptual questions, how-to guides, issue resolution, and code review.
A significant portion of the interactions are multi-turn conversations (33.2% in PRs and 36.9% in issues), indicating developers' efforts to refine or expand upon ChatGPT's responses through follow-up inquiries.
Developers use shared conversations with ChatGPT to facilitate their contributions in specific roles, such as authors of PRs, code reviewers, or collaborators on issues. The top purpose of sharing ChatGPT conversations with others was to reference to a source of solution and reference to a potential solution. They were most frequently shared as links in code review comments or issue comments.
Researchers also used their analysis to create a flow chart that represents how engineers engage and follow-up with ChatGPT in conversation (below).
The application
This paper analyses the most popular ways developers use ChatGPT to assist them in their software engineering work. It highlights ChatGPT's adoption for a wide array of tasks including coding, conceptual knowledge, and new topic understanding, reflecting its versatility in addressing diverse software engineering inquiries.
For teams wondering whether they are getting the most value out of ChatGPT, consider the different ways you can be using ChatGPT to improve team productivity. For example,
ChatGPT can do more than just help with high-level conceptual understanding, but can be used to assist with direct code generation.
Sharing conversation snippets with teammates can also aid with knowledge transfer and clarification in code reviews or issue topics.
Note: be mindful that if teammates are using AI-assisted tools for code generation, there could be a risk of more insecure code making it into production.
—
We hope this research gave your team new and creative ways to leverage AI to improve productivity. Have a great week, and happy Research Monday!
Lizzie
From the Quotient team