RDEL #6: Software Engineering Productivity: How do you measure it? (Part II)
A look into the SPACE framework, and how to put this framework into action in engineering orgs.
Happy Research Monday đŹ. This week, we continue our discussion on software engineering productivity. Now that we have defined software engineering productivity, how do we measure it?
The context:
Today, developer productivity is one of the biggest topics on managersâ minds. In fact, a Forrester study found that 87% of engineering managers rated improving developer productivity as a âhighâ or âcriticalâ priority, the highest of any process. Managers have the ability to pull data from tools like Github, JIRA, CircleCI and view countless metrics for their team, but this has led to a general sense of overwhelm. In a sea of information, which metric (or combination of metrics) actually matters?
Researchers have been studying this concept for decades. The evidence has become overwhelming that there is no single measure of productivity in software engineering. Rather, a team can measure productivity using a combination of metrics that span both system and perception metrics.
But what is that combination? In 2021, researchers finally found the answer.
The research:
In 2021, researchers at Microsoft published the SPACE framework, a groundbreaking framework that captures the most important dimensions of developer productivity. Since its publishing it has grown in popularity, with consensus across the engineering community that this constellation of metrics adequately captures the tension of measuring productivity in software engineering.
The SPACE acronym stands for:
Satisfaction and well-being
What is it: Satisfaction is how fulfilled developers feel with their work, team, tools, or culture. Well-being covers how happy they are, and how their work impacts it.
Example metrics: developer efficacy, perception of burnout, developer satisfaction
Performance
What is it: The outcome of the work a developer team does. This measure looks at whether the teamâs output does what it was meant to do.
Example metrics: Reliability of system, number of bugs logged, customer satisfaction
Activity
What is it: The actions or outputs completed in the course of performing work. Activity provides a high-level view of what type of work developers spend their time on.
Example metrics: Pull request velocity, deployment frequency, on-call participation
Communication and Collaboration
What is it: How people and teams communicate and work together. This includes information sharing, dependability, transparency and awareness of work, and effectiveness of problem solving.
Example metrics: Documentation discoverability, onboarding time, pull request cycle time
Efficiency and Flow
What is it: The ability to complete work or make progress with minimal interruptions or delay.
Example metrics: lead time for changes, number of interruptions, wait time for code reviews
The application:
This paper provides a great framework for measuring productivity in software engineers, but itâs also fair to say that it takes a few steps to implement. To put this framework to use, here are some useful tips:
When implementing this framework, start with a goal - what do you want your team to focus on?
Then, pick a combination of metrics that span at least 3 categories of SPACE, mixing both system and perception metrics. Be mindful of how many metrics you pick - too many metrics can lead to confusion and lower motivation.
To capture perception metrics, create a developer survey to share out on a regular basis. Treat perception metrics as leading indicators of change, and system metrics as quantifiable evidence.
Talk to your team on a regular basis, because these metrics matter to more than just the manager. When productivity is more comprehensively measured, engineers will see it as a team sport to work on together. Conversely, when productivity is not comprehensively measured (ie without perception metrics), this can make improving productivity more of a âtop downâ initiative and engineers might not feel fairly represented.
As a reminder from our last post, productivity is a measure of outputs (effectiveness) divided by inputs (efficiency). In the SPACE model, effectiveness is related to satisfaction and performance. Efficiency is driven by activity, communication/collaboration, and efficiency/flow.
With a clear definition and framework for developer productivity, your teams are ready to realize their best potential đ. Wishing you all a fantastic Research Monday!
Lizzie
From the Quotient team