RDEL #79: How can engineering teams apply the SPACE framework to improve productivity? (with Dr. Jenna Butler, co-author of SPACE)
What metrics a team should pick, how often they should change, and how to turn metrics into actions (and measure improvement).
Welcome back to Research-Driven Engineering Leadership. Each week, we pose an interesting topic in engineering leadership and apply the latest research in the field to drive to an answer.
It’s been three years since the SPACE framework for developer productivity first emerged, and since then it has grown in both popularity and success. While the framework offered a couple examples, it did not detail how to us it regularly to drive consistent, iterative improvements in productivity. This week, we sit down with SPACE co-author Dr. Jenna Butler to ask: how can engineering teams apply the SPACE framework to improve productivity?
Dr. Jenna Butler is a Principal AI Researcher at Microsoft, and one of the co-authors of the SPACE framework. She is at the forefront of the development of new strategies for increasing productivity and wellbeing in the workplace. Her current research focuses on the impact of AI on developers (including the 2024 New Future of Work Report), as well as the emerging trends in hybrid and remote work.
With that, here’s our interview with Dr. Butler.
Q: It’s been three years since the SPACE framework first emerged, and it’s popularity has grown tremendously. While SPACE is a great framework for selecting metrics, but it does not specifically prescribe what metrics to pick. How do you recommend selecting the metrics that make sense for a team?
Dr. Butler: We recommend picking at least 3 metrics, from different categories of “SPACE” to look at together. As far as which ones, it comes down to what matters to a team at a specific time. For instance, if you know your team struggles with burn out, tracking a metric from the satisfaction and well-being category is likely very important; if your team is constantly being interrupted, you might want to make sure you have something from the efficiency and flow category.
I also recommend teams include one metric that is something they’re already great at. That way, they can celebrate a win and make sure it doesn’t drop when pursuing growth in other areas.
Q: What are some challenges teams may run into when selecting metrics?
Dr Butler: One challenge is deciding which metrics are best for a team - you need to balance choosing a small enough number to focus, but not making it so small that you don’t get the data you need. Any time there is a framework that is designed for an entire field, you will have to grapple with how it applies to your specific context.
Beyond that more conceptual issue, teams might also face the issue of not having the telemetry to track what they want to track. There might need to be some data engineering and implementation in order to get the metrics that matter most for you.
Q: Should every team should use a standard set of metrics, or change those metrics depending on a specific goal or context?
Dr. Butler: Each team has a unique set of needs, since it will be made up of a unique set of people. I love the idea of gathering a broader set of data from the team that captures numerous dimensions of their developer experience (you can use a tool like Quotient to support this). That initial data can inform what metrics make sense for your team to track. As the team grows and changes over time, the particular SPACE metrics will likely change as well.
Q: The idea that SPACE metrics should change depending on the goal or context of a team is something that isn’t often covered, but feels very important to discuss. How should a team evaluate when it’s time to change the metrics they are looking at?
SPACE metrics should be held for long enough to learn something from them. I don’t believe changing them too often (ie every quarter) would be wise because it takes time for humans to achieve behavioral change. That said, if a metric is no longer helpful for learning or decision making, it is probably time to change it. At the end of the day, the primary value of the data is to serve the team in their efforts to improve their productivity.
Once you have seen a plateau in a metric and feel it has settled into a place the team is comfortable with, you can choose to keep it as a benchmark for long term monitoring, or graduate to something else. What you graduate to will depend on the next change a team is undergoing - whether its growth, transformation, or something else.
Q: One thing the SPACE framework doesn’t talk about too much is who is responsible for understanding this data. Should people at all levels of an organization be viewing the same SPACE metrics? Or should CTOs and VPs be looking at different metrics than say, an Engineering Manager or Lead?
I think it’s important to be able to slice the data to the level you want to investigate. If I was an engineering manager, I’d likely want to see SPACE metrics for my team*. I might also want to know about other teams that are doing really well in certain areas to learn from them. As a VP or CTO, I’d want a higher-level view. Either way, I’d likely want to look at the same set of metrics.
I’m a big fan of the OKR framework, and one of its main super powers is “Focus and Commit”. When the whole ship is steering in the same direction, the boat moves a lot better! However, if you have certain teams that need to improve in particular areas, especially over the short term, having them look at a slightly different set of metrics to achieve that goal has value. Ideally, I’d want to see a set of metrics that are shared across the whole organization, with the ability to add one or two unique metrics on a team by team basis. This will keep consistency in the org but allow for individual teams to measure what is unique for them.
*These should be at a high enough altitude that there aren’t privacy issues, or risks of using the data in a negative way to identify “bad performers” - which doesn’t work!
Q: We’ve spent a lot of time talking about how to choose the right metrics, but I want to end our conversation with what I believe is the most important question. How do you recommend a team uses this information to actually drive outcomes? How do you go from information to action?
The first step is to first share that data with the whole team, and create space for the team to discuss how the metrics align with their own experience. I’m a big fan of data democracy! Sharing the data also allows the team to reflect on which goal they believe will have the best outcome, and buy into that goal. A previous study showed that a reflective goal setting process in software engineering can lead to more positive behavior change and higher impact.
After choosing a goal, I recommend teams select an action that they believe addresses that goal. Allowing the team to pick an action that aligns with their way of working, and whose progress can be measured, will increase the success outcomes on the team. As for sourcing the right action, there are numerous studies (many covered in previous RDEL reports) that can generate ideas, as well as experiences from other teams in the company. I also know that tools like Quotient can recommend research-backed actions, which can lower the bar to taking the right action.
—
A huge thank you for Dr. Butler for joining us and providing clear, actionable advice on how to apply the SPACE framework to improve developer productivity.
As always…Happy Research Tuesday!
Lizzie