The (un)Importance of Measurements ⁉️

Feb 17, 2023 3:00 pm

Happy Friday,

Last week I wrote about OKRs, and it got a bit of attention, so I thought I'd write this time about the world of measurements.

Overwhelmingly people measure what is easy to count.

We count points, stories, lines of code, sign-ups, defects, bugs, releases, etc.

There are two problems with this:

  1. It is easy to overwhelm by all these counts
  2. The counts don't inform a decision

I tend to believe that you should measure what will help inform your decisions.

So before you go creating dashboards with all these counts and averages of counts, ask yourself, "What decisions will I need to make?"

Here're some common decision points that plague folks that you can easily create measures or signals for:

  • Is our product viable for our users?
  • Do we have enough capacity?
  • Do we need to invest in more quality?
  • What should we focus on next?
  • Where should we improve things?

These questions seem big, but for every group and product, there is a very approachable and narrow way to measure these questions. These measures will evolve and change over time just as your teams and products do too. Let's take the capacity problem as an example.

Let's say your development team has a velocity of 35. Well, that's neat, but it doesn't help us know if we've got the right amount of capacity for our product. So we're going to say capacity should minimally match the demands of our users. We could now look at things from the lens of user loss/gain per work item. So then, if we want our user base to grow 20% every month, we can use this to back into a capacity equation for Takt Time which is a ratio of production to demand, and then we know our capacity.

While this sounds complicated, these measurements are built out of those same counts above. Only now they inform a decision. If our takt time or capacity is too low, we can't grow at our desired rate. We need to grow the team or improve efficiency, and we have data to support it.

Some of you might say, "How do you know how many users a work item is worth," admittedly, this is where things do jump into the land of assumption, but that was already happening with things like story points and the nebulous idea of "Value." This makes the assumption discrete. If you really want to go crazy, you could look back over time to see how accurate those assumptions are and then adjust accordingly.

Anyway, I hope this gives an example of measures and why most places are better off throwing them away. However, recognizing the decisions involved can now refine those measures to be impactful, and you don't need many.

If you have questions, let me know what they are. Even better, if you have an example you'd like to share, let me know!


Ryan Latta

PS: I wrote a longer article about measures on my site if you're interested.