Models and Measures

Mar 22, 2024 2:26 pm

Happy Friday!

One of the things I've done my whole career is try to quantify the impact of improvements.

I distinctly remember obsessing over the size of the iPhone app builds I created and working hard to optimize the assets to keep the size below a certain point so that cellular app updates would function. If it were larger, you'd have to be on wifi. Another job I was joyful about was when I needed to look at how long API requests took and find ways to keep them within contractual parameters.

You might be looking at those and think that's nothing special, so let me take it a step further.

In the job directly after the one I mentioned last, I modeled the entire development process across 12 products and showed how a continuous delivery pipeline could save $120k every month. Then, I modeled another group's development capacity and simulated it to show we were 10 years behind schedule before rescoping to something we could accomplish in 3 months.

There are more examples like this I could share, but attempting to quantify things has been a part of how I work forever, and while it may be imperfect, I do believe it has provided me with a certain level of insight that others appreciate.

So, let me tell you a couple of quick things to help you get started to do the same.

The first step to modeling is to study the actual process. It needs to be the one that happens, and not the one we wish happened. From here, we can look at the timings of each step. While knowing the duration of a step is important, often, the wait times between steps are even more important. With this data you'll wind up with a series of steps and a set of ranges for durations and wait times and who is involved.

Now, with this information, you can look at an improved version of this same model. In general, I recommend starting with a 10-20% improvement model. Even this will present as a huge improvement, even if it's wildly conservative compared to the plan.

When you want to show the improvement and its impact, you often need two more pieces of information: the frequency and cost. If you're improving a process that happens once a year, the improvement needs to be significant. If it happens many times a month, then a small improvement will be compounded quickly. The cost is fairly easy to back into. One of my favorite ways to do this is to calculate an hourly rate for those involved and use that.

Now, let me show you an example. Let's look at saving a day of development time for a team. A work item has at least 3 people working on it. There will be a primary developer, a senior developer doing a review, and someone from QA testing it. On average, it takes 6 days for the team to complete an item, and they complete 8 or so items each sprint.

With this information, the only thing I'm missing is the cost. So here are some numbers I like to use broken down to day rates:

  • Developer: 80k/yr or $336/day
  • Senior Developer: 100k/yr or $408/day
  • QA: 60k/yr or $250/day

You can obviously adjust all of these to be closer to what your place is, but I start with this because they tend to be conservative.

So now we're going to say that a 6-day process takes 5 days. The really basic calculation will be: (Sum of day rate) * work items per sprint.

This would be: (336 + 408 + 250) * 8 = $7952 per sprint

Over a year of 24 sprints (Assuming 2-week sprints): $190,848

But what if this applied to more than one team? Multiply that by the number of teams.

Want to be more granular? Look at how many days of involvement for each person. Say the code review takes 2 days, and the test takes 1 day. You can be even more specific.

So, let's go back to this first $190k cost reduction. The question is, how much will it cost to get those savings?

Now, while this is a very narrow example, hopefully, it's inviting a whole lot of "What about" questions. For example, if it takes one day less to complete work, will they be able to complete more work throughout the year? Maybe, and you could model that too! What if its more people involved or its more variable? You can incorporate the variability using something like a Monte Carlo simulation if you need that level of confidence, but most of the time, it's not important.

In other words, this basic example can serve as a launching point for more involved and detailed modeling to help communicate the impact of improvement both as cost-saving and value-adding. Rarely is one truly isolated improvement, but I like to start with the easiest-to-model improvement and go from there.

I have a lot of energy for this topic, so I'd love to hear your stories about modeling things or help you work through a situation!

If you'd like to learn more and you've got an appetite for some dry material, I recommend the book, "How to Measure Anything."