A Case for Tests
Mar 11, 2022 11:33 pm
I know all of you wake up each Friday in anticipation for my emails, and last week I missed and this week its coming late. I apologize for that, and you're welcome and encouraged to reply with a verbal thrashing.
I've suddenly had to start traveling again and its messing me up.
There's something peculiar about me that you might know by now. I tend to love the things in the software industry others find unlovable. I love topics like testing, security, legacy systems, and so on. I was in a fascinating meeting just this week where upcoming legislation changes around CCPA and the FTC's updated Safeguard's Rule and almost thought to bore you to tears with me rambling about its implications.
But I'm not.
And you should, if you're in software, get a quick synopsis on the changes. Our industry's time of doing whatever it wants and feeling invulnerable is ending rapidly, and there are massive opportunities for companies who see it and can get ahead of it.
Instead, I want to share a story from a client related to testing software. The team gathered for their morning stand-up, and after everyone did their discussion they moved on to talk about bigger topics. One of which was a weird bug in a core part of their system. By core I mean the company lives and dies by getting this right.
Anyway, I sat on the call and listened as a group of the CTO and 4 senior developers tried to make sense of things. Now, you might think they were coming up with a solution, but for the first 40 minutes they were trying to understand their current solution. They then moved on to debating the merits of each other's ideas.
I chimed in close to an hour in and asked, "Do you have enough tests around this part of your system where you could make changes and have confidence there wouldn't be new surprises?"
They got quiet, some said yes, then the lead said, "No, we actually need a few more tests before I'd say that's true, but why are you asking?"
I said, "Well, you've been talking for close to an hour and a lot of that time was just trying to understand what your current solution actually does. If it had all those tests you could kind of skip right to trying stuff because you'd know if you got it right or not within a minute or two. So I was wondering if this conversation is for our own edification or part of the work because it isn't tested well enough."
Now, what happens next isn't that important, because what I want to do is use this snapshot as to the age-old argument that testing isn't practical or economical argument.
Now, I'm making this bet that as I write this that writing the tests would have been cheaper than the one phone call. I'm going to find out as I write just as you do as you read.
So we have a 1 hour phone call with 4 senior devs and a CTO. Lets see some back of the napkin stuff.
I'm going to take an average salary for everyone at $100k for easy math. That gives an hourly rate of $52 an hour, and the price of this one hour call at $260.
Now, let's take one of those devs and have him write the 5 tests and spend an hour on a fix. Each test takes 20 minutes. Testing costs $87 and the fix costs $52 for a total of $139.
So there you go, the call cost twice as much as writing a few tests. Some of you probably caught that the cost for the call didn't include the fix, that would've cost extra as well as the manual testing.
There're plenty of things to poke at with this example, but let me ask you this, does this get better or worse as the next issue comes up? Running those tests takes minutes, but having another call every time something goes wrong takes quite a bit.
Something to think about when there are discussions about deferring testing. It is never cheaper or more practical and rarely faster.