You deploy the code, you own the fallout. Here is how to survive on-call.
How exactly do you ensure an architecture decision made a year ago doesn't eventually cripple your system?
Data. "Without a benchmark and without data, you are just a person with an opinion," as Julio Falbo bluntly stated in our podcast conversation.
Too many developers—even Seniors—believe their job is done when a PR is merged on Friday at 6 PM. They check a couple of logs, close their laptops, and hope the on-call engineer handles whatever blows up over the weekend. Julio argues fiercely against this. As you step up in seniority, you have to transition to the mindset that if you push the code, your name is on it, and you own the outcome.
One of the most practical things Julio shared was his use of **ephemeral dashboards** in tools like Datadog. Instead of flooding the primary monitoring screens with noise, he creates temporary, custom metrics (sometimes toggled via feature flags) purely to track the rollout of his specific microservice change. Once the change is stable and proven, he cleans it up.
But observability isn't just about collecting data—it's about how you react to it. Julio dropped a golden rule of system alerts: Never create an alert that doesn't require an action.
We've all seen teams set up warnings for a circuit breaker retry that ultimately succeeds on the second attempt. If it self-heals and nobody needs to do anything, do not wake up an engineer at 3 AM. Only trigger alerts when a human actually needs to step in.
Key Takeaways:
- Document Everything (ADRs): Write Architecture Decision Records. If things break a year from now, you need to know why you chose that path over the alternatives. Do not rely on your memory.
- Build Ephemeral Dashboards: Create temporary, throw-away dashboards to intensely monitor a new deployment. Don't clutter the main company dash, but don't fly blind either.
- The Golden Rule of Alerts: If an error happens but a retry mechanism handles it gracefully, do not trigger a critical alert. Only page someone if an immediate, manual action is required.
Listen to the full deep dive here: https://www.youtube.com/watch?v=ReP23HQwe3A
How are you handling this in your architecture? Hit reply.
Cheers,
Silas Candiolli
P.S. I hope this 3-part series challenged your perspective! Keep an eye out for our next deep dive episode soon.
0 comments