I lied

I know the logic is false. But I use it anyway. 😬

As a data engineer, you inevitably come across old code.

You know the ones I'm talking about. It's the 1000+ lines of SQL migrated between 3 platforms by 4 engineers who no longer work at the company. It represents the business logic signed off by Jethro before he left 6 years ago.

And now it's given to you.

Your job is to convert the logic for yet another platform.

You dig through the code, and you find something's a bit off. The SQL logic doesn't accurately reflect the business processes. You have a choice:

1. Make the correction.
2. Or leave things the way there are.

A younger me fixed code immediately. The current me leaves things the way there... at least to start.

Here's why: there's something more important than writing correct code.

It's building trust.

Yes, I'm talking about the squishy emotion the stakeholder has for you, that feeling that lets him believe your work is legit.

Because here's the lay of the land: This logic has been used for years, maybe decades. No one has questioned it. It's been through multiple platforms and supports the executive dashboard the CEO uses. You know... that dashboard that all other reports MUST match.

And me? I'm just the new guy. I'm unknown. Worse, I'm a potential risk.

So when I come and say, "Hey, this code looks wrong" on day 1, it falls on deaf ears. 😞

But... if I can replicate the logic to perfectly match the totals from the old platform... I build trust. I establish yourself as someone who can get stuff done.

After I've built that trust, I can circle back to the stakeholder and say, "Hey, I saw some things during the migration that we should look at." Now the stakeholder is more likely to believe me. 😃

It takes longer, but matching imperfect logic in your early days on a team sets the stage for long-term impact. I've done this many times (Oracle to AWS EMR, Hadoop to GCP BigQuery). The platforms and companies change, but the problem is consistent.

If stakeholders don't trust YOU as a human, they have no reason to trust the data you curate.

So the next time you see logic that seems off, you may want to replicate it in the new platform.

Note the issue. Bring up your concerns later, AFTER you've acquired some street cred.

I'm rooting for you.

Cheers,

KP

In case you missed it

Slowly Changing Dimensions: In my first year as a DE, I mucked up the CFO's report. SCDs saved my butt.

Here's a blog and video on commonly used SCD types.

Side note: My non-techie wife laughed at this one, so you nerds should love it.

Dev Joke

A SQL query walks into a bar, sees two tables, and asks "May I join you?"