People say I'm a control freak.
But there's a good reason.
I don't like when the rug pulls out from under me. When I fall down, I want it to be my fault, not someone else's.
Here's how it goes: I start a new project with a new team. They use some online interface to run queries against a cloud data warehouse. It's looked like the following:
- Hosted Jupyter notebook running against AWS EMR
- Google Vertex AI notebook running against Google BigQuery
- Snowflake worksheet running against... Snowflake
In each case, there's a separation between where the user writes their code (some notebook or worksheet) and where the data is crunched (the data warehouse).
Things work great in the early days.
Then poop hits the fan.
Mishap #1: The worksheets/notebooks disappear
True story: The company's cloud infrastructure team was doing some spring cleaning. Along the way, they accidentally removed storage that supported my cloud notebooks.
My work for the past few months was gone. There were no backups. End of story.
Mishap #2: My worksheets/notebooks are inaccessible
Another true story: The cloud security team was updating IAM permissions. In doing so, they locked me out of my notebook. All those computations and sexy graphs were in my notebook. I needed them for a presentation.
But... I couldn't get to my work. 😐
I opened an IT ticket. Two days passed before I could get back to the notebook.
Needless to say, the stakeholder was not happy.
Mishap #3: Cloud data warehouse migrates from "legacy" worksheet to "modern" workspace
I get it. Cloud data warehouses need to innovate to set themselves apart from competitors. But do you really need to re-do the space I write queries?
The idea of a cloud worksheet is simple: see the data available, query the data, and see the results.
Every few years, cloud data warehouses release a new take on the cloud worksheet or cloud notebook. And they give a way to move (i.e. force) users to the new way of doing things.
Sometimes the migration is smooth. More times, it's been rough.
One time, my code in legacy worksheets was lost during conversion. Another time, the migration was me copy-pasting code from the legacy worksheet to the modern worksheet... one. at. a. time.
This happened when Google BigQuery moved from "classic saved queries" to BigQuery Studio.
It's happening to me now as Snowflake moves from worksheets to workspaces.
At best, the transition distracts me from getting my job done. At worst, bugs arise, and I potentially lose my work.
The saying goes, "Fool me once, shame on you. Fool me twice, shame on me." I'll try any web-based UI of a cloud data warehouse. I actually like the ease of use of cloud worksheets. But as soon as a mishap happens, I'm out.
That's when I set up a LOCAL notebook workflow, using Jupyter Notebooks or Marimo Notebooks. I write some boilerplate Python code to connect to the data warehouse at the top of the notebook. And then I'm back to querying the data warehouse from a notebook.
But this time, I OWN the notebook. The code and query results sit on my machine (backed up on OneDrive). Those sexy graphs are mine to gawk at.
This gives me back my focus. I can keep doing my job (flirting with data) without the distraction of learning a new platform with a buggy rollout.
And when the cloud workspace goes down or the security team makes an "oops," I keep the fruits of my labor. 😀
Best,
KP
P.S. Marimo notebooks are new for me. I'm using them to query against Snowflake, and I love the interactive and out-of-the-box features. Let me know if you've found any cool use cases for Marimo.
In case you missed it
Here's a blog and video on shoving files into the cloud data warehouse. Things don't go as planned, and we troubleshoot pesky files.
Dad joke
What do you call a camel with no humps?
Humphrey.
0 comments