Django is the perfect example. It has sane defaults and consistent convention, but isn’t enforced. It’s just Python you can modify however you need as you grow, you’re not locked into any vendor, it has batteries included that will solve problems for you later you aren’t even aware you’ll need to solve up front.
But most importantly, it creates a shared cognitive model that drastically lowers friction within a team.
The author is saying, there isn’t an open solution with strong convention but also escape hatches that enables a data team to hold a shared mental model (like Django).
And that’s consistent with my experience. Anyone on our team can jump into a Django issue, but looking at someone’s flavor of SQL thing they built is like, “give me a day to wrap my brain around what you’re doing and what data this is and I’ll get back to you”.
One problem is, I think, that SQL is great at handling data that’s consistently structured and normalized, but most data projects are loaded with one-off exceptions, and SQL is less great at handling those, and is better handled in a general purpose programming language. And that’s where some kind of “structured chain of units of work” is needed.
Created by dbt Labs, it's a hugely popular tool that "enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications."
Last I checked, their Slack community had 50,000+ members.
But most importantly, it creates a shared cognitive model that drastically lowers friction within a team.
The author is saying, there isn’t an open solution with strong convention but also escape hatches that enables a data team to hold a shared mental model (like Django).
And that’s consistent with my experience. Anyone on our team can jump into a Django issue, but looking at someone’s flavor of SQL thing they built is like, “give me a day to wrap my brain around what you’re doing and what data this is and I’ll get back to you”.
One problem is, I think, that SQL is great at handling data that’s consistently structured and normalized, but most data projects are loaded with one-off exceptions, and SQL is less great at handling those, and is better handled in a general purpose programming language. And that’s where some kind of “structured chain of units of work” is needed.