Sitting through dbt Coalesce conference this week, my biggest takeaway is that data is less trustworthy than ever and people are fired up about it. Often times, an analytics engineer is really just a pissed off analyst who has the tools and motivation to make things better for everyone else.
The momentum around dbt is palpable. The rise of the “analytics engineer” is a real movement born out of necessity (and genius branding).
Analysts can no longer sit idly by and fix their data in the visualization layer or wait on tech teams to build a better pipeline. The analytics engineering movement will take back control over data quality and insights.
As a product manager turned data analyst, I always felt helpless when it came to fixing, cleaning, and transforming the underlying data I was working with. I used to pride myself on how long and crazy my ad-hoc queries were. Then, I was introduced to LookML and then dbt. Both tools were completely empowering and transformational for my own work as an analyst.
In many places, the problem is only getting worse. The explosion of data and SaaS tools combined with the ease of automating “Extract and Load” means you can have true unadulterated data chaos in under an hour.
The data chaos is reaching fever pitch and fueling the rise of the Analytics Engineer.
A few key trends are converging:
- The shift from ETL to ELT means folks who know SQL have the power to transform and clean data once it arrives in the warehouse. Data warehouses are powerful enough to handle the transformation workload
- Tools like dbt enable analysts with SQL-based workflows and enable them to work like software engineers do.
- Data consumers have come to simply expect data at their fingertips yet at all times.
- Organizations (large and small) are recognizing the need to invest more resources in data modeling.
If your organization isn’t on board with the movement, it’s time to get on the train. It’s almost certain that people in your organization don’t feel like they can get the data they need, can’t make sense of their data, or generally don’t trust the data in their reports or dashboards. Analytics engineering sets out to solve these issues. Are you hiring for one? (Plug: We can also help.)
Other key and related trends:
- Testing tools/features like dbt tests, Great Expectations, and Looker data validations will continue to grow in popularity and necessity. Run automated tests to ensure your data (And your transformation logic) is correct.
- End users of data will begin to have more insight into if their data is “clean”. Features like like dbt Exposures with help with that (wicked cool!).
- Beyond the transformation layer, dashboards are also broken. They can’t be source controlled, deployed and tested like the rest of the stack. They are rigid and can’t be customized to truly deliver a compelling user experience that data consumers need. (If this bullet resonates, please reach out: we’re solving this problem for lots and lots of companies).
As always happy to chat about data and I tweet about it a lot too.