Up until my current role, I’ve always worked on data teams of people with various skill sets. Usually there is a data engineer, a few data analysts, a manager, and an analytics engineer.
In my current role, the data team is a full-stack analytics engineering team. Each of us works across the entire pipeline of our project, from configuring tasks in Snowflake for ingesting source data to building dashboards in ThoughtSpot.
It works well because there is no “passing of the baton”, which tends to slow things down.
Before, analytics engineers would need to communicate with data engineers to ensure the right data was available. This typically took a few weeks due to prioritization and communication issues. Then, after the analytics engineer would work on their data model, they’d have to talk with the data analyst to explain how it works. That would be a back-and-forth process to understand how the data is modeled and how it could be used to build dashboards. Because things often get lost in translation, the data analyst would then need to have the same business conversations with stakeholders that the analytics engineer already had.
In the past, I had projects that dragged out for months due to lag in communication, poor prioritization, and misunderstandings when they should have only taken a few weeks.
All of this could have been prevented if each of us worked on our own projects end to end.
AI now makes it easier than ever to bridge these skill gaps. Data analysts can build data pipelines with the help of AI, build scalable dbt projects, and write Python scripts to help automate processes. Previously, they may not have had these skills, or they may not have been as advanced as needed. Now, a basic understanding can take you quite far.
With AI making technical work less intimidating, the business knowledge of data analysts and analytics engineers become even more prized. You can’t have well-thought out, scalable solutions without understanding the in’s and out’s of the business and being able to understand your stakeholder’s problems. This is where data engineers need to lean in.
If data engineers and data analysts are both expected to know the technical skills and business communication skills because of AI, that then means they are essentially morphing into analytics engineers.
To keep up with the times and transition into a full-stack analytics engineer, data engineers need to:
simplify complex technical solutions and concepts so that stakeholders understand what they are proposing
understand common business metrics like AOV, CPC, LTV, GMV, etc.
prioritize projects based on business needs and revenue potential
communicate with stakeholders to get to the root of their problems
For data analysts to grow into a more technical full-stack analytics engineering role (and not get left in the dust by AI), they need to:
master the basics of version control, specifically Git
evaluate raw data for data quality issues and clean it for downstream usage
model, document, and test data in a dbt project
build data pipelines end-to-end including data ingestion, data modeling, and orchestration
Luckily, I’m currently teaching you how to build a data pipeline using tools like ClickHouse, dbt, and Claude Code. We are 3 weeks in and it’s not too late to join the challenge!
Data roles aren’t going anywhere, but they are evolving.
I don’t want to fear monger, but I’m not sure the data analyst role will exist in a few years. It’s not that the skillset they have isn’t important, it absolutely is. In fact, I think the skillset of talking with the business and translating their needs is actually the hardest to learn! Sorry data engineers, have fun learning that 😜
It’s just that data analysts now need to expand their skillset to also focus on the technical skills, which I think is the easier thing to learn. Start now and you’ll have time to grow into these as they become more and more important as AI evolves.
I’m curious to hear your thoughts on this.
Have an awesome week!
Madison
Looking to upskill your analytics engineering skills?
Hire me to train your data team in analytics engineering principles (reply to this email)
Buy my ebook on the ABCs of analytics engineering
Invest in a paid newsletter subscription to participate in the Data Pipeline Summer challenge
Join the waitlist for my data analyst to analytics engineer course
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.