Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
Many junior or graduate students, or even those who work in the data engineering field for 3-4 years, reach out and ask me something like this:
“I feel I lack fundamentals; I don’t know what I'm missing and don’t know where to begin “
The reasons are usually:
“Vendors are doing a great job in abstracting the complexity behind the scenes; data engineers now just need to interact through the interface.”
“AI could do anything, so I don’t have motivation to learn”
“Too many things to learn““
—
I have been thinking about this problem for a while.
I wrote about the data engineering roadmap here, but I think it’s quite overwhelming for those who want to gain the fundamental knowledge fast and efficiently.
—
From first principles, data is just bytes: 0 and 1 written to your disk. When working with data, you do one of these, or a combination of them:
Storing the data in 0 and 1.
Bringing those 0 and 1 into RAM for CPU processing.
Moving those 0 and 1 to other machines
That’s it.
No matter how complex the pipeline, how big the data platform, or what the business requirements look like, any data physical actions can be observed with only these three.
Thus, if one can gain knowledge of these three, I believe they would have 80% of the necessary data infra fundamentals; all other things can be learned from there.
And, to make the learning more concrete, I chose three technologies/frameworks to learn:
Parquet
Arrow
Spark
In the rest of the article, I give you exactly what you should learn from Parquet, Arrow, and Spark, and how you can leverage them to scale your learning to nearly all other data systems.
In the scope of this article, I only go through each one and list out which concepts/topics you should learn so you can build your learning based from those topics; in other words these concepts have the highest ROI: when you learn them, you can definitely learn nearly anything related to data infrastructure.
Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.