Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
You know what? After finishing this article, I realized it would be even better to have a companion article that walks through a hands-on project to reinforce the concepts covered here.
I'm designing a practical stream-processing project you can learn from. See you next week ;)
For a long time, batch processing has been a commodity; it's easy to get your head around, it serves many use cases, and it’s straightforward to implement.
In contrast, stream processing has been like a high-end product; only big players want and use it. It has many more concepts to learn, and it’s more complex to implement than batch processing.
—
AI has changed a lot of things.
One of those is to give organizations a more compelling reason to adopt real-time analytics.
Instant business decisions, model training with the latest data, or agent observability.
(At least that is the thing a lot of data vendors are saying now)
Low-latency insights are usually the result of stream processing, the computational paradigm where data is processed continuously.
—
In this article, I share my experience and research so you can start learning stream processing on your own.
Important note: This is a learning guideline; it does not guarantee you can successfully run a real-time analytics project in production. That requires a lot of factors and cover them not in the scope of this article.
The first thing you have to learn is the difference between batch and stream processing. Because it will define everything you learn later, from motivation and performance monitoring to fault tolerance.
—
In batch processing, you gather the data over a period and then process it in a single operation.
For example, you read and process a Parquet file (with Spark or whatever framework you prefer), and then write the results somewhere.
Or, you take a snapshot of an OLTP database, transform it with SQL, and load it into the OLAP database.
The two scenarios share a property: the data is captured in a defined boundary. It could be an hour, a day, 3 days, or a week of data.
When a system processes data in a batch, it knows the size of the data, which simplifies the processing a lot.
—
For stream processing, there isn’t a defined boundary.
It follows the nature of the data flow.
Most of the data we work with doesn’t have a natural boundary; it is unbounded: from the user interaction on the website to data from IoT sensors (unless the website is down or the sensor runs out of power).
We place artificial boundaries on the data flow so the processing can happen seamlessly (batch processing)
When a database does aggregation and joins, it comes down to looking for records with the same key. With batch processing, the searching scope is defined. This makes the system know when the searching is done, thus finishing the aggregation or join.
With stream processing, aggregation and joins have, in theory, an infinite search space, which requires us to apply techniques (windowing and watermarks; we'll be back to these later) to limit it temporarily.
Regarding fault tolerance, you reload the batch data from the source (if the batch is still available). Typically, you reprocess the entire batch, not part of it.
In stream processing, things are different: you need to know which stream position to restart processing from, and the source must be able to “replay” the data stream from that position.
In a batch, the performance of the processing is usually defined by:
The data volume? 100GB, 1TB or 10TB
How long does processing take? 1 hour, 2 hours or 12 hours.
Typically, these two define and identify a less straightforward factor: your required throughput.
Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.