The transformation of your data from data sources to final trainable dataset is data pipeline.
via xkcd @ https://xkcd.com/2054/
Built in Scheduler
Easily deployable on AWS
Receive large amounts of data regularly and sporadically, we’ve decided to perform this action once an hour, ensuring we don’t miss out on any important data.
A delay in data delivery because of the source platform’s poor performance
Data pipeline that hits an outside service to get some data and that service happens to be down (outage), or maybe the format of the data changes
Fluctuating amount of data being consumed and outputted from the source
Frequency of on demand polling the data source is conflicting duplicating scheduled polling.
Data gets clogged up at the slow step; this leads to a bottleneck in the pipeline.
No Loops, DAGs Only: Loops in pipelines are bad. There is a reason most (if not all) frameworks available enforce DAGs for pipeline building: cycles introduce unnecessary complexity and make it very difficult to reason about your system.
Avoid second order effects of writing custom pipeline from scratch, all pipelines require some custom effort; the real question is how much you actually need to do, and what framework will support you best.
Accomodate changing requirements for data pipelines (large or small) with backword comptability and flexibility for fast iterations i.e first-class support for dataflow allowing flexiblibility of pipeline to be designed to abstract the database logic into a separate set of tasks from the core data transformations. This would minimize the impact of changing those tasks (e.g. changing the underlying data warehouse)
Managing complex workflows via a proven Scheduler. When trying to run periodic tasks you decide to write an algorithm that takes the output of one job and uses it as the input for another job. This task depends on preceding jobs to be completed successfully, because they have to run in chronological order. If the first task didn’t run properly, it could create errors later on. Your scheduler should be able to handle those kinds of situations, without requiring you to be constantly on the lookout for bugs and errors while wading knee-deep into your code and its dependencies. You might fall into the trap of building these schedulers yourself. However, this is probably not the best idea, because it’s likely you will encounter challenges that can be easily overcome by automation.
Metrics, Metadata, & Logging: keeping track of data pipeline metrics and quickly diagnosing issues that you were previously unaware of [Grafana for metrics, Logstash for logging and sentry for errors]
Fallback design for unexpected external data sources scenarios: sample this scenario in your data pipeline that hits an outside service to get some data and that service happens to be down, or maybe the format of the data changes. Your pipeline needs to know how to handle cases like this to prevent errors from taking down the system. Mechanisms such as retries, error handling, fallback alternate data sources queueing can really solve this problem
Scaling Out in the face of bottlenecks: A delay in data delivery is a critical error in pipeline development, especially when that pipeline is used in a production environment. Cluster auto-scaling and web server load balancing are life savers: use them. A system of scale needs to be planned out and (preferably, as to avoid another aspect of maintenance) automatically enabled. One way of doing this consists of placing checks which are aware of the amount of data queued up to be processed for that step and scaling the amount of resources accordingly.
https://medium.com/the-prefect-blog/pipeline-pitfalls-57fe558cd76
https://otonomo.io/blog/luigi-airflow-pinball-and-chronos-comparing-workflow-management-systems/
https://kapernikov.com/a-comparison-of-data-processing-frameworks/
https://stories.dask.org/en/latest/prefect-workflows.html
https://medium.com/snaptravel/building-a-data-pipeline-to-handle-50m-events-day-with-a-startup-budget-part-1-636399bd00aa
https://towardsdatascience.com/data-science-for-startups-data-pipelines-786f6746a59a
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.