Snowflake Spend Optimization, Data Modeling, Airflowctl, In-demand Data Skills, and Streaming Data

Instacart paid Snowflake at least $28 million in one-year costs - Instacart is going public soon and in their S-1 prospectus they disclosed their Snowflake spend. This was their spending in 2021 and they managed to reduce it to $15 million in 2022. To be fair, this number by itself does not tell us much. Regardless of the actual number, it is clear Instacart is spending tens of millions of dollars on Snowflake alone. This does not surprise me. When a company is growing and money is cheap, efficiency may not be a priority. Just like social networks are designed to grab your attention and time, cloud computing makes it easy to overspend. Need a new FTP service, it is easy to spin up a new Virtual Machine using an image from the marketplace. Looking at alternatives may require more experience with other services or digging hard through documentation. In Snowflake’s case, the spending happens either because of over-provisioned compute resources or using a large amount of storage. We will do a deep dive into Snowflake costs in a future post. In the meantime, I would like you to point to an excellent free resource (it’s a Tableau Public dashboard) to keep tabs on your Snowflake spending.

Credits: @laurenbalik

Relationship between data model, granularity, and use cases — Should you use large tables or model your data using a semantic layer in the BI tools? Ryan Janssen has distilled this in a nice visual. The answer is often highly contextual. Are you trying a build a quick prototype by Monday and your boss has asked to build something on a Friday? BI Tools like Power BI/Tableau may be a great fit. Are you trying to bring data from different granularities together? Look into modeled data in a data warehouse. One Big Table approach especially when you are storing granular data is very flexible. Just be aware of non-additive metrics such as mean or median.

airflowctl - Airflow is a workflow orchestration tool. What does it mean? Let us say, you are trying to move some data from one source system, transform it, and then load it into a destination system (e.g. a database) you need to —

A. Create a task (a process or a script) to move data from one Amazon S3.

B. Second task to transform the data (e.g. create aggregations, summary)

C. Load this processed data into a destination database.

In addition to all of the above, you need to schedule the above activities and handle failures and retries. Airflow helps you centrally accomplish this for all of your data engineering activities.

Get up and running with Airflow on your PC/Mac with a one-liner.

pip install airflowctl && airflowctl init my_airflow_project --build-start

credits @kaxil

Streaming Data Tutorials - Mitch Seymour is a data engineer and has done fantastic illustrations to explain streaming processing concepts, Apache Kafka, and Flink. Check out this Redpanda Masterclass slide deck with 200+ pages of illustrations. He has created a nice animated illustration of stream processing and Kafka concepts called “Gently Down the Stream”

Top in-demand skills for data professionals

Following are top in-demand skills based on job postings in datanerd.tech

Credits: Benjamin Rogojan also known as “Seattle Data Guy”.

Looking for a job in data? Learn SQL, Python, and AWS.