How to Structure an AI-Ready Data Warehouse

Data warehouses are the source of truth of any modern data stack. It is where the data is stored and queried, making it one of the most important components.

Every tool in your data pipeline will touch your data warehouse.

With AI, everyone is using MCPs and various tooling to connect directly to your data warehouse, making security and guardrails more important than ever.

My data team has been working for months to make sure our PII is safe before folks on our team can begin to connect these tools to Snowflake.

In today’s rendition of Data Pipeline Summer, we will first learn what makes a data warehouse safe and ready for AI. We will cover:

• how to protect PII data and raw data

• how to manage users and roles

• how to connect AI tooling to your warehouse

We will end with this week’s challenge- loading the raw data for our challenge into ClickHouse.

If you missed last week’s introduction to data pipelines, be sure to give that a read before diving deeper into data warehouses.

Let’s get started!

Customers trust us to keep their data safe. AI is a tool that threatens that safety, especially when the data is email addresses, social security numbers, and credit card numbers. Some data is more dangerous when leaked than other, but that doesn’t mean all of it doesn’t matter.

Whenever AI touches your data, a black box is created. Sure, in some cases it tells you exactly what it’s doing with it, but others you don’t know what’s happening under the hood.

Because of this, it’s our responsibility as analytics engineers to protect this data and ensure AI only has access to the data we feel comfortable it having.

Analytics cloud data warehouses work just fine with AI. They allow AI to easily query structured data. In combination with a context layer (which we will talk about at the end of the series), you can query structured data as well as take advantage of unstructured data like your Notion documents and Slack messages.

Some teams may want to take advantage of storing large amounts of unstructured data outside platforms like Notion and Slack. For this type of work, it’s best to use something like a data lakehouse which stores structured and unstructured data. However, if you aren’t already using a lakehouse, you most likely don’t need one to take advantage of AI.