Introduction

In this post, I will explain my own view of main concepts of a data lakehouse platform.

First let start with an overview schema with main components. Next, I will detail each component. Before to conclude, I will propose a full detailed schema of my view of the Data Lakehouse Platform.

Overview schema

Explanation of each component

Foundation

Enables to secure, operate and monitor the platform.

Foundation - Observability

Logs, metrics and traces to supervise insights about the performance, health, and behavior of the platform.

Foundation - Security

Network, backup and access management to protect the platform from unauthorized access, disruption or destruction.

Foundation - CI/CD

Enables more frequent and reliable releases by automating build, tests and deployment.

Storage

Storage - Files

Enables to store data in a reliable way for batch processing.

Storage - Files - Storage layer - on premise

A distributed storage, scalable and highly available hosted on your own servers, can be a distributed file system or an object storage.

Storage - Files - Storage layer - cloud

A distributed storage, scalable and highly available hosted on cloud provider, can be an object storage.

Storage - Files - File format layer

File format used to store data in the storage layer, can be a row based format or a column based format.

Storage - Files - Table format layer

Metadata files on top of file format to bring ACID properties, time travel, data compaction, schema management, ...

Storage - Files - Interoperability layer

Enable interoperability between table formats.

Storage - Files - Catalog (metastore)

Manage table metadata and tracking metadata versions.

Storage - Events

Enables to store events for end-to-end streaming and real-time use cases.

Storage - Events - Event hub

The specialized storage system designed to handle high-throughput, real-time data streams, often using distributed architectures and optimized data formats for efficient ingestion, processing, and retrieval.

Storage - Events - Schema registry

The centralized repository to store and manage data schema.

Data flow

How the data go through the platform, from data sources to data consumers.

Data flow - Data sources

Stands for any data sources used to populate the Lakehouse, can be operational database, API, file or whatever else that contains data. This is external to the Lakehouse.

Data flow - Data ingestion

Extract and load process used to import data into the Lakehouse from datasources. No transformation is performed at this step.

Data flow - Medaillon architecture

Logical data organization from raw to data to business-level aggregated data.

Data flow - Medaillon architecture - Bronze

Raw flexible data without transformations or schema.

Data flow - Medaillon architecture - Silver

Validated, cleaned, enriched data with schema enforcement, prepped and aligned for analytical use.

Data flow - Medaillon architecture - Gold

Aggregated, anonymised data, optimized for read performance, trusted and suitable for end-users.

Data flow - Data serving

Expose data to the consumers, can be a federated query engine, an API, a data sharing, a warehouse or any technology that enables data exposure to end users.

Data flow - Data consuming

Stands for the end users that consume Lakehouse data, this is external to the Lakehouse.

Governance

Enables to manage, control and monitor data.

Governance - Data catalog

The centralized inventory of all data assets metadata that makes data more visible, understandable, auditable and leverage self-service access.

Governance - Data observability

The ability to understand and monitor the health, performance, and behavior of data and data systems in real-time. The key aspects are data quality, data lineage, data freshness, schema enforcement, volume and timeliness.

Governance - Data Observability - Data lineage

The process of tracking, tracing and visualizing of data's origin, transformation, and usage throughout its lifecycle from data sources to consumption.

Governance - Data Observability - Data quality

Measures how well a dataset meets criteria for accuracy, completeness, validity, consistency, reliability and fitness for intended purposes.

Governance - Data security

The process of safeguarding data throughout its lifecycle to protect it from corruption, theft, integrity or unauthorized access.

Compute

Enables to process data at scale.

Compute - Orchestrator

Automates the workflow and coordination of data pipelines, ensuring data flows seamlessly between different processes.

Compute - Data engineering

Pipelines to clean, transform and prepare data to ensure it is reliable, accessible, and ready for analysis.

Compute - Data Warehouse

Manages large datasets from various sources for analysis and reporting purposes.

Compute - Data science & AI

Extracts insights and knowledge from data using statistical and machine learning techniques and creates intelligent systems.

Full detailed schema

Conclusion

A Data Lakehouse Platform is a complex platform that involves many concepts and components. Master it can take a while but it’s useful to have an overview in mind.

I hope you enjoy this post about my view of the Data Lakehouse Platform. Don’t hesitate to ask questions or share your own view.