The Easy and Open Data Lakehouse Platform
Overdue Last read · last published · next check
Last read 14 hours ago, longer than this feed's 6 hours schedule.
This is Part 14 of a 15-part Apache Iceberg Masterclass. Part 13 covered streaming approaches. This article is a practical walkthrough of working with Iceberg on Dremio Cloud, covering table creation, data ingestion, optimization, semantic layer construction, and AI-powered analytics. Table of Contents Getting Started Step 1: Sign Up and Connect Storage Dremio creates an Open Catalog for your…
This is Part 13 of a 15-part Apache Iceberg Masterclass. Part 12 covered Python and MPP engines. This article covers the three primary approaches to streaming data into Iceberg tables and the operational trade-offs each creates. Iceberg was designed for batch analytics, but most production data arrives continuously. Streaming ingestion bridges this gap by committing data to Iceberg […] The post…
This is Part 12 of a 15-part Apache Iceberg Masterclass. Part 11 covered metadata tables. This article covers the two main ways to access Iceberg data: directly from Python libraries and through MPP (massively parallel processing) query engines. Table of Contents The Python Ecosystem for Iceberg PyIceberg: Native Python Access PyIceberg is the official Python library for Apache […] The post Using…
Iceberg 1.11.0, released in May 2026, contains the full implementation of the v3 spec(ification), including features like deletion vectors, the VARIANT type, and row lineage. But what comes next? V4, obviously, as four comes after 3. However, it's still early days on the v4 spec, so we are a ways off from the next version […] The post Apache Iceberg v4: An Efficiency Rewrite of the Table Format…
This is Part 11 of a 15-part Apache Iceberg Masterclass. Part 10 covered maintenance operations. This article covers the metadata tables that let you inspect Iceberg table internals using standard SQL. Iceberg exposes its internal metadata as queryable virtual tables. You can use them to check table health, debug performance issues, audit changes, and build monitoring dashboards. No […] The post…
This is Part 10 of a 15-part Apache Iceberg Masterclass. Part 9 covered how tables degrade. This article covers the four maintenance operations that keep Iceberg tables healthy and the three approaches to running them. Table of Contents The Four Maintenance Operations 1. Compaction (File Rewriting) Compaction reads small files, merges them into optimally-sized files (128-512 MB), and […] The post…
Apache Ossie is currently undergoing incubation at The Apache Software Foundation (ASF). If you've been following the Open Semantic Interchange project — the open specification for semantic layer and ontology — there's an important update. The project has been accepted into the Apache Incubator under a new name: Apache Ossie (Incubating). The spec, the community, and […] The post Apache Ossie…
This is Part 9 of a 15-part Apache Iceberg Masterclass. Part 8 covered embedded catalogs. This article explains the five ways Iceberg table storage degrades and how to detect each problem before it impacts query performance. An Iceberg table that works well on day one will not work well on day 365 without maintenance. Every append, update, and […] The post How Data Lake Table Storage Degrades Over…
Apache Parquet is the recommended file format used in every modern data platform, and for good reason. But what are those reasons? And would it really matter if you stuck with CSV? The short answer is "YES". The slightly longer answer is "Yes, because columns". The full answer is below, so keep on reading to […] The post What’s The Deal With Apache Parquet? appeared first on Dremio .
This is Part 8 of a 15-part Apache Iceberg Masterclass. Part 7 covered the traditional catalog landscape. This article examines a newer approach: embedding the catalog directly inside the storage layer. Traditional Iceberg architectures have three components: the query engine, a standalone catalog, and object storage. Embedded catalogs collapse the catalog into the storage layer itself, reducing…
For most of its history, the semantic layer was considered a solved problem. You built it once, business users queried it wherever it lived, and (hopefully) everyone would agreed on what "revenue" meant. However, much like the information in your data dictionary, the popularity of the semantic layer went stale and businesses turned to new, […] The post The Semantic Layer: From Human Shortcut to…
