PowerBI Fabric and Data in General
I have been using AI in VS Code and Onelake, initially trying to make sense of Chat with your data (without much success, that thing is very hard and we need some breakthrough), and more recently for data engineering. I noticed something: AI has become useful enough in the last couple of months that it Continue reading "AI, dbt and Iceberg are already changing data engineering"
duckrun is a package I built using AI exclusively, to solve pain points I hit when using Fabric Python notebooks. I like DuckDB very much, but I was tired of manually discovering table names every time and writing long Python deltalake code just to write a Delta table, so I combined those two packages under Continue reading "duckrun"
1. delta-rs is an ACID Delta writer delta-rs implements the Delta Lake protocol natively. merge, update, and delete go through optimistic concurrency control on every commit. No external coordinator, no catalog service. Two writers race for the same version of the log, one wins, the other retries. All you need is a path. No metastore to provision, no catalog endpoint, Continue reading "How far…
Four years ago I wrote a blog about using DuckDB with Power BI in DirectQuery. It got a fair number of likes on LinkedIn 🙂 along with the one comment I didn t want to hear: how does this work in production? (Craig, if you re reading this, you were right.) Back then I thought the technology Continue reading "Power BI with DuckDB, 4 years later"
DuckLake supports multi-writer just fine — but only if your catalog is a real database, like Postgres (there s some interest in SQL Server support too). But if all you have is object storage and a SQLite or DuckDB file as the catalog, you re stuck with single-writer: object stores aren t real filesystems, so the DB file Continue reading "Ensuring safe single-writer for DuckLake on OneLake using…
TL;DR: Iceberg s value is sociological, not technical. And if you care about lightweight, single-process engines like datafusion and duckdb, it s probably your best shot at first-class lakehouse support with Wide interoperability. The first real data engineering work I did was an ingestion pipeline built on pandas and Parquet with Hive-style partitioning — an environment where Continue reading…
Microsoft Fabric lets you dynamically configure the number of vCores for a Python notebook session at runtime — but only when the notebook is triggered from a pipeline. If you run it interactively, the parameter is simply ignored and the default kicks in. This is genuinely useful: you can right-size compute on a job-by-job basis Continue reading "Dynamic Vertical Scaling in Microsoft Fabric Python…
TL;DR : The high-end version of this problem is mostly solved. GPT-4 or Claude Opus paired with a mature proprietary semantic layer like Microsoft Power BI s will handle natural language queries reliably in production, This blog is a much narrower use case: how do small language models (4B parameters, 4GB VRAM) perform when paired with Continue reading "Text-to-SQL using Local LLMs"
Microsoft Fabric now has a proper CLI deploy, and it works. I built a fully automated CI/CD pipeline that deploys a Python notebook, Lakehouse, Semantic Model, and Data Pipeline to Fabric using nothing but the fab CLI and GitHub Actions. Here s what I learned along the way , what works great, what to watch out for, and Continue reading "Deploying to Microsoft Fabric with the Fabric CLI: First…
A complete data pipeline running on Microsoft Fabric that downloads public data, transforms it into a star schema, exports it as Delta Lake tables, and serves it through a Power BI semantic model with Direct Lake — all from a single Python notebook and using pure SQL all the code is available in github and Continue reading "Building a Data Pipeline Using VSCode and Claude Out of Thin Air"