Recent content on rmoff's random ramblings
Sheesh, 162 links this month. What is going ON? There is just too much interesting material being written, and that’s after filtering out clanker slop. I’m pretty sure I need to curtail the scope of these link posts at some point. I added the 🔥 emoji a while back to make it easier for folk to just skim for the top links. It’s been a quiet blogging month for me, with just one post capturing my…
Having clankers churn out content to feed the clankers is a futile game to play. It’s an arms-race to the bottom of a cesspit.
June has been a busy month—113 links below for your enjoyment and delectation. I’m going to share one extra link up here with you though, but it’s not my fault if it wrecks your productivity! My friend Kris Jenkins has written this devishly simple but addictive browser-based game: Escape the Moon .
Welcome to May’s Interesting Links ! This month saw the Current conference in London with the usual 5k run , lots of familiar faces and friendly conversations—and plenty of excellent breakout sessions too. It seems live-tweeting conferences isn’t a thing any more, with only myself and Thomas Cooper seeming to post anything, but if you want you can go review the hashtag feed on BlueSky for some…
Like a young child coming home from kindergarten with their latest crayon scrawls, the internet is currently awash with people sharing their AI-generated work. And just like the young child’s drawings, much of that work should be proudly put up on the walls within the artist’s house—and no further.
Another year, another Current—another 5k run/walk for anyone who’d like to join! Did I mostly copy-and-paste this from last year’s post ? You bet I did!
It’s now a joy to simply read any blog post that’s not AI-generated.
A bit of a streamlined edition, this month. Lots of interesting links still, but less commentary. You can put that down to me prevaricating on getting my previous blog about Materialized Tables in Apache Flink finished, and leaving myself little time to work on this one :) Not including the detailed narration actually knocks a bunch of time off the preparation—I’d be interested in your feedback as…
Flink added support for what it calls Materialized Tables in 1.20 , released in 2024. You can read about the design and motivations in FLIP-435 . In a nutshell, Materialized Tables provide a way to include the SQL to populate and refresh a table as part of its definition.
After cobbling together my own eval for Claude , I was interested to discover harbor . It’s described as: A framework for evaluating and optimizing agents and models in container environments. Which sounds kinda cool, right?
As a non-Java coder, for the last ten years I’ve stumbled my way through the JVM-centric world of "big data" (as it was called then), relying on my wits with SQL and config files to just about muddle through. One of the things that drew me to Kafka Connect was that I could build integrations between Kafka and other systems without needing to write Java, and the same again for ksqlDB and Flink…
I’ve had a huge amount of fun this month exploring quite what AI (in the form of Claude Code) can do for a data engineer. Rather than just hack around at a prompt, I took a bit more of a considered approach to it, building a harness to test out different prompts and skills. You can read my write-up here, the headline of which is that literally Claude Code isn’t going to replace data engineers…
I wanted to explore the extent to which Claude Code could build a data pipeline using dbt without iterative prompting. What difference did skills, models, and the prompt itself make? I’ve written in a separate post about what I found ( yes it’s good; no it’s not going to replace data engineers, yet ). In this post I’m going to show how I ran these tests (with Claude) and analysed the results…
tl;dr I use AI heavily on this blog. I don’t use AI to write any content. As any followers of my blog will have seen recently, I am a big fan of the productivity —and enjoyment—that AI can bring to one’s work. (In fact, I firmly believe that to opt out of using AI is a somewhat negative step to take in terms of one’s career.) Here’s how I don’t use AI, and never will :
Ten years late (but hopefully not a dollar short ) I recently figured out what all the fuss about dbt is about . Well that’s cute, Robin , you might be saying. Congratulations for catching up on what data/analytics engineers have been doing for years now. But you see, coding by hand is so 2025. Didn’t you hear? AI is going to replace data engineers. No it’s not (at least, not yet). In fact, used…
This is an addendum to the main post about using Claude Code with dbt . It shows an excerpt of a Claude session log so you can see exactly what goes on "under the covers" . For full details of the prompt, commentary, and conclusions, see Claude Code isn’t going to replace data engineers (yet) . Here we can see the steps that Claude Code takes as it figures out for itself anomalies in the data and…
Yes, you’re right AI slop is ruining the internet . Given half a chance AI will delete your inbox or worse (even if you work in Safety and Alignment at Meta): Nothing humbles you like telling your OpenClaw “confirm before acting” and watching it speedrun deleting your inbox. I couldn’t stop it from my phone. I had to RUN to my Mac mini like I was defusing a bomb. pic.twitter.com/XAxyRwPJ5R —…
Phew, what a month! February may be shorter but that’s not diminished the wealth of truly interesting posts I’ve found to share with you this month.
How do you use your LLM coding agent? Mine is usually Claude the proofreader, Claude the bash monkey, Claude the webdev. All these things are about tasks completed . Read this, write that code, fix that web page. This week I gave Claude a new job.
Apparently, you can teach an old dog new tricks. Last year I wrote a blog post about building a data processing pipeline using DuckDB to ingest weather sensor data from the UK’s Environment Agency . The pipeline was based around a set of SQL scripts, and whilst it used important data engineering practices like data modelling, it sidestepped the elephant in the room for code-based pipelines: dbt.
In which Claude and [A]I play at being webdevs. For some reflections on the bigger picture of AI as a productivity tool for developers, have a look at the companion post to this one . I used to speak at a lot of conferences and meetups, and published my talks on a site called noti.st . It’s free to use, but you could pay for bells and whistles including a custom domain, which I duly did:…
Funnily enough, Charles Dickens was talking about late 18th century Europe rather than the state of AI and LLMs in 2026, but here goes: It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of light, it was the season of darkness, it was the spring of hope, it was…
LLMs are rapidly changing how we use the internet. Remember just a few years ago when you’d search for something on Google and scroll through the results like some kind of Neanderthal? Heck, you might even click through to page 2 if you were feeling spicy. These days— and, knowing how this stuff ages, I should perhaps be less broad than "these days" and say just "in January 2026" —Google’s AI…
This is the twelfth edition of this newsletter in its current form. It’s great to see the audience for it growing, and consistently positive reception when I share it. Nice words always inspire me to carry on with it :D The substack edition (which is exactly the same content but sent out by email), is also picking up views and subscribers.
In late 2025 the company behind MinIO decided to abandon it to pursue other commercial interests. As well as upsetting a bunch of folk, it also put the cat amongst the pigeons of many software demos that relied on MinIO to emulate S3 storage locally, not to mention build pipelines that used it for validating S3 compatibility. In this blog post I’m going to look at some alternatives to MinIO.
"What are the must-have apps to install on my new Mac?"… "Which tool makes you the most productive?"… "Do you still use Alfred?"… All these questions and more—and the answer to all of them is Raycast!
Well it’s that time of year already! Whilst munching on a mince pie , enjoy the final Interesting Links for 2025. It’s been a busy twelve months for me; this time last year I was signing off from my last company, which went on to be acquired —and last week I found out that my current company (Confluent) is to be acquired by IBM . Despite my reaction against any kind of cheese moving , I figure…
Reddit is one of the longer-standing platforms on the internet, bringing together folk to discuss, rant, grumble, and troll others on all sorts of topics, from Kafka to data engineering to nerding out over really bright torches to grumbling about the state of the country —and a whole lot more. As a social network it’s a prime candidate for using graph analysis to examine how people interact—and in…
Welcome to the 10th edition of Interesting Links . I’ve got over a hundred links for you this month—all of them, IMHO, interesting :)
As part of compiling the monthly interesting links posts, I go through a ton of RSS feeds, sourced from specific blogs that I follow as well as general aggregators. These aggregators include quality sources like InfoQ, and certain tags on lobste.rs. Here I’ll often find some good articles that I missed in my general travels around the social media feeds in the previous month. I also, so you…
Ever tried to hammer a nail in with a potato? Nor me, but that’s what I’ve felt like I’ve been attempting to do when trying to really understand agents, as well as to come up with an example agent to build. As I wrote about previously , citing Simon Willison, an LLM agent runs tools in a loop to achieve a goal . Unlike building ETL/ELT pipelines, these were some new concepts that I was struggling…
At Current 2025 in New Orleans this year we built a demo for the Day 2 keynote that would automagically summarise what was happening in the room, as reported by members of the audience. Here’s how we did it!
The latest Thoughtworks TechRadar is out. Here are some of the more data-related ‘blips’ (as they’re called on the radar) that I noticed.
What with Current NOLA 2025 happening this week, and some very last minute preparations for the demo at the keynote on day 2, this month’s links roundup is pushing it right up to the wire :) The demo was pretty cool, and finally I have a good example of how this AI stuff actually fits into a workflow ;) I’ll write it up as a blog post (or two, probably)—stay tuned!
A presentation about effective blog writing for developers, covering why to blog, what to write about, and how to structure your content.
A short series of notes for myself as I learn more about the AI ecosystem as of Autumn [Fall] 2025. The driver for all this is understanding more about Apache Flink’s Flink Agents project, and Confluent’s Streaming Agents . I started off this series —somewhat randomly, with hindsight—looking at Model Context Protocol ( MCP ) . It’s a helper technology to make things easier to use and provide a…
Sneaking it in just before the end of the month! It’s a bumper set of links this month—I started with an original backlog of 125 links to get through. Some fell by the wayside, but plenty of others (78, to be precise) made the cut. With no further ado, let’s get cracking!
Having looked at MCP , Models , and RAG , I realised that I’ve been mentally skirting around something that I don’t really understand, so I’m going to expose myself to some ridicule here and try to understand better: what’s the difference between AI and ML? Aren’t they just the same?
A short series of notes for myself as I learn more about the AI ecosystem as of September 2025. The driver for all this is understanding more about Apache Flink’s Flink Agents project, and Confluent’s Streaming Agents . Having poked around MCP and Models , next up is RAG. RAG has been one of the buzzwords of the last couple of years, with any vendor worth its salt finding a way to crowbar it into…
A short series of notes for myself as I learn more about the AI ecosystem as of September 2025. The driver for all this is understanding more about Apache Flink’s Flink Agents project, and Confluent’s Streaming Agents . Having poked around MCP and got a broad idea of what it is, I want to next look at Models. What used to be as simple as " I used AI " actually boils down into several discrete…
A short series of notes for myself as I learn more about the AI ecosystem as of September 2025. The driver for all this is understanding more about Apache Flink’s Flink Agents project, and Confluent’s Streaming Agents . The first thing I want to understand better is MCP.
You can find previous editions of Interesting Links here .
You’ve got data in Apache Kafka . You want to get that data into Apache Iceberg . What’s the best way to do it? Perhaps invariably, the answer is: IT DEPENDS . But fear not: here is a guide to help you navigate your way to choosing the best solution for you 🫵.
This is a quick blog post to remind me how to connect Apache Flink to a Kafka topic on Confluent Cloud. You may wonder why you’d want to do this, given that Confluent Cloud for Apache Flink is a much easier way to run Flink SQL. But, for whatever reason, you’re here and you want to understand the necessary incantations to get this connectivity to work.
First up, allow me a shameless plug for my blog posts this month: Writing to Apache Iceberg on S3 using Kafka Connect with Glue catalog . Keeping your Data Lakehouse in Order: Table Maintenance in Apache Iceberg . 🔥 Building Streaming Data Pipelines, Part 2: Data Processing and Enrichment with Flink SQL (see also Part 1 )
Iceberg nicely decouples storage from ingest and query (yay!). When we say "decouples" it’s a fancy way of saying "doesn’t do". Which, in the case of ingest and query, is really powerful. It means that we can store data in an open format, populated by one or more tools, and queried by the same, or other tools. Iceberg gets to be very opinionated and optimised around what it was built for (storing…
Without wanting to mix my temperature metaphors, Iceberg is the new hawtness, and getting data into it from other places is a common task. I wrote previously about using Flink SQL to do this , and today I’m going to look at doing the same using Kafka Connect. Kafka Connect can send data to Iceberg from any Kafka topic. The source Kafka topic(s) can be populated by a Kafka Connect source connector…
In this blog post I’ll show how you can use Flink SQL to write to Iceberg on S3, storing metadata about the Iceberg tables in the AWS Glue Data Catalog . First off, I’ll walk through the dependencies and a simple smoke-test, and then put it into practice using it to write data from a Kafka topic to Iceberg.
After a week’s holiday ("vacation", for y’all in the US) without a glance at anything work-related, what joy to return and find that the DuckDB folk have been busy, not only with the recent 1.3.0 DuckDB release , but also a brand new project called DuckLake . Here are my brief notes on DuckLake.