Recent content on Andrew Jones
Hey, happy Friday! In this week’s newsletter I write about what ownership is (it’s complicated!). There’s also links to articles on closing the context gap, getting your data team a seat at the table, and creating data & AI use cases. What is ownership? Ownership is a complicated thing. Often people are assigned ownership, for example by having their name listed as such in a…
Hello 👋 This week I write about how, when our data output needs to be reliable, so does our input. There’s also links to posts on using data contracts to build a shared language, federated data architecture with DuckDB, and Bitol (ODCS) becoming a graduate project at LF AI & Data. Reliable output needs a reliable upstream Years ago at GoCardless, we started feeding our data into ML models…
Hello 👋 In this week’s newsletter I write about limiting what you have in your data contract and only including fields that drive action, typically through automation. There’s also links to articles on how a single source of truth initiative should focus on governance, inverting data architectures by federating data, and shifting quality left within a data warehouse. In this…
Hey, hope your week’s been good! In this week’s newsletter I write about how infrastructure/configuration as code is a prerequisite for automation, whether you are automating with platform tooling or with agents. There’s also links to articles on improving agentic analytics, building an Agentic Development Platform, and Apache Ossie. IaC is a prerequisite for automation One of…
Hello, hope you’ve had a good week! This week we had a hackathon and I built a data support agent 🤖 It was fun! I’ll probably write about what I learned soon. On to the newsletter, and today is the fourth and final post in my series of data contracts anti-patterns, and is about change management for changes that impact the meaning of data, but not the structure. There’s also…
Hello 👋 Welcome to the third part in my series of data contract anti-patterns, following on from: Contracts as documentation Checkbox compliance This one is all about defining SLOs that actually make a difference. There’s also links to articles on semantic vs context layers, building a semantic layer at Lyft, and data contracts in KCL. Data contracts anti-pattern #3: Vanity SLOs…
Hello! This is the second in my series of data contracts anti-patterns, following on from last week’s post on contracts as documentation . Again, this one looks like progress, but in reality has no impact. There are also links to articles on data platform incident management and encoding your data expert. Finally I had a fun discussion on What every Beginner Should Know About Data Governance…
Hello 👋 Hope you’ve had a good week! I’ve seen a lot of people succeed with data contracts, but I’ve also seen a lot of people struggle. So, over the next four weeks I’m going to write up 4 common anti-patterns I’ve seen. Interestingly, each of them looks like progress, but they fail to solve the underlying problems. I’ll kick this series off with the most…
Hello 👋 I’m writing to you from Data Mesh Live at Antwerp, where I’ve given a full day workshop and a well received talk on implementing a self-service data platform. Besides that I have lots to digest from some excellent talks and made many new great connections! This week’s newsletter comes from that talk and is about scaling data governance by moving away from documents and…
Hello 👋 In this week’s newsletter I talk about the importance of meeting your users where they are. There’s also links to articles on a meshy approach to building data models, SQL patterns for catching fraud, and the case for data marketplaces. Meet your users where they are Around 8 years ago, before we built a self-serve data platform around the concept of data contracts, we built…
Hello again 👋 In this week’s newsletter I write about how we can’t just blame data producers, we need to look at ourselves and the support we give them. There’s also links to context, finally achieving self-serve analytics, and synthetic customers. Enjoy! You can’t just blame data producers for poor data quality As data teams we know what poor quality data looks like. We…
Hey, hope you’re well :) Big week for me personally as I started my new role at LocalStack , where I’ll be supporting our data team as we define and scale our data platform architecture. On to the newsletter, and this week I write about how to make your golden paths actually golden. There’s also links to articles on self-serve data, north star metrics for AI data products, and…
Following on from my trick to change my terminal colour based on the project (git repo) I’m in, Simon Späti shared that he does the same for his editor, Neovim. Which got me thinking, could I do the same for my editor, VS Code? Turns out I could by writing a small VS Code extension, with AI assistance. This only changes the colour of the title bar, rather than the entire background like my…
Hello 👋 In this week’s newsletter I write about the simple concept of data contracts, and its power. There’s also links to articles on the 7-table fallacy and measurement engineering. Data contracts are a simple concept In a virtual book signing earlier this week at ODSC East , I was asked: For teams new to the concept, how would you explain data contracts in simple terms? My answer:…
Hello 👋 Today I write about how the quality of a data product is not just the data, it’s the code behind it too. There’s also links to articles on building an LLM platform, speed when decentralising, and building decision systems. The quality of a data product is its data and its code Often when we think about the quality of a data product, we think only about the data quality, such…
A service catalog is an increasingly common component of internal developer platforms. They primarily serve two personas/use cases. The first is a system owner/engineer, who are responsible for services and/or the code behind them. The service catalog gives them one place to go to find out everything about their service or any other service they might be interested in, e.g. to integrate with or to…
Here’s a small but nice improvement I’ve made to my terminal recently that has been effective for me. I generally have many terminal windows open across many git-based projects, which I’m regularly switching between. Now, my terminal windows have different colours for each project. This is what it looks like: This is the code I’ve added to my zshrc , written with AI…
Hello 👋 This week I write about finding the right level of abstraction. There’s also links to articles on the outbox pattern at scale, first 90 days as a CDO, and measuring latency. The right level of abstraction When building a (data) platform you end up thinking a lot about the abstractions you are providing, and the trade-offs they cause. On one hand, you want to abstract away some…
Hey 👋 This week I introduce the contract-driven data platform, and link to an in-depth post I wrote for Modern Data 101. There’s also links to articles on the AI wave, the relevance of business models to data, and a self-improving Text-to-SQL engine. If you have some time off over Easter I hope you have a great time 🐰🥚 The Contract-driven Data Platform Traditional data platforms are a mix…
Hello again 👋 This week I write about the data reliability question you’re avoiding, and whether you are making the right trade-offs. There’s also links to articles on relating rigour, how your pipeline might succeed but your data did not, and how decisions are really made. Finally, last call for the early bird pricing for my in-person Implementing a Data Mesh with Data Contracts…
Hey! Hope you had a good week :) Today I write about what it actually means to take responsibility for data. There’s also links to articles on DucbDB transpilation to reduce warehouse costs, ontologies on Snowflake, and observability becoming a bottleneck. Finally a reminder: Early bird pricing on my only in-person data contracts course this year ends at the end of the month. Do join me in…
Hello 👋 This week I propose some common data assumptions that I think are worth challenging. There’s also links to articles on data platforms not being a destination, ORKs for data platform prioritisation, and query federation. Also, early bird pricing on my only in-person course this year ends at the end of the month. Do join me in Antwerp! . 3 common data assumptions worth challenging I…
Hey friends 👋 Today I write about the bystander effect that happens when ownership and responsibility is unclear. There’s also links to a data contract-driven data mesh, why data engineers should care about knowledge graphs, and support for graphs in BigQuery. The bystander effect on data ownership and responsibility The bystander effect is a social phenomenon where individuals are less…
Hey! Hope you’re well :) Today I write about what ‘contract’ really means in data contracts (hint: it’s interfaces!). There’s also links to articles on ontologies, why data pipelines exist, and how data quality is always an ownership problem. Also, early bird pricing for Data Mesh Live , where I am speaking and also running my only workshop this year , ends on 28th…
Hello again! This week I write about how the intentional application of friction in the data platform can guide user behaviour. There’s also links to a correct-by-design lakehouse, a state of data engineering survey, and anomaly detection at Doordash. Also a quick reminder, my only in-person workshop this year is happening in June in Belgium. Do join us! The case for intentional friction in…
Hello 👋 This week I write about what happens when a data contract is “breached”, since that is a common question I get asked (although data contracts are not only about enforcement!). There’s also links to articles on the next data bottleneck, barefoot data platforms, and Clickhouse + Postgres. Finally, a spicy pun. What happens when a data contract is breached? Data contracts…
Hey, hope you had a good week! In this week’s newsletter I write about the impact of AI-assisted code on your data platform, and how it moves the bottleneck. There’s also links to articles on how most teams are not building governance, going from data engineering to knowledge engineering, and the case for agent contracts. And finally, a colourful pun. Enjoy! The impact of AI-assisted…
Hello 👋 This week I’m writing/thinking out loud about integration vs interoperability, and whether we really need to centralise everything in a data warehouse before we can make use of data. There’s also links to articles on the foundation for context graphs, operationalising data science, and small data. Integration vs Interoperability In any organisation of size there will be many…
Hello 👋 hope you had good week! Once again I’ll be running my in-person workshop, Implementing a Data Mesh with Data Contracts , in June in Belgium alongside Data Mesh Live . It will likely be the only in-person workshop I run this year. Do join us if you can! Hit reply if you have any questions :) Now, on to the newsletter, and today I’m writing about reversing Conway’s law to…
Happy new year! If you had a break I hope it was a good one :) Today I’m writing about features of architecture, and the impact they have on processes and ways of working. Also links to articles on orchestration success in decentralised architectures, data modelling for private markets, and unifying batch and streaming. A feature of architecture The process we have and the way we work…
Hello for the last time in 2025! Today I’m sharing 5 examples of data contracts in practice. Also links to articles on the latest release of ODCS, the unavoidable movement of data, and being a Staff+ engineer in infra/platform teams. Thanks for reading this year, and I’ll see you in the next one! 5 data contract implementations in the wild Because many companies make it difficult to…
Hey friends 👋 This week I write about reducing the difficulty of being a data owner. Also links to articles on data contracts at VMO2, metadata as common language, and BlaBlaCars data copilot. Finally, a snowman pun. Enjoy! Reducing the difficulty of being a data owner It’s often difficult to find someone who wants to own data. That’s because it’s difficult to be a data owner.…
Hey, hope you’ve had a good week :) Today I’m writing about pushing on open doors when getting started with data contracts. Also links to articles on the ownership of data quality, the basics of data architecture, and OTel with lakehouses. Also, on Monday I’ll be live on LinkedIn / Substack / YouTube talking about Driving Data Quality with Data Contracts: What Data & AI PMs…
Hey friends 👋 This week I write about how we can use the iPhone model to create integrated developer, data, and other platform features, and the change in internal developer platforms that need to make this happen. Also links to articles on data as code, data platforms for data scientists, and stream and batch analytics with Iceberg. The iPhone model for integrated platforms One of the problems…
Hello 👋 This week I write about how taking control makes you a bottleneck. Also links to articles on a decade of AI platform, context layers, and a scope creep game. Finally, if you’re in London next week join me as I present at the London Platform User Group (LOPUG) meetup on contract-driven data platforms. I’m particularly looking forward to this talk as it’s for platform…
Happy Halloween, if that’s your thing! Today I write about the internal cost of data quality issues - the costs to your team. Also links to articles on why you’ll never have a FAANG data infrastructure, handling multiple data platforms, and the dbt blues. The internal cost of data quality issues Data teams often spend a lot of time articulating the cost of poor data quality to the…
Happy Friday! In this weeks newsletter I write about keeping data contracts for your users simple even as you add more capabilities. Also links to articles on implementing data contacts, data platforms as a product, and watermarks when streaming. Keeping data contracts simple Data contracts can power a number of platform capabilities, including change management, data quality checks, data…
Hello 👋 This week I write about source-aligned data products and whether they really solve the problems we had, or if instead we need a more suitable output port on the source system. Also links to articles on scaling data platform teams, the automation paradox in data governance, and the minimalist data contract. Source-aligned data products, or output ports on the source? When it comes to…
Hey, hope you’ve had a great week. Today I write about creating a business case for improving data quality. There’s also links to articles on ODPS hitting 1.0, a modern data platform with a focus on data products, and the data platform at Decathlon. A business case for data quality Many data engineering teams spend a lot of their time struggling to deal with upstream data. That…
Happy Friday! Thanks to everyone who has signed up for my new course ! I’m so humbled by the response to it so far :). Today’s post is also from the course and describes why data contracts are becoming so important. The 25% off code , W1NX99YYT5 , is valid until midnight tonight UTC, so grab it now if you haven’t already! There’s also links to articles comparing data…
Hey 👋 It’s big week for me, as I’m launching my Implementing Data Contracts course! More on that below, including a special discount code as a thank you for being on my newsletter :) There’s also links to articles on data architectures of the future, dbt at scale, and the power of small wins for data ROI. Implementing Data Contracts course now live! I’ve been running my…
Hello! I’ve been working hard on an online, self-paced version of my Implementing Data Contracts course, and I can’t wait to share it with you! It should be ready in the next week or so, and I’ll send you an email when it is. For now, onto the newsletter, and this week’s post comes from part of the course and describes 3 steps to getting started with data contracts. There…
Hey 👋 This week I write about how CI is still too late in the SDLC to prevent major data model changes. There’s also links to articles on IaC, what’s still true in data engineering, and Nike’s changes to their streaming architecture. CI is too late in the SDLC to identify data changes I once worked on a data team that attempted to review PRs from the software engineering teams…
Hey friends 👋 Today we have the final part of my mini-series on data publishing patterns, covering the listen to yourself pattern. There’s also links to skipping the DWH, caching, and TimescaleDB. The listen to yourself pattern for publishing events As we’ve been discussing, with data contracts we want to move to a model where data and events are published to consumers, rather than…
Hey, hope you’ve had a good week! Today’s post is the second part of a mini-series on publishing events, and today I explain the outbox pattern. There’s also links to articles on silos, integrations, and building reliable log delivery. The outbox pattern for publishing events As mentioned last week , with data contracts we want to move to a model where data and events are…
Happy Friday! This week is the start of a mini-series on publishing events. Also links on unlearning data architecture, a simple data governance framework, and AI for data engineers. Finally, on Wednesday (20th August) I’ll be live on Loosely Coupled comparing Data Mesh and Application Integration with Karol Skrzymowski and Rachel Barton . Should be fun - join us on LinkedIn or YouTube !…
Hello 👋 Today I write about how our software engineers used data lineage. There’s also links to articles on comparing data on deploy, alignment in the age of autonomy, and reducing warehouse costs. How our software engineers used data lineage Data lineage is the process of tracking the movement and usage of data through an organisation. Often this tracking information is populated by…
Singer is an open-source framework from Stitch that separates the reading from a source (a tap ) and the writing to a data store (a target ) by having them communicate with each other using a simple JSON-based protocol. While a bit dated and without much love from Stitch over the years, I still think it’s a good framework that allows a data engineer to focus on either the reading or the writing of…
Hey, welcome to August! In today’s newsletter a short reminder on giving data producers the why when you’re asking them for better quality data. Also links on being a business first data leader, thinking beyond the warehouse to build AI agents, and extending lineage to the operational plane. Data producers need to know why It’s all very well saying that data producers need to do…
Hello! First, some personal news : After nearly 8 years I’m leaving GoCardless. It’s been a great journey, building a data platform from scratch at a scale-up, but it’s time for something new. I’m still exploring what that will be and will take some time to consider that over the summer. So, if you need a leader or engineer for your data platform/engineering teams, let me…