Recent content on Pablo Lopez - Software Engineer & AI Contractor | Edinburgh, UK
I load tested a Databricks SQL Warehouse as the backend for a high concurrency API, to find out whether you can skip the operational database. Here is what the numbers showed, and the more useful lesson underneath them.
This is the story of how I overengineered a software project because I made the wrong assumptions too early. This was originally going to be a success story of how I managed to rescue my friend’s website from his web developer who was keeping it hostage. I was going to write about how I used AI to rewrite a whole website in a day, despite me not being a web developer. In the end, I realised…
Over the last 3 years I’ve spent a lot of time building highly scalable Data APIs, to enable teams to access data from different sources with minimal effort. There have been several mistakes made along the way, and many lessons learned. I believe these lessons will form strong foundations to build performant Data APIs from day one. Before diving into the lessons, I’ll define Data API.…
Reflecting back to the time I was deciding whether to start my own blog site has made me remember that it wasn’t an easy decisions. Now I’ve been writting blogs for several years, and I rarely care about whether I’m an expert or a novice in the subject that I’m writing about. But this wasn’t always the case. I used to be really worried that I would write about…
Over the past few months I’ve been using AI agents a lot to automate many engineering tasks that otherwise would take me much longer to do. I’m a firm believer that the era of writing code by hand has finished. With the right workflows these tools can do a lot of the work for us. AI is improving our productivity, and it will continue more and more as the tools mature.
Step 1 — Target Audience The Reader A junior, mid, or senior data engineer who builds data pipelines in the warehouse (DBT, Databricks, Snowflake) and orchestrates them with Airflow. Skills and knowledge: Git, Python, SQL, PySpark, Airflow. Strong business logic knowledge — they understand the data they work with, not just the technical layer. They are not necesarily very strong software…
Module 9: Ownership, security and new ways of working Closing section. Reflection and discussion. No hands-on needed. Leaves attendees with the right mindset to take this back to their team. Cognitive debt of using agents (TODO: write this up) Security is part of ownership — what agents get wrong, what you are responsible for Scope your environments properly — credentials, prod vs dev, worst-case…
Resources There Is an AI Code Review Bubble - Honest take on why most AI code review tools are fundamentally limited. Shallow diff-only reviews produce noise, not signal. ( HN discussion, 351 points ) How We Made Our AI Code Review Bot Stop Leaving Nitpicky Comments - Practical deep-dive into the #1 problem: LLMs leave too many low-value comments. Details what worked and what didn’t to…
Resources Claude Code: Agent Teams - Official documentation on running multi-agent teams with Claude Code. Claude Code: Best practices for agentic coding - Includes guidance on running headless agents, parallelizing work, and using sub-agents. ( HN discussion, 614 points ) Don’t Build Multi-Agents - Cognition (Devin) argues against naive multi-agent architectures. Specialized sub-agents…
Introduction At this point we have already seen how to use spec driven development to create a structured engineering workflow and using feedback loops te ensure the code is validated and meeting the necessary standards. We already have a very solid workflow we can go and use. Engineering is more often than not, not done on your own. You usually work in a team where everyone is trying to achieve…
Introduction Once you get to the implementation phase, you can simply ask your agent to implement the task and end it there. The agent will produce the code based on the specifications. If that’s all you do, then you still have the job of going through the validation steps required to ensure the code is ready for production, such as running unit tests, locally testing functionality to check…
Module 6: CLIs, MCPs and debugging Hands-on: attendees connect a CLI or MCP to their agent and use it in a spec or debug session. What is a CLI tool? — terminal access to external systems, examples What is an MCP? — structured tool access, how it differs from a CLI, JSON config example When to use them — three scenarios: spec validation, triggering external systems, debugging MCP setup walkthrough…
Module 3: Spec driven development First hands-on moment. Attendees write a spec for a real or dummy feature by the end of this module. Why spec-first? — what changes when you plan before implementing The planning phase overview — spec → plan → tasks as a pipeline, not a negotiation The spec.md file — purpose, required sections, what good looks like vs what bad looks like The annotation loop — how…
Introduction — what you will have set up by the end of this chapter IDE agents vs CLI agents — the two flavours, narrow scope to CLI for this chapter Understanding the workspace — mental model of what the agent can see, do, and access before you install anything Installing an agent — now the reader knows what they are installing and why Running your first prompt — first real interaction, what to…
Module 1: How we got here Sets the stage. No hands-on. By the end, attendees understand why this is happening now and what an agent actually is. Why this course? ROI framing — what changes for developers, teams, and organisations The copy-paste problem — where most developers are today What is an LLM? What is an agent? — very short, no-jargon explanation; just enough to understand the rest of the…
Coding agents work best when they have a good feedback loop they can use to verify their changes. Without a good local environment and feedback loop, agentic development quickly becomes a waste of time. My first experience with software development was javascript, building websites using frameworks like React. I loved my development setup back then. I would always have a local server,…
Pablo Lopez Santori Edinburgh, UK | pablo.lopez.santori@gmail.com | github.com/wanderdust | wanderdust.github.io/newsletter Summary AI and Machine Learning Engineer with 6+ years of experience building production AI systems, from LLM-powered agents and RAG pipelines through to model training, MLOps, and ML feature platforms. Strong background in AWS, Bedrock, and Kubernetes, with hands-on…
If you were to peek inside Postgres and see how the tables are stored, you’d be surprised they look nothing like the outuput you get when you do a SELECT query. In this post I look inside Postgres to understand how tables are stored in the filesystem. The goal of this post is nothing more than to understand a small part of the whole Postgres ecosystem to demystify things a bit. It is a step…
This has been one of the years where I’ve learned the most things, probably since I first started learning programming almost 10 years ago. I thought it would be a great exercise to put it into writing to reflect on it all. The Dvorak keyboard layout The first thing I set out to learn in 2025 was how to type using the Dvorak keyboard layout. The main reason was simply because it sounded like…
The hardware Compute nodes (thinkpads) NAS (Optional) Setting up the nodes 2.1) Installing Ubuntu Server 2.2) Setting up a static IP on the nodes See which is your ethernet name by looking at ip addr. It is the one that starts with enp0… Use that as the ethernet name in the netplan/50-cloud init file. Use neplan apply…
I have passed the AWS Architect Professional exam with a score of 791/1000! Preparing for the AWS exam I have recently completed the AWS Architect Associate Exam, and I thought the Professional would be a nice next step considering all the prep work I had done. I took about three months between passing the Associate and taking the Professional exam. For the Associate exam I spent a lot of hours…
In this post I wanted to describe the different options if you are considering a data role. All of the information in this blog post is based on my own work experience working in different data roles over the last five years in Scotland. The goal is to help you get a better idea of the data landscape so you can make more informed decisions coming out of university and getting a job.
Today I wanted to write about something that happens to me at work all the time: the process of asking for technical help, helps me discover the answer to my question. For example, today I was stuck on some terraform config that wasn’t showing some of the changes I was expecting to see in the diff. I decided to start writing a message in a support channel. As I was writing the message I…
Typical Academic Roles (Career Progression) Role Approx. Salary Range Notes Teaching Assistant 30s Entry-level teaching support Teaching Fellow (Junior Lecturer) 40s Intermediate level Lecturer 50s Senior teaching or research position University & Application Notes Universities: Edi: Edinbugrh Uni, Napier, Heriot Watt. Fife: Dundee? St Andrews? College Edi: Edinburgh College Fife: ? Experience…
Slowly changing dimensions, what a confusing name. This is the sort of term I’ve been hearing for a while now, but never really bothered to properly look up what it actually means. As it turns out, all it means is how you handle row updates/deletes in your data pipelines. Whether you decide to maintain historical data in your tables by appending new rows when a change happens, or you decide…
Interviewing for other jobs in tech can make you a more competent engineer. My premise for this argument is that when you prepare an interview for a new role you need to prepare to answer questions, technical and non-technical, in a domain that is different from your day to day work. This forces you to step out of your daily work and read and think about different domains and technologies. During…
When working with distributed systems you can encounter terminology such as “replicas” or “standby” intstances. When I first started learning about distrubuted systems, I assumed “replicas” and “standby” were the same thing, but called differently when talking about scaling or high availability. While they can be the same thing, that is not always…
Practical strategies to protect your serverless applications from runaway AWS bills, including caching, rate limiting, WAF rules, and DynamoDB safeguards.
GOAL: Highlight the differences between these strategies. When to use one or the other. When they can be used together Audience: Software engineers who don’t know so much about distrubuted systems. Challenge; write it as a one off - as if you were in a typewriter.
After ignoring the whole “agent” hype for a while, I started to get that inevitable feeling of being left behind. That changed when I came across this blog post explaining the basic flow of an agent. It made me want to try it for myself and see what it actually makes an agent. This post is my walkthrough of that process: step by step, showing how to turn an LLM into an agent. The code here is…
I recently passed the AWS Solutions Architect Associate exam with a score of 848. This certification covers a wide range of AWS services which has taken a good amount of work and effort to prepare. In this blog post I wanted to share my experience preparing for the exam, sharing some info about the exam, and what has worked for me. Preparation I have about five years of on-and-off AWS experience,…
Introduction This is a guide to myself if I need to deploy an Appsync App in the future. Appsync is a Serverless AWS offering for implementing GraphQL APIs. It integrates nicely with Aurora and DynamoDB. Other integrations are done via Lambdas. Architecture Overview Compontents AWS AppSync : The managed GraphQL service that processes API requests Aurora PostgreSQL : Serverless relational database…
I recently had to run load testing for an API that fetches data from Postgres. I was monitoring Postgres when running the first test, and I noticed that hundreds of connections were being opened. I was also seeing runtime errors in Postgres such as “no more connections allowed” or “out of shared memory”. Clearly the database was not prepared for the load. So I started to…
In this post, guest writer Billy interviews Pablo on learning Dvorak. Pablo started to learn Dvorak in January 2025. This is an interview about his experience. First of all, can you explain what Dvorak is? Dvorak is a keyboard layout created in the 1930s, proposed as a more efficient alternative to Qwerty. Dvorak groups letters on the keyboard based on how often they are used (in the English…
Replication refers to having copies of your data across different instances. It gives you redundancy when one of your instances goes down so you can still serve data to users from the remaining instances. Another benefit of a replicated system is being able to load balance your requests across different instances. This can be useful when the load of serving all the requests from a single instance…
Fiction Orbital This one tells the story of a team of 5 austronauts in their space mission. It is less than a 100 pages long, but it is one of those books that requires your full attention, where every sentence and every word counts. It is beautifully written. Stormlight Archives 5: Wind and Truth Brandon Sanderson never dissapoints. Like all of Stormlight Archives, the longer the book the better…
I currently have a basic setup in NeoVim: I use NvChad by default with treesitter to get pretty text highlighting, telescope to navigate files, conform for all my formatting needs and some python utilities such as pyright. This is enough to get me by. The thing that has been annoying me the most lately is the terminal workflow in NeoVim. To start with, the terminal does not save any history across…
In this post, I share how to build a Reverse ETL pipeline to upsert data from Delta (Databricks) into Postgres to provide sub-second response times to our tables. The goal is to make warehouse data available to downstream systems that require millisecond response times. These systems could be front-end applications that need to consume this data, or online machine learning models that require…
The other day my team came across a peculiar situation with our Postgres database. We had this table partitioned by a primary key, which we’ll call order_id. We realised one of the queries was taking a very long time to execute, and like good SQL debuggers we looked at the query plan. Let’s pretend this was the slow query: EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 12345 ; Upon…
I recently needed to run load tests on a PostgreSQL database. Postgres is no question a reliable and scalable database ready for production use cases, but there might be times when you’ll need to confirm it can handle your specific use case. This might be necessary if you’re using a PostgreSQL database hosted by a third party or if you want to check if your current instance size and specs can…
We all know that person that always asks for unit tests on your Pull Request. You curse under your breath wondering why they have nothing better to do than ruining your day. Unit tests let you know if your codebase is behaving as expected. As long as you’ve written good tests, you can be happy your codebase is healthy if tests are showing green . Any time a test appears in red , you will know that…
Indexes are like going to the airport and looking at the signs to see where your gate is located. The signs tell you how to get to your gate quickly. The alternative to not having signs is walking the whole airport randomly until you come across your gate. So by having the signs you can reach your gate a lot faster. Makes sense right? This is the idea behind indexes.
Learning To prepare for the exam I took the Udemy Kubernetes Certified Application Developer (CKAD) with Tests which I could access for free through my company. The course covers all the topics required for the exam with labs to practice each section which I found super helpful. The labs are hosted on KodeKloud and they are free to access as part of the course. I had some previous experience…
This is my story of how I went from non-tech into becoming a Machine Learning Engineer. I must have told this story multiple times in many different forms, but this will be the first time it goes into writing. Enjoy. I have a bachelor’s degree in Commerce. I found the last few years of school so exhausting that the thought of doing an engineering degree put me off. Four + years of no life? Oft, no…