Recent content on Memo Garcia
For a while I’ve been thinking about an app to keep track of my health properly, where I’m in control, fully local LLMs (with the option to use remote LLMs if desired). I’m getting old and I’ve been going more and more to multiple doctors and clinics, and I wanted a place to centralize and visualize my health in an easy way. It’s open source:…
How many tokens does “building a SaaS project” cost and how many of those are non-requirements? The parts you need but are not core to the product. I have asked my trusted codex to build SaaS foundations several times already. Auth, tenant isolation, billing, transactional email, database migrations, structured logging, feature flags and so on. Every new project starts with the same…
In a 900 person organization, we still could not answer a simple release question: what exactly are we shipping to the customer? We had one binary to release. My manager thought all features should go out. His manager had been told by the director that only X, Y, and Z were part of the customer commitment. The manager of an adjacent team wanted X, Y, and A, but only for some customers.
You used to have a single place to look at your build and your deploy, now you have two systems. GitHub Actions doing both CI and CD is simple. Push code, the workflow builds it, the workflow deploys it. One pipeline, one log, one place to check. The deploy step is right there after the build step. You can see the whole thing. This simplicity is a trap. I wrote about this before.
Why does my build work on my machine but not on yours? If you’ve asked this more than once, the answer is almost always: your build system is lying to you. It’s pulling in implicit state from the environment, from cached artifacts, from the order things happened to run last Tuesday. Make and most build tools are fine until they aren’t, and you only find out they aren’t…
Does this email already exist? How you answer depends on scale. 100 users Anything works. SELECT EXISTS (SELECT 1 FROM users WHERE email = 'me@memo.mx'); No index? Doesn’t matter. The database scans 100 rows in microseconds. Add a UNIQUE constraint and move on. 10,000 users Still comfortable, but add a unique index: CREATE UNIQUE INDEX idx_users_email ON users (email); Most products treat…
The severity of an incident is not determined by the technology that failed, but by the number of people impacted by it. A 3-person team can run an incident in a Slack thread. A 300-person organization cannot. That gap is where SRE gets interesting. What changes At small scale, the person who gets paged probably wrote the code. At large scale, the person who gets paged might not know which team…
In the previous section, we manually configured Istio to split traffic between two versions. While powerful, doing this by hand for every release is a fast track to burnout. We want an automated controller that says: “Deploy the new version, send 10% of traffic to it, wait a minute, check Prometheus to see if the error rate spiked. If it looks good, increase to 50%. If it fails, instantly…
We’ve spent a lot of time building a resilient, multi-region architecture. We have multiple replicas, automated rollbacks, and cross-cluster routing. But architecture diagrams lie. The only way to know if your system is actually resilient is to break it on purpose. This is Chaos Engineering. We are going to intentionally inject failures into our clusters to verify that our safety mechanisms…
We’ve reached the final stage of our multi-region deployment lab. We have GitOps, service meshes, dynamic IAM, progressive delivery, and chaos testing. But none of that matters if the application falls over the moment it receives real user traffic. To prove our architecture is ready for production, we need to test it under load and measure our latency against our Service Level Objectives…
Continuous Deployment is great until you automatically deploy a critical bug to every region simultaneously. If our Argo CD pipeline instantly updates 100% of our pods to a broken version, our global availability drops to zero. To prevent this, we need to decouple deployment (starting the new pods) from release (sending user traffic to those pods). Since we already have Istio installed for our…
We have a GitOps pipeline waiting to deploy our app, but how does the new image actually get built and telling Argo CD to deploy it? The classic anti-pattern is having a developer build a Docker image on their laptop, push it with the latest tag, and then run kubectl rollout restart. This breaks the GitOps contract and makes debugging impossible because latest means something different every day.…
We have our GitOps pipeline, our cross-cluster mesh, and secure IAM roles. Now, we actually need something to deploy. When deploying applications across multiple regions, you almost always run into the configuration drift problem. Ninety percent of your deployment manifest is identical (the container image, the ports, the health checks), but ten percent needs to be highly specific to the region…
When your applications run in Kubernetes, they eventually need to talk to cloud services - reading from an S3 bucket, publishing to an SNS topic, or querying DynamoDB. The absolute worst way to do this is by hardcoding AWS Access Keys into your application’s environment variables or secrets. It’s a massive security vulnerability waiting to happen. The slightly less terrible way is…
Now that we can deploy to multiple clusters, we need them to talk to each other. If a service in the US needs to reach a service in the EU, how does that traffic route? How do we secure it? We could expose everything to the public internet via Ingress, but that’s a massive security risk for internal API traffic. Instead, we’ll use Istio to build a multi-primary, multi-network service…
If you’re managing multiple clusters, applying manifests by hand with kubectl apply quickly becomes a nightmare. You lose track of what is deployed where, configuration drift sets in, and rolling back a bad change turns into an archeological dig through your terminal history. The solution is GitOps. We want our GitHub repository to be the absolute single source of truth. If a manifest…
Building systems across multiple regions is notoriously difficult. You have to navigate latency, configuration drift, and complex networking topologies. Before we dive into the fun stuff - like GitOps with Argo CD or cross-cluster routing with Istio - we need to establish a rock-solid foundation. In this lab series, we’re going to build a realistic multi-region deployment from the ground up.…
Train teams on multi region, high throughput, chaos tested GitOps without burning money. Assumptions/Constraints Money note: We mock “multi region” with two Kubernetes clusters, not two AWS regions. You have two kube contexts reachable from your workstation. You control a container registry and a GitHub repo. Optional AWS: use IRSA or EKS Pod Identity in Lab 4. Labs Part 1 - Baseline Repo and…
I’m currently building mooomooo, driven by my personal frustration with disconnected and manual work that I have to do in JIRA. My vision is to unify software lifecycle management, daily operations, and issue tracking into a single cohesive system that genuinely reflects the current state of an organization. By doing so, mooomooo aims to eliminate tedious manual synchronization and keep…
Problem You need safe, predictable traffic control in prod. VirtualServices + DestinationRules give version routing, timeouts/retries, and circuit breaking without touching code. Approach Define versioned subsets in a DestinationRule. Route via VirtualService with weights (canary → rollout). Add timeout, retries, and outlier detection. Shift weights, watch metrics, then cut over. Example (Config)…
Leadership Sync Positive Affirmations for Site Reliability Engineers Senior Engineer The Hustle MAKRO | Microsoft Excel Stream Highlights 3/19 Designer’s Table
Why do I need to build a docker container on every commit? Why do I need to download shit load of data every time my CI runs?
Implementing authentication from scratch often starts simple but quickly escalates into weeks of navigating RFC specifications and protocol implementations. Keycloak addresses this complexity by providing a comprehensive identity management solution. What is Keycloak? Keycloak is an open-source identity and access management solution that handles authentication and authorization at enterprise…
The reason why newcomers try to do everything at once or add everything they can is to prove they can do anything with the new knowledge they acquire. Veterans tend to make things simpler.
Understanding how HTTP requests traverse an Istio service mesh is fundamental to effectively operating and troubleshooting microservices architectures. This technical analysis examines the complete request lifecycle, from external ingress through service-to-service communication, detailing the mechanisms that enable Istio’s traffic management, security, and observability capabilities. The…
Just say no.
Have you ever asked an LLM, “Build me a CRUD API in FastAPI,” and just hoped for the best? LLMs are great at taking broad requests and letting us talk to computers like they’re people. But that freeform style can bring surprises and make it tough to get the same result twice. Behavior-Driven Development, or BDD, offers a clear framework. It guides you to: Define exactly what you expect…
3 ideas that help me be a bit more happier every day: The journey is where the fun is, not at the destination Allow yourself to be happy Find a purpose in life
Think of a network as a building full of rooms, doors, and room numbers. A packet starts to make sense when you can picture where it is trying to go. License: CC BY-NC-ND 4.0 Part One: Buildings as Networks Chapter 1: The Room, the Door, and the Room Number The building Imagine a building full of rooms, each with a purpose. Some are for sleeping, some for storing utilities, some for parties.
What is a system? A set of pieces that work together to perform a function. When those pieces work together, emergence arises! Emergence as a concept refers to the idea that the whole is greater than the sum of its parts. New properties and behaviors emerge at the system level that cannot be understood by analyzing the parts in isolation. Systems thinking is bla bla bla, you can read it from…
In AWS, in order to access resources in other accounts without creating new users or handling passwords, you can use sts:AssumeRole. Let’s say that you have some resources in AccountA (AWS Managed Prometheus for example) that you want to access from AccountB Account A In AccountA create a role account_a_role that has 2 types of policies: a Trust Relationship that define which entities can…
For me, lateral thinking is: Solving problems using an indirect and creative approach. Using reasoning that is not immediately obvious and involving ideas that might not be obtainable by using traditional step-by-step logic. One “simple” approach to “think outside the box” is: Understanding WHAT you want to achieve. Understanding WHY you want to achieve it. Knowing the…
I’ve been thinking a lot about how we interact with technology, and I’ve come to realize that being familiar with something doesn’t always mean it’s simple. In fact, it can sometimes make things more complicated. For example, when I’m working on code or documentation, I find myself getting too close to the issue. It’s easy to lose sight of the bigger picture and…
In order to make it faster, we need to understand why it is slow.
Which architecture should I choose? I don’t think this is the right question to ask. A better question would be, given the current state of my service/product, which architecture will provide what I’m looking for? For example, performance, independent deployments, application boundaries, etc. For example, many people mention that we should start with a monolith and while I agree with…
GitHub actions are a problem because they lock you by the balls and you cannot reproduce your pipelines. Getting to depend on all those small Actions saves 5 minutes today, only to make migrations immensely painful tomorrow. Build, package, and release software should be written as standalone scripts that in principle could even run in the developer’s machine. Moving them to CI is just…
A pattern of shared assumptions that groups have learned as they solve problems of external adaptation and internal integration, that has worked well enough in the past to be considered valid and therefore to be taught to new members as the correct way to PERCEIVE, THINK and FEEL – Edgar Schein Culture evolves over time, driven by both external influences and internal dynamics. It’s…
Look outside your building. Domain knowledge transfer means taking a solution from one field and applying it to another. Software engineering can be insular. We think our problems are unique. They rarely are. Other disciplines have already solved variations of our exact issues. Read about biology, economics, or urban planning. A city’s traffic grid solves the same routing problems as a…
A made up mind is hard to change (Jeff Bezos or Confucius, I don’t know who, but the phrase got stuck with me) If you stand still, you fall behind. What worked yesterday might not work tomorrow. You have to adapt. Guessing as a Strategy Guessing gets a bad reputation. In technology, it is a requirement. You make predictions based on what you know. You experiment. You fail. You learn from the…
Society seems to follow a cycle of moods that last around 20 years each. Each cycle is called “Turning” NOTE This is an unfalsifiable theory, so take it with a grain of salt. High The first turning is a high, which occurs after a crisis. During the high, institutions are strong and individualism is weak. Society is confident about where it wants to go collectively, though those outside…
First of, congratulations! Few words before moving on: There is no such thing as DevOps Engineer, DevOps is a philosophy, a way of working. Your DevOps role will depend on your organization structure and maturity. Some organizations use DevOps, SRE and Platform engineering interchangeably, don’t worry. Focus on the goal. You will work in a team, empathy is mandatory. A lot of people ask, do I…
As in the ship of Theseus… If you replace each Kubernetes component… what is it that you get at the end? An API, an ecosystem and the sum of its parts. BTW, is kubernetes an overkill? Yes and No I think is a necessary evil to avoid selling your soul to a cloud provider.
Or is it a perspective? is it the result of the human ego? or is it just a communication problem? If you look at biology, evolution has found a way to design its systems in a way that each component has a defined interface to communicate, and more importantly, each component is free to “experiment” or evolve independently from each other by random mutations. When changes in one component require a…
Docker multi-stage build is a great way to build a container images with a minimal footprint. Compiled languages like Go or Rust can take advantage of this by just shipping a binary to a container This is an example from the official docs: FROM golang:1.16 WORKDIR /go/src/github.com/alexellis/href-counter/ RUN go get -d -v golang.org/x/net/html COPY app.go ./ RUN CGO_ENABLED=0 go build -a…
Big O Notation (or the Big O) is used to describe how long and complex an operation will be based on its input. Complexity could mean that an operation takes N amount of time, or N amount of memory, N CPU resources, etc. There are some notations to describe this: O(n) -> The complexity grows linearly based on the size of the input. O(n^2) -> Grows at a square ratio of its input.
Ambient mesh is a new data plane mode for Istio that doesn’t rely on sidecars. It gives users the option to forgo sidecar proxies in favor of a mesh data plane that’s integrated into your infrastructure. Ambient mesh benefits are: Minimal configuration for traffic encryption. Same configuration for L7 policies as ”normal service mesh”. Take less resources because no sidecars are needed. Easier…
aka Pareto principle. After writing Python for over 10 years, I can code in it as naturally as I speak Spanish or English. The honeymoon period is over, though. Python has some disadvantages I’m no longer willing to tolerate: Distribution complexity: Sharing code requires the right interpreter + pip registry access on target environments. Air-gapped environments become nightmares. Tools like…
Update Jan 2023: Is OpenStack Still Needed in 2022? - Thierry Carrez, Open Infrastructure Foundation And why Kubernetes “won”. I owe my career to OpenStack and to all its contributors. I have made excellent friends, I learned a lot from them and the project itself. For that and more, thanks a lot OpenStack. However… Even though OpenStack has never been better, I can’t shake the feeling that is…
TOOL Website TOOL - Lateralus Black Then White are All I see In my infancy Red and yellow then came to be Reaching out to me Lets me see As below so above and beyond I imagine Drawn beyond the lines of reason Push the envelope Watch it bend Over thinking, over analyzing, separates the body from the mind Withering my intuition, missing opportunities and I must Feed my will to feel my moment Drawing…
Define the scope of your system Start by asking this broad questions: Why is your system required? This will help you find the reason why this system or organization exists What is the goal of your system? These two questions will help you understand your organization’s requirements, use them as a starting place, then clarify as much as you can those answers so you can start building a clear…