Recent content on Yes Way Jose
This month I’m listening to the audiobook “Influence: Science and Practice” by Dr. Robert Cialdini to understand the psychology of persuasion, why people say yes, through six core principles: reciprocation, commitment and consistency, social proof, liking, authority, and scarcity. Cialdini is an academic, not a salesman, and he provides compelling examples and anecdotes. I…
One useful tool I use when I want to make a hard decision easy: I picture myself at 80 years old and ask, will I regret not doing this? If the answer is yes, I have to try it. If the answer is no, I can let it go and move on without losing sleep over it. I learned about this reading The Everything Store by Brad Stone. The idea is called the Regret Minimization Framework. When Jeff Bezos was…
One of the things I learned at Amazon is how to manage a workload significantly larger than any other place I’ve ever worked at. I understand the reason for it: at Amazon the team that creates the service owns it. That means they are the operators and responders as well as designers and testers. As a result, senior engineers often have more than one project going on at any given time. So how…
One big challenge that comes up a lot at work is that there are always more things to do than time and people to do them. There’s always tech debt that we want to pay off. Long-term migrations away from systems that are going to reach their end of life, that weird bug that nobody knows for sure what’s causing it, the big project announced publicly that has a lot of visibility. Meetings…
I just started reading an early access ebook copy of The Linux Memory Manager by Lorenzo Stoakes, thanks to the book bundle Linux the Good Stuff . I am trying to learn all about memory management in Linux. This is deep stuff. One time, long ago, at a different job, I implemented malloc for an embedded system running on a custom-made RTOS. At the time I felt like a l33t hack3r even though I’m…
Being Oncall I’m primary oncall for my AWS Service this week, so I might not have enough time to write every day, but as promised here’s a quick thing, what’s oncall like? I’m not going to give you stories today, but I can tell you what concepts go through my head during oncall week. Monitoring Distributed Systems. Logs, metrics, alarms, how to read them, troubleshoot…
There’s a mental model that comes up a lot in my day to day. When I’m solving a problem, I start by thinking about the result I want, and work backwards from there. The challenge is having the clarity to define the problem and taking care that I’m solving the right problem. This is why I start with the goal in mind. I use the working backwards method when designing a feature. We…
“It’s tough to make predictions, especially about the future” –Yogi Berra Making decisions is a life skill, no doubt about that. Making decisions when you have limited info is common and it’s what eventually separates good leaders from the rest. On today’s episode of “One small thing per day” I want to mention a gem I learned reading “ Thinking…
I recently finished reading Thanks for the feedback by Douglas Stone and Sheila Heen and I can’t recommend it enough, especially now that it’s performance review season at many companies. I’ve taken many lessons from this book, but I want to focus on something that was super insightful when trying to understand people’s interactions. All of us have blind spots. Things that…
A new goal: publish a tiny micro post every day I have been meaning to write for a long time. I want to publish updates about what I’ve been learning in real time, but I always want to give beefy, meaningful, and helpful content, and that makes me put things off. So I think that starting now (bad timing, I know, this is not an April Fools’ joke, I promise), I will publish daily: one…
The Problem Containers Solve Docker emerged as a lightweight alternative to virtual machines. VMs consumed significant resources and took 3-5 minutes to boot, making horizontal scaling expensive. Containers package applications with dependencies into images that start in seconds, not minutes. The Networking Challenge Without network connectivity, containers offer limited utility. Running a single…
A look at the AI and ML books that shaped my understanding of artificial intelligence, data science, and the ethics of technology.
Ever wondered what happens under the hood when you launch an ECS task with awsvpc networking? Let's explore how network namespaces are put together when you run containers in ECS Managed Instances.
What is the one metric for sucess in your career? Lately I’ve been thinking about what it means to grow in my career. Like, How do I know I am doing a good job? Is there a metric I can track? It turns out there’s people way smarter than me that have already thought about that and written about it. Tanya Reilly wrote an excellent book on that subject The Staff Engineer’s Path…
I’m lucky that I get to work with people who are good at what they do. The corporate-speak term we use at Amazon is the Leadership Principle of “Right A Lot” For me it’s about having a scientific mindset, or more plainly, constantly wondering, “how does this work?” “Why are things working this way, instead of that way?” or my favority “What…
Practical Engineering by Peng Zhang I want to give a quick shout out to Peng’s blog peng.fyi Peng works with me writing software for AWS Fargate’s Dataplane. I have been following his posts the last few days and I really like this blog because: Technical topics that apply to most folks in my team His posts are short and to the point. He illustrates his points with code. The code is…
Book Review I’m almost finished with The Siren’s Call by Chris Hayes and it’s interesting enough to call out here. The book starts with an excellent explanation of what attention is and how it works. Then the book presents two analogies to use when thinking about attention. The first one is to think of Attention as a resource that drives the economy, like labor, it is…
This week I had an Eureka moment at work. You see, there are many things I see at work that I would like to improve. Technical debt in our codebase. Inefficient processes. Communication silos across teams. The list of things one can improve never ends, this is true in all software shops. System complixity grows as new functionality gets added, inefficiencies optimized, bugs fixed. Secuirty…
The Problem Containers Solve Docker emerged as a lightweight alternative to virtual machines. VMs consumed significant resources and took 3-5 minutes to boot, making horizontal scaling expensive. Containers package applications with dependencies into images that start in seconds, not minutes. The Networking Challenge Without network connectivity, containers offer limited utility. Running a single…
I have been an official Go programmer for three years now. Unlike many people in my team, I remember the day Google announced go. I don’t remember if it was in Hacker News or /programming reddit, but I do remember watching the go math package compiling in less than a second, at the time, I was writing C++ for an embedded system. Building the whole model as we used to say took 45 minutes,…
A curated list of falsehoods programmers believe in. The Code we write is a representation of the things we believe are true about the world. Every one has just one name, right? Well, most of the time, yes. All of the time? No. If you write an app for yourself, or your small business most of these assumptions are fine. If you write code for millions of users, you are going to find exceptions. Some…
Book Summary: Thinking in Systems Let’s say you want to build the perfect self-driving vehicle. This thing you want to make is composed of many parts. There’s the car itself made up of many subparts (engine, tires, transmission, etc) as well as the AI tech. Realistically you will need a bunch of speciallized embedded systems as well as a central computer to orchestrate everything.…
The Top 3 Podcasts for Software Developers Go Time by Changelog link This is the podcast to keep up-to-date with all things Go. The jokes are nerdy and the hosts are sometimes not as funny as they think they are, but the content is great and they have a wide set of guests in the show that make it a must for all people who write go for a living
Article: Lessons Learned from Twenty Years of Site Reliability Engineering Link to article The site realibility team at Google put together a summary of the lessons they have learned over the years. I am glad they decided to share. The best way to learn is by trial and error. Want your product or service to be better? Launch it, monitor it, and learn from the mistakes. It nice to learn from…
Book Review: The Staff Engineer’s Path by Tanya Reilly Tanya Reilly gives a guide for individual contributor software engineers who wish to grow their career but do not want to become managers. It gives insights about what a staff engineer does, and what you need to do to perform at that level. This is a technology-agnostic book. It gives the reader a high level view of the functional areas…
Go Russ Cox put out an article yesteday about adding the abilities to run coroutines in go. Today I learned the difference between a goroutine and a coroutine . Coroutine is a concurrency pattern in which only one runs at a time. Say we have coroutine A and B. B waits while A runs then A yields to B and A waits while B runs. It turns out this is useful in a few scenarios.
Lately I have been learning about File Systems from the book “Operating Systems. Three Easy Pieces” by Remzi Arpaci-Dusseau. I used to think that I knew how file systems worked because the interface open, read and write is so straight forward, what else could there be to it? But then at work some weird issues come up where some weird behaviour happens, like, du says the disk has space…
I’m trying to get through all my technical books that I’ve adquired, and never gotten around to. Not going to lie, the Knuth books are intimidating. They are actually not that bad to get through, but they are books that I pick up, read a few pages on a specfic project, try to do a problem or two, and that’s it. The other books are less intimidating, more doable, I’m pretty…
Link to Paper This paper, published in July 1974 is remarkable because the design decisions that were made back then by these guys working at Bell Labs on an operating system for the PDP-11 are still relevant. I am still struggling to create a mental model of the unix file system, the fact that it looks like a single tree with the root at the top while simultaneously you can have multiple devices…
Link to Paper From the abstract: “This paper analyzes the scalability of seven system applications running on Linux on a 48-core computer…using mostly standard parallel programming techniques -this paper introduces one new technique sloppy counters these bottlencek can be removed from the kernl or avoided by changing the application slightly” This paper has an excellent system…
Go I have been writing go since a little after last year. I actually remember hearing about go when it first came out, back then I honestly never thought I’d be getting paid to work in it. Even though I’ve been writing code in go for a while, I don’t think I know the language in enough depth to consider myself a go expert. I want to change that. So I am going to start writing…
Omega Link to paper Omega was the second cluster manager system built by Google. It is Borg’s sucessor and it was designed as a happy medium between Borg’s centralized scheduler architecture and Mesos’s two-level approach where the placement is delegated to the running framework. Omega shares the state of the cluster among leaders and uses optimistic concurrency control (detect…
Borg Link to Paper Borg is the cluster management system that runs hundreds of thousands of jobs at Google, it is the original system, it’s sucessor Omega was written as a reaction to the lessons learned from it. Kubernetes is the third system written with the lessons from those two. This paper helped me understand a few things about my own system since we have our own cluster managenet and…
Link to Paper This paper was published originally in 1983 by the legendary folks from the Xerox Palo Alto Research Center. The hints and tips should sound familiar but it’s interesting to notice the layer the author is talking about, these guys were designing at very low level. The fact that the same rules apply now it’s remarkable. It turns out breaking up a system into the right…
Link to Paper This paper is a list of recommendations for running a large-scale system that aims to keep quality high and costs low. The author comes from the Windows Live Services Platform but this might as well be read as an Amazon internal guide since I didn’t see a thing in this list that we don’t do (or aim to do) in AWS. EDIT: Of course all these things sound familiar! The author…
Link to paper Today’s paper comes thanks to Will Larson this is a recommended paper in his book Elegant Puzzle This paper builds on the concepts from the the CAP Theorem which essentially says that when it comes to distributed systems you can only have 2 out of these 3 qualities: Consistency Avalaibility Partition Tolerance Then it introduces two concepts harvest and yield which is…
Day three: Design patterns for container-based distributed systems Link to paper Containers provide the ability to package, deploy and reuse applications using a natural isolation boundary. Developers can expose application-specific functionality as interfaces as well as more generic hooks for many systems like metrics, health, etc. In this paper the authors present the readers with two types of…
Welcome to day 2 of my paper-every-day journey. Today we’re going to cover Borg, Omega and Kubernetes This paper goes over how Kubernetes, the de-facto, open-source orchestrator of containers was developed using the lessons learned from building Borg and Omega, Google’s internal job orchestrator. Kubernetes is a container orchestration system that aims to make developing and deploying…
I’ve decided to read a research paper each day. I am doing this in order to get better at my craft. I want to be better at designing software and it has been shown in research that the people who are really good at what they do are those that do “delibarate practice” I am most definitely NOT the best software system designer out there, but talent is overated and I intend to…
About Me I’m Jose Villalta, an infrastructure engineer at Amazon Web Services building the systems that power millions of containerized workloads. I maintain the Fargate agent - the data plane that creates and manages serverless containers at AWS scale. My Path to Infrastructure My career has been about building systems that work when it matters most. I spent seven years at Motorola writing…
Started a new page to write about tech stuff. Work in Progress
Summary Amazon 2018 - present AWS Fargate: Dataplane team. Writing Golang mostly. Worked on retail site for 2.5 years using the away team model. Impinj 2014 - 2018 Wrote Ruby on Rails Motorola 2007 - 2014 Low level firmware Co-op Intern at IBM 2005 - 2006 ASIC design model testing Education Master’s Degree in Electrical and Computer Engineer from the University of Florida Bachelor’s…