If I have to pick one thing that’s very hard with large scale software, it would be what engineers fail the most – it’s not communication, not leadership or some kind of social skills. It’s planning and executing a migration. How often did you (or someone in your org) design a new shiny system to replace a legacy one, but failed at closing the migration resulting in the need to maintain two…
A (sadly) too common assumption is that a long design doc means that the problem being solved is more important, more complex and/or more impactful. This is wrong. The only thing a longer doc guarantee is that it takes longer for people to read. As a senior IC, my time is limited (and precious). I don’t have time to read 50 pages for a simple problem unless either the solution is complex and/or…
One question I guess every now and then as a senior staff software engineer is “how many hours a week do I work?”. Behind the question, there’s the implicit assumption that the main cost of being a staff+ software engineer means you have to work long hours. My personal experience is a bit different – the true cost of being a very senior engineer is responsibilities (and stress). The more senior…
One of the most common skills I use as a software engineer isn’t some syntax about threads, promises or coroutine, but how to quickly manipulate data – this can be being able to quickly extract the files responsible for test failures from a large log, parsing a json blob to extract only one field, removing duplicated entries in a file etc. This is something you should be able to do from multiple…
When you join a big tech company as a new grad, there are some expectations that you reach a senior level in a timely manner – e.g. Google/Meta/Databricks expect you to go from L3 -> L5 in 4-5 years. While there are some external factors to grow to staff and beyond, the path to senior is fairly straightforward in these companies. This means that if you leave before you reach that level (e.g. as a…
I strongly believe you should invest into automatically formatting your code, because beyond esthetic reasons, having code automatically formatted allows: To reduce the cognitive overhead for engineers – you can just write code that compiles and it will be automatically styled, e.g. you don’t have to worry about whitespace issues. Code is easier to read too, it’s more consistent. To ease large…
There’s the myth that engineers want to constantly refactor for esthetic reasons and that PMs must constantly fight such efforts. My experience is that this is a terrible dynamic: You, as an engineer, should refactor code as you see fit – you are free to take shortcuts and hack things for a quick launch, but you’re ultimately the owner of your code. If you pile terrible hacks, they will eventually…
One issue that plagues many companies is when they use waterfall processes – e.g. the engineer wait for the perfect design doc from their TL, their TL waits for the perfect PRD from their product manager, and their product manager waits for customer feedback, etc. and somewhere a manager waits for the quarter to end 😅 This is a cultural issue and can kill companies. Everyone here can improve the…
One of the most common “complaints” I hear from engineers is the lack of scope or opportunities. In growing companies, my 2 cents is that the problem isn’t the lack of opportunities, but whether you are able to recognize and grasp them. It’s a bit hard to provide an exhaustive list of where the opportunities might be for you, but in general if you look back at what you have worked on and try to…
If you look at compensation for very senior engineer in the US (e.g. E7 at Meta, L7 at Databricks or L8 at Google – see levels.fyi for the data), they may not make sense – how can one justify a 7 digits compensation for a single person? There are a lot of considerations that go into compensation (responsibilities, job market etc.), but one of them is impact. For senior ICs, their impact is not…
As much as being a software engineer is not a competition on who can type faster, I think being comfortable typing fast is a significant edge.. While you can churn out code faster if you can quickly type, the most important part is that you can rely on muscle memory to type. You can focus purely on the content of your document/business logic of your code without having to spend an ounce of effort…
Having a mentor will help you grow a lot – usually your mentor is your local TL but there seems to be some kind of hidden expectation that you can only have one mentor. I’ll get straight to the point, you can (and should if you desire) have multiple. Becoming a mentor isn’t becoming some kind of lord – it’s not even an official role. Some people might be mentors without knowing it (e.g. I think…
ORMs (Object Relational Mapping) get either a lot of love or a lot of hate – people usually love them because it’s a higher level API than directly hitting the database – so they don’t need to know the exact names of the tables involved, which keys are used for joins etc. Others hate them because they potentially hide SQL issues (e.g. join operation done without index). I personally think that ORM…
While I was at Google, everyone’s level was public by default. One could opt out of showing their level but the vast majority of engineers had their level public. I recently learned that now levels are hidden by default – it’s unclear why though it looks like it was tied with a migration of the tool displaying levels. My personal take is that levels should be public: It’s useful to calibrate…
In a large tech company, the fastest way to increase your comp is to get promoted. You may get a slightly bigger comp increase if you ask for it, but it likely won’t be significant. In big tech companies, compensations are automatically computed based on a magic formula mostly relying on location, level and rating. Once the initial numbers are out, managers have a bit of leeway to adjust comp as…
When we talk about latency, we often take goals like P90 under 300 ms or some other numbers. I however think that it’s more important to control the tail of your latency. Namely, I think the latency of a service should be constant or, if it depends on n , n should be capped at a reasonable threshold. If you don’t, you can take a goal at P50, P90 but your tail is unbounded – and it usually gets…
These are very much incomplete thoughts on managing people out, aka firing people. I’ve never fired anyone – I never had to but I’ve noticed some kind of paradox among engineers: Companies firing poor performers are bad – they create poor work life balance for the employees Companies not firing poor performers are bad – these bad performers have a negative impact on the company and people complain…
In (big) tech companies, promos are decided by a promo committee. These committees exist to make sure promotions are based on facts and are fair across the company. If you can join a promo session, you should – especially if you have never done it before. Different companies have different requirements for who can be part of these committees – the common rule I’ve seen in our industry is that you…
The problem isn’t the monorepo, it’s the spaghetti code that’s inside. Using a monorepo is both a blessing and a curse. It’s a blessing because you can easily share/reuse code across the whole company. This makes creating choke points for enforcement significantly easier. But it’s also a curse, it’s very easy to end up with spaghetti code where internal APIs are unstable at best, non-existent at…
During the last few design interviews I conducted, the candidate said something along the lines of I’ll use NoSQL because it’s faster I’ll use a SQL database because it’s more reliable about some database Throwing such a statement reflects poorly on the candidate in my opinion as it shows a shallow understanding of different types of databases and/or an attempt at showing off by throwing some buzz…
As a new grad, it may be tempting to join a company as a QA (Quality Assurance) engineer – it’s often slightly easier to get in as a QA engineer than a standard software engineer, and you’re still an engineer. My recommendation would be to not take this position if you can have a software engineer offer in another company. If you are hoping to transfer ladder from QA engineer to software engineer,…
As software engineers, we sometimes refer to our work as “craft” – part of it is because we take pride in our work. We tend to cherish our work (e.g. our open source libraries), polish our software over a long period and relentlessly chase perfection. If you have done good work, you should be proud of it – the same way an artisan would be proud of their product. You should however be cautious and…
I often hear engineers complaining that they cannot grow because their company hires too many senior engineers and that it should instead grow its current employees. If you work in a growing tech company, new hires aren’t the reason you can’t get promoted. In a growing company where staffing is the most scarce resource, there are tons of unstaffed opportunities for you to grow and get you to the…
Have you ever faced a problem that was so difficult or so large that you gave up before even trying? This is something that’s holding back a lot of engineers – including senior ones and maybe especially senior ones. When you are a junior engineer, your knowledge is limited so you may be scared of trying to resolve problems where you have no expertise. My best advice is to Give it a shot Ask for…
Correlation is not causation but it’s enough to rollback. Every now and then, an incident happens and the author of a change that lines up with the beginning of the outage will argue that their change cannot be the root cause and that correlation is not causation – and push back against a rollback. While I agree that correlation is not causation, large scale systems are so complex that your…
The title kind of implies that FAANG is at the top and while this is something we can debate, the truth is that they rank companies in different tiers. By “they”, I mostly mean recruiters, managers and hiring committees. The goal of this post is to give a bit of background on why (I think) big tech companies tend to classify schools/companies for hiring purposes. The assumption is that good…
There’s a broad consensus that little ergonomic improvements (e.g. using a curved/split keyboard) helps a lot with your physical well-being over time. I think the same is true for your mental well-being. As engineers, we have the chance to be able to invest in/build our own workflows/tools but don’t do it enough. You can make simple but effective improvements to your mental well being (and overall…
One of the best thing that happened to me (beside marrying my wife – she reads my posts for time to time 😅) is to have had the opportunities to live abroad (in Europe, China and the US). When you live in a different country, you have the chance to discover different cultures and different worlds. A few interesting illustrations are: Living in Europe, you can exercise every day. While you can also…
This post was prompted by Strava heatmaps being broken last night even though Strava’s health status page was showing all green – it’s hard to detect a minor/partial outage of your system. For example one thing we struggled for some time in YouTube Ads was reliably being able to detect outages on the Kids apps. These apps generate so little traffic compared to the main apps that we had to create a…