Recent content on TBNL
Recently I merged a pull request that was like any other pull request. But things unfolded quite differently. Things didn’t break, but could have. Things shouldn’t have happened, but did. The typical unlikely combination of factors, involving Argo CD ApplicationSets and mysterious AWS EKS Fargate nodes, resulting in some head-scratching, troubleshooting and interesting take-aways. In…
During my weekly long runs I listen to podcasts. Mostly about tech or music, and a couple of months ago I listened to an episode that contained some quotes that kept resonating: “Going caveman” and “you do you”. Let’s unpack what they mean in the context of working in tech, the AI-ification of everything, and the state of the world today1. A bit of background The…
Introduction Over the past ten years or so, my role has gradually shifted from software to platforms. More towards the ‘ops’ side of things, but coming from a background that values APIs, automation, artifacts and guardrails in the form of automated tests. And I found out that a lot of best practices from software engineering can be adapted and applied to modern ops practices as well.…
Recently I passed the last of the exams that grants the Golden Kubestronaut title. And it was quite the journey. About fifteen years ago, I dabbled a bit in certification via PHP and Zend. Back then, PHP tried to be corporate Java, including the design patterns and certifications that come with it. And I went along! Let’s say it has been a phase, in which I built quite some monstrosities 1.
Past Friday I had a focus day, hardly any meetings and time to do some scheduled deep work. Also, Fridays in general are ‘release day’, which always means anticipating on new album drops1. This Friday saw the luxury of two excellent new albums being released: Biographyte by Cytotoxin and Aspiral by Epica. Both albums where ‘a lot is happening’. For a bit of context: I work…
Introduction Past week, I attended FastFlowConf, “a Team Topologies Conference”. As the Team Topologies site describes itself: Team Topologies is the result of years of research into how successful leaders design team-of-teams organizations delivering business outcomes through technology Team Topologies, to me, feels like a logical evolution of Agile. Most practices described in the…
Introduction Recently I needed to test a Keycloak upgrade. This required me to deploy both the new Keycloak version and a sample OIDC application on my local Kubernetes setup. And it pointed me to a thing I kept postponing: Improve my local development DNS, routing and TLS setup The challenge Until recently, I used urls like keycloak.127.0.0.1.nip.io:8443. This points to 127.0.0.1 port 8443 which…
Introduction At KubeCon 2024 in Paris, CNCF introduced the Kubestronaut program. Quoting: The Kubestronaut program recognises community leaders who have consistently invested in their ongoing education and grown their skill level with Kubernetes. Individuals who have successfully passed every CNCF’s Kubernetes certifications – CKA, CKAD, CKS, KCNA, KCSA – will receive the title of “Kubestronaut”…
Introduction These days, applying Infrastructure as Code (IaC) changes is typically automated by using pipelines or GitOps. This Infrastructure Lifecycle shares some characteristics with the Software Development Lifecycle: It starts with defining a goal, which is then followed by authoring of code, checks and tests, and a code review step. However, there are also some aspects that make the…
When dealing with traffic of a news site like NU.nl, one quickly learns to expect the unexpected: You know there will be traffic surges, but you can not predict when they will be. This is quite unlike, for example, ecommerce where the holiday season can be predicted. Or big event ticket sales, where the moment tickets will start to sell, is known beforehand. With breaking news, however, the moment…
Introduction To me, ever since occasionally needing to use it, ‘enterprise software’ is the definition of software that makes people feel unhappy. Except perhaps for the people that make money of it. It miraculously gets something done, despite its horrendous user experience, in the process sapping all energy of whoever is forced to use it1. So, when I recently saw this tweet by Jason…
Introduction After IBM acquired Red Hat in 2019, it recently has acquired HashiCorp. So now Big Blue not only has Ansible in its portfolio, but also Terraform. Tech articles and social media galore, and many people have found out that if you mix ‘Terraform’ and ‘Ansible’, you end up with ‘Terrible’. Depending on your tech stack, it is a great combination:…
Introduction In a presentation about CI/CD I gave recently, I briefly mentioned the 12 factor methodology. Somewhere along the lines of “You might find some good practices there”, and summarizing it as: artifact configuration + --------------- deployment After the talk, a colleague of way back, came to me and said: “You were way too mild in suggesting it. It’s mandatory,…
A new year: Reflection time This article originally appeared on the Fullstaq blog As we have moved from 2023 to 2024, it is a good moment to reflect. Undoubtedly, one of the biggest topics of the past year was the rise of AI. But somewhat closer to my day-to-day job, there were some events that stood out: The Amazon Prime move from serverless microservices to ‘monolith’ blogpost. Which…
Introduction Feature deployments can help teams to validate work in progress, and test pull requests before merging. This article will address some things to consider when setting up feature deployments. Furthermore, an implementation in use at NU.nl based on AWS, Kubernetes, Helm and Akamai, will be described. Parallel tracks. Source: Bing Image Creator In this article: Introduction The use case…
Introduction Recently we were performing an EKS upgrade. We use managed node groups, and updates have become routine without any surprises. And then, hardly a surprise: Surprise! 1 PodEvictionFailure: Reached max retries while trying to evict pods from nodes in node group ng_a_on_demand It is worth noting that nothing breaks because of this. The node group update procedure is basically a…
Introduction The last time Kubernetes Community Day took place in the Netherlands was September 2019. Then the pandemic turned the world upside down, and suddenly we find ourselves in February 2023 for the next event. In those three years, the event became bigger, doubling in all aspects: The number of days (2), number of tracks (2), and audience now totaling 450 attendees. So either people fancy…
Introduction Once or twice a year, NU.nl organizes a hackathon, allowing its IT staff to dabble in new technology and pursue ideas, usually guided by several themes. 2023 kicked off with a hackathon having the themes ‘NU2030’, ‘NU.nl from scratch’ and ‘A greener NU.nl’. Mid-2022 I visited EdgeCase, a 1-day conference about running Kubernetes at the edge. One of…
ImagePullBackOff Recently we experienced a specific ECR production image being absent that really should not have been purged by our lifecycle hooks. It happened in two scenarios: EKS node replacement, new node can’t pull image. Rollback (we make mistakes). The rollback involved multiple images, all but this particular one were still present on ECR. As they should. So far we’ve been…
Edgecase 2022: Kubernetes at the edge Finally, after the pandemic abruptly changed every meetup and conference into a video conference, in-person events are possible again! The first conference I attended since early 2020 was Edgecase 2022, organized by Fullstaq and focusing on running Kubernetes at the edge. This is not directly what we are doing or intend on doing, since we use AWS.…
EKS and its hunger for IP addresses Kubernetes allows running highly diverse workloads with similar effort. From a user perspective there’s little difference between running 2 pods on a node, each consuming 2 vCPU, and running tens of pods each consuming 0.05 vCPU. Looking at the network however, there is a big difference: Each pod needs to have a unique IP address. In most Kubernetes…
Recently I worked on some infrastructure changes that resulted in terraform plan showing more, and more impactful, changes than expected. Diving deeper, it appeared that a lot of the planned changes could be avoided by some preparations, resulting in a terraform apply with no impact at all. I love it when a terraform plan comes together. Source: PeteFarrow, via redbubble.com Ordered lists and…
Recently we migrated our CDNs from Cloudfront to Akamai. We use Terraform for infrastructure as code (IaC) and luckily it supports Akamai as well. Since we had Cloudfront distributions for pretty much every environment, it served as a good moment to reflect on what we’ve taken for granted in the past years, especially since Akamai has the concept of a ‘staging network’ which…
Recently we discovered that what used to be a minor line in our monthly billing graph, had become several strokes wider, amounting to a couple of hundred dollars on the monthly bill. This line represented EFS, wich we use, but by no means in large amounts. So we thought… As it appeared we were using around 3 terabyte of data. Investigating We use EFS as our go-to persistent storage solution…
Slides of the presentation I gave at a developer.overheid.nl meetup on the topic of moving from Kops to EKS. This meetup took place (online) at Sept. 29, 2020. Slides can be seen full screen as well.
Friday sept. 13th the small conference ‘Kubernetes Community Day 2019’ took place in Pakhuis de Zwijger in Amsterdam. Small-scale events like this are a nice counter-balance to the massive scale of CNCF/KubeCon: No flying for miles. Just one track so no need to choose and no missing out on interesting talks. From what I understood this was one of the goals: Being a smaller, more…
Slides of the presentation I gave about Kubernetes practices and learnings at NU.nl. This presentation was the first of two at the Dutch Kubernetes meetup at the Sanoma Netherlands offices, that took place on Sept. 5th 2019
The end of May 2019 saw 7700 people, myself included, visit Barcelona to attend ‘KubeCon + CloudNativeCon Europe 2019’. That’s a lot of people and a significant increase of previous editions (Copenhagen 2018: 4300, Berlin 2017: 1500, according to this Dutch source). Next year’s edition will be ‘around the corner’ in Amsterdam and is projected to attract at least…
Since a number of months we (NU.nl development team) operate a small number of Kubernetes clusters. We see the potential of Kubernetes and how it can increase our productivity and how it can improve our CI/CD practices. Currently we run part of our logging and building toolset on Kubernetes, plus some small (internal) customer facing workloads, with the plan to move more applications there once we…
Tuesday 4th and wednesday 5th the DevOps Conference 2018 took place in Munich. Offering a program filled with talks addressing ‘DevOps’ topics such as CI/CD, Kubernetes, Security, Company culture and change, Microservices, I was lucky enough to attend. Below you’ll find a slightly redacted version of the notes I took during the talks and off-track chats I had with several people.…
TL;DR: Configure CloudFront to set the Cloudfront-Forwarded-Proto in order to allow a Django application to know the client’s request protocol. Recently, while developing an API that makes use of Django Rest Framework and is delivered using CloudFront, we noticed the absolute URLs it generated to be HTTP, whereas the CloudFront distribution is HTTPS only. Not really surprising when thinking…
TL;DR: It’s possible to configure HAProxy to have separate health check and service endpoints, allowing to set up different firewall rules for each. Scroll down for config samples. Case outline When designing a highly available service on EC2, AWS Elastic Load balancers are quite often a key component. A common setup is to have an internet facing ELB forward requests to EC2 instances that…
TL;DR: PyCharm and IntelliJ have some odd defaults that can mask mistakes until they hit CI. Introduction In a project we’re working on we recently witnessed quite some builds fail on unit tests that apprently passed on local development environments. As it turned out all of the failures where related to import errors and were caused by developers accidentally importing relative to the main…
Until recently I relied on OS X native python for anything 2.7.x and Homebrew’s python3 for anything 3.x.x. As already implied by the lots of ‘x’-es: Far from ideal. But it worked. Until I accidentally ran a brew upgrade which put my python 3 version to a bleeding edge 3.6.0. Newer than what’s provisioned on our stack and besides that I ran into some errors when installing…
Introduction For some years my blog has been inactive. Now this could imply that ‘not much was happening’ on a professional level but that hasn’t been the case, on the contrary. Recently I’ve been thinking about adding new content and it became clear that my old Wordpress setup wasn’t the best fit anymore for my needs. Several people I know have been very positive…
The professional I’m employed as lead developer / architect of the web team of NU.nl (Part of DPG Media), the largest news site in the Netherlands. I do DevOps, AWS Cloud engineering, Kubernetes, Python, Javascript and quite some other buzzwords. I studied Industrial Design Engineering so I’m comfortable with talking about UI, UX, colors and pixels as well. The way I (aspire to) work…
Nowadays developing web applications usually requires a flexible process due to changing business logic, shifting priorities or new insights. Besides choosing the right methodology this also requires designing the application in such a way that this flexibility can be achieved. Domain Driven Design fits this process as it isolates business logic in the Domain layer and separates it from…
The yearly dutch PHP event is getting close so time to check out the schedule. Once again there are lots of interesting sessions to choose from so I probably will miss some of the good stuff. Mixing ‘simply interesting’ and ‘directly usable in day-to-day job’ results in the preliminary list: Day 1 TDD and Getting Paid – Because indeed TDD is good and can be hard to keep doing thoroughly Pursuing…
A while ago during a project we were asked to implement autofocus on the generic search field that every page in the application has. At a first glance a pretty straightforward task, from a technical perspective that is. Not from a user perspective, as indicated by a colleague mentioning his dislike of such autofocus fields “because they prevent backspace going to the previous page”. In this post…
A while ago I installed Zend Server Community Edition on OS X which was pretty straightforward. It was only recently that I found out that, as opposed to mysql which worked fine, mysqldump didn’t work correctly and terminated with the error: ``A while ago I installed Zend Server Community Edition on OS X which was pretty straightforward. It was only recently that I found out that, as opposed to…
Past weekend the Amsterdam RAI was the centre of the PHP universe as there the 2010 edition of the Dutch PHP Conference was held. Similar to past year it consisted of two presentation days, which I attended, preceded by a tutorial day. Among the presentations I attended on the first day were: Kevlin Henney’s keynote presentation, titled 97 Things every programmer should know. I suppose every…
In my previous post about Zend_Form I showed how, using Zend_Form, a form’s structure can be separated from it’s presentation and how to use custom Decorators and Validators. The example used showed a form that is tightly coupled to a record in a database: One form edits one record. There are however numerous occasions where no ‘one to one’ connection exists and where the fields that need to be…
Most components of Zend Framework can be used without using the entire framework and Zend_Form is no exception. It’s a versatile component that can be customized to great extent. The payoff is that seemingly easy tasks can seem quite complex to complete and involve concepts like Decorators and View Helpers. Complexity is increased by the fact that most tasks can be achieved in multiple ways. Forms…
About five months after having enjoyed server-side talks at DPC09 it was now time for front-end matters: Fronteers 2009. There’s no exaggeration in the description on the fronteers site: A stellar line up of speakers who are at the front of what’s happening in web-development. Generally speaking I really liked most of the talks and some of them pointed me to some interesting new techniques and…
PHP Exceptions can greatly assist in implementing various error scenario’s into an application. Before PHP5 one had to resort to specific return values or drastic measures like trigger_error() . Planning exceptions, I found out, is just as important as class design. At any point where a developer needs to handle the possibility of an exception being thrown he needs to know: What Exceptions can I…
In software development projects, paying proper attention to usability aspects, can greatly help ‘getting the message functionality across’. Usability is a field of expertise on its own and involves techniques like wireframes , prototyping and card sorting . Not every project is the same and (sadly) lack of time or budget can prevent specialized interaction designers to be involved in the project.…
Yesterday (sept. 29th) I went to the Freshheads office in Tilburg to attend the monthly PHPBenelux meeting. As it appeared it was right around the corner of the 013 venue so it was an easy find. Two talks were scheduled and Stefan Koopmanschap kicked off the meeting with a presentation titled “Integrating Symfony and Zend Framework” (slides). After a short introduction pointing out the benefits of…
When doing client-side developing there are times that jQuery’s get-this-do-that nature doesn’t provide all that is needed. For more complex applications I usually find myself creating javascript objects that ‘control’ a specific part of the page’s interaction. In the objects the application’s state is tracked, references to other objects (could be relevant DOM nodes) are stored and event handlers…
In a recent project I’ve been working on, we have used the ‘ Domain Model ‘ to describe and design our application. Doing so we decouple persistency logic from the objects that are being passed around and modified throughout our application: The Domain objects. So what in MVC is often referred to as ‘model’ is actually a combination of a persistency layer, a service layer and a Domain layer. The…
The biggest PHP event in Holland is over. Two great days have passed and it feels like it were just two hours. I didn’t attend the tutorial day so at friday after a brief intro by Cal Evans (with great cartoony visuals ) the event kicked off with the opening keynote by Andrei Zmievski. A talk about what makes PHP the language it is and about where PHP is heading with 5.3 and 6. It had humor,…