Prometheus does not need to be hugely complicated, nor a massive resource hog, provided you follow some principles. Background My last #prometheus posts have been exclusively about large scale production setups, and the difficulties this pulls in. I would like to argue that these difficulties are often self-imposed, and a combination result of inadequate cardinality control and induced demand .…
To aid a memory obese prometheus , I recently helped in attempting to slowly shift a cluster over to prometheus agent mode sending data to thanos receive over the last couple of months. I have now personally given up on this goal due to a variety of reasons, and this post explores why. Background Setup The original setup we started out with is basically this (via 2022 ecosystem post ): and we were…
Unusual helpers for cluster level resource debugging.
The year is coming to a close, and as a small amount of self-reflection of my relatively fortunate and stable situation, here's a list of things in life that made me happy. Video In general, I like well presented video exposés, impressive video game footage, and watch competitions from everything from piano competitions to SC2 esports tournaments. Some frequented returns:…
As time goes by, I find myself increasingly disinterested in actually travelling out to a convention, when my preferred method of consuming conference talks is overwhelmingly VOD form with 2x/FF potential. This is doubly so when the convention is increasingly corporate (like KubeCon), and the CNCF youtube channel is in general high quality. Get it before your ad-blockers break. This post…
As a former Quake player, I have been a long-term user of guake for a quick access terminal on Linux. However, as guake is gnome specific, continued use became problematic once a mac got thrust upon me for work. So herein is a workaround for making your own drop-down terminal for Linux and Mac by utilizing alacritty + zellij . Looks Behold; a semi-transparent, tabbed terminal hovering above two…
In an attempt to torture my chair shaped husk of a body, I somehow ran almost 2500km this year. Mostly to see what would happen, but also to take part in the Royal Parks Half Marathon in London . Milestones In the process of going up to 70km weeks - something which became routine remarkably quickly - I ended up beating pretty much all of my endurance records: 41:51 10k 39:29 10k 11:15 3k 1:29:34…
Round two of escapism this year. This is a post for myself with various information as a conclusion to my recent fixation into designing an efficient and compact factorio megabase . Plan I keep coming back to this game about once a year. There is something very soothing about the type of play this game encourages. Playing factorio basically feels like being part of some parallel software project;…
In a brief bout of escapism from the world and responsibilities, I booted up Baldur's Gate 2 with my brother. It's an amazing game, once you have figured out how to roll your character. For today's installment; rather than telling you about the game, let's talk about the maths behind rolling a 2e character for BG2 , and then running simulations with weird X -based linux tools.…
As part of my work life in the past year, a chunk of my day-to-day life has consisted of maintaining a prometheus installation on top of a sizable kubernetes cluster. My original feeling was "this is not that bad with kube-prometheus-stack ", but this sentiment has worsened somewhat with the realisation that more and more customizations and pieces were needed for large scale use. Half a year later…
After almost 4 years of running D&D almost every week, my big campaign concluded recently. This week marked our retrospective, and the release of several relics of the campaign. One in particular was the campaign repo - previously described as my foaming campaign brain . This is a brief post about the setup, and discovered pros/cons associated with it. Campaign Campaign first. We held 126…
First KubeCon in a while I haven't done anything for (didn't even buy an ticket). This post is largely for myself, but thought I'd put some thoughts here public. All talks referenced were recently published on the CNCF youtube channel , and the posts here are really just my notes (make of them what you will). My interest areas this kubecon fall broadly into these categories;…
After a quarter year of extensive improvements to kube , it's time to take a birds-eye view of what we got, and showcase some of the recent improvements. After all, it's been about 40 kube releases , one major version of tokio , one extremely prolific new contributor , and one kubecon talk since my (very outdated) last blog post . Crates As of 0.51.0 , With modules and crates now…
In 2018, we introduced the explosive Cube of Force into our home campaign. We really did not expect it to cause us/me that much grief at the time, but despite long canonicalisation meetings, our interpretation was not even internally consistent. At the time we dealt with this by nerfing Cube down into the ground gradually rolling out alternatives, but it's time we dug into the concept of…
After 2 years of running a D&D campaign almost every week, my note taking setup reached several breaking points. If you're using OneNote or another online managed system for tracking notes/cities/npcs/pcs/events, but know how to use programmers tools like git and code ; boy are there a world of advantages available to you. This is a story of my original note talking setup,…
Despite having been a lazy Cinnamon customer since the Linux Mint days, recently some interesting benchmarks started surfacing about input lag in various Window Mangagers, and this made me want to experiment a little. This is a log of stuff I needed to tweak to get LXDE working well. LXDE LXDE is a free desktop environment with comparatively low resource requirements. This makes it especially…
It's been about a month since we released kube , a new rust client library for kubernetes. We covered the initial release, but it was full of naive optimism and uncertainty. Would the generic setup work with native objects? How far would it extend? Non-standard objects? Patch handling? Event handling? Surely, it'd be a fools errand to write an entire client library? With the last 0.10.0…
When interacting with kubernetes it's generally been standard practice to use either client-go via go, or kubectl via shell. While these are good, non-controversial choices, the advancement of client libraries, and smarter openapi bindings, combined with the generics and procedural macros of rust-lang , it's now quite possible to write fully fledged kube operators, using slim rust kube…
Authenticating with large kubernetes clusters often risks you dealing with complicated provider logic and sometimes policies outside your control. While controllers and operators authenticate with service accounts directly, this is only true inside the cluster. That is, unless you can impersonate the service account from outside. EDIT: as of Kubernetes 1.24 the bypass method no longer works .…
Fighting a CR 13 Vampire at level 7 in total darkness is no easy feat. As part of an emergency backup adventure ( EBA ) in the middle of a long running D&D campaign, our largely chaotic party of misfits (save for our low int paladin whose deity has yet to be named in his backstory) was given the poem that leads them into (and helps them navigate) White Plume Mountain : It's a classic dungeon,…
The now defunct unicorn startup babylon health needed to micrate about 50 microservices to Kubernetes in early 2018. At Kubernetes 1.8, supporting tooling was weak, but the company pace was fast. This is an historically updated post about shipcat , a standardisation tool written to control the declarative format and lifecycle of every microservice, and get safety quickly. This article was updated…
QWERTY sucks. You always learn it wrong, it's optimized for not jamming your typewriters, and it's got a bunch of history with shortcut keys. My brother started using DVORAK (a similarly unconventional layout) which aims to reduce the number of awkward finger movements using analysis of language and layout experimentations. My problem with DVORAK is that they moved important shortcut…
A particularly tricky problem with tournament scheduling that made it into tournament/duel . This turned out to take a little more mental effort than initially expected and is documented herein for historical reasons. The problem of how to order seeds into the tournament brackets seemed almost a bit random. But after some hours on the couch with pen and paper and ivc to talk to, we found a…
An easter egg and puzzle solution for Dungeon Siege 2 Broken World . After doing the first two rooms of the Treasure Hunt quest, you can attempt two additional puzzles of the same type, but these are ridiculously hard. If you made the second one, you have no doubt noticed that these are much trickier than the general lightning reflection puzzle in the original DS2. The third is manageable with a…