Recent content on David - Musings of an SRE
While working on multi-homing my ASN network earlier to add Vultr as my secondary upstream, I ran into an issue where my traffic coming out of my Vultr BGP session was seen as coming from Vultr’s IP instead of my own static IP prefix.
This caused a bunch of problems with my secondary DNS servers for TheLittleHost, as they suddenly did not recognise my hidden master server because its IP…
After planning for this for a while, I finally found some time yesterday to finish hooking up my network to support multi-home.
One of the biggest obstacle that got me procrastinating was configuring the wireguard between my Home and my Datacenter. I needed to make sure that regardless of which direction traffic comes from, either one of the edge routers will be able to send traffic destined…
Just started to take a more visual approach to feeding my dopamine-starved brain by tracking my goals to keep me motivated each day.
Downloaded this Calendar Template, printed a bunch of them for this month and put them together in my clipboard.
Every time I perform a task each day, I would draw it on the calendar, and if I perform it consecutively enough (interval depends on the task), I…
Growing up, I’ve always felt like I needed something to dedicate my life to.
Maybe that’s where this obsession began.
I’ve always been drawn to stories about people who devote themselves completely to something.
Watching Japan Hour growing up, now aired on CNA in Singapore, has always inspired me. The show often features Japanese craftspeople working on what seem like…
Just last month, after at least 6 years of avoiding medication by convincing my different doctors that I am in the middle of working of lifestyle changes to bring my cholesterol down, I’ve decided to admit to myself that the charades is up.
I am not going to make meaningful progress bringing my cholesterol down. No amount of exercise and diet is going to significantly reduce it.…
Some notes of my own experience publishing my Obsidian Vault with Quartz and Fyra.
Context [!info]
Quartz is a static-site generator by Jacky Zhao similar to Hugo/Eleventy. Fyra is a simple zero-config CLI-based static site deployment platform tool. This guide will guide you through the process of publishing your Obisidian Vault onto the Internet within a couple of…
Some of my friends know this, I have an addiction.
I buy domain names. A lot of them, enough that my registrar actually paginates the list on its dashboard for me.
Not just for fun, but because I’m always thinking of ideas I want to build. And like most builders, I mostly never fully launch these ideas or websites. But there’s one thing I almost always set up first: email.
Now Playing on IRC
Back in the IRC days of the late 90s and early 2000s, chat was full of people sharing the songs they were playing on their media players.
I never quite understood how it worked. All I knew was that I had to install a plugin or script, and instantly the title of the MP3 I was listening to on Winamp would magically appear as text on IRC.
When I first saw videos of the Silverstone FLP02 ATX case unveiled at Computex 2025 early this year, I knew I just had to get it.
I have actually forgotten about it until about a week ago when I saw reviews of it appearing onto Youtube. This made me quickly search to see if it was available and if it was, whether my local PC parts retailers carried it.
Spoiler: They did.
My Desktop’s PSU recently caught fire while running a local LLM task and some intensive gaming. Luckily after quickly shutting down the computer, the fire appear to self-extinguish itself and caused no further damage other than a smokey room.
But this meant that I needed to get a new PSU.
I made a quick trip to Sim Lim Square, a retail mall in Singapore that was once dubbed as the…
The Problem That Kept Me Up at Night Picture this: you’re running a DNS service that handles over 100 queries per second. Every time you need to deploy an update or restart the service, you’re essentially playing Russian roulette with your users’ internet connectivity.
I learned this the hard way when we had a critical security patch that needed to go out. The traditional…
A friend recently asked me why I still buy old datacenter SSDs (some over 10 years old) instead of brand-new consumer ones. It’s a fair question worth unpacking.
While it is true that consumer SSDs use newer technologies and are usually comparatively cheaper, datacenter/enterprise SSDs are built differently with a typical write endurance of 2-10 DWPD, while consumer SSDs typically fall…
Running a ZFS boot pool for your Proxmox system can lead to unexpected boot issues if you do not copy the bootloaders after replacing a disk that failed.
When you use a ZFS pool for your boot disk, only the content of your boot disk is copied and your boot partitions are left untouched or uncreated.
To ensure that your replacement disk will still be able to boot up properly, you need to…
Moving my Dell R630 from my previous rack to a new rack in a new physical location was supposed to be a quick affair but it wouldn’t power on after the relocation. While the PSU and iDRAC port showed the power LED and iDRAC port lighted up after plugging the wires in, it wouldn’t boot despite repeatedly pressing the power button. Using network capture, this is how I managed to regain…
[Update 2026]: Ready to deploy that Hugo site? Skip the complex server setups. I built a CLI-first static site host called Fyra.sh. It’s a CLI-first static host running on my own bare metal in a proper Tier-3 datacenter. Just fyra push and your site is live. Free to start.
I’ve been trying to get DecapCMS working with my Hugo site locally without having to rely on netlify while…
If you’ve ever worked with Amazon SNS and subscribed an HTTP/HTTPS endpoint, you’ve probably hit this gotcha:
SNS sends you a JSON payload… but sets the Content-Type header to text/plain.
🤨 Why Is That a Problem? When SNS sends messages like SubscriptionConfirmation or Notification, the body is valid JSON. But since the header says text/plain, most web frameworks (like Rails,…
Running a beta is weird. On one hand, you desperately want to know when someone signs up. On the other hand, constantly checking Google Forms responses feels like watching water boil.
I’ve been collecting signups for Relaye.io using Google Forms for a while now. It does the job, but the lack of real-time notifications drove me nuts. Sure, I could get emails, but who doesn’t already…
Context I am currently working on a new simple email hosting product targeting solopreneurs and hobbyists and have been building out my own smtp server for this purpose.
These are just some of my notes.
Mail Headers When processing email coming into your smtp server, MAIL FROM, RCPT TO do not form the content of your incoming mail. Instead they are used by the receiving mail server just…
Some days, I find myself feeling deeply disillusioned about the things I’ve been building, questioning whether I’ve just been deluding myself into thinking they were useful. Other days, it feels like the tools I’ve created truly serve a purpose and have real value.
Today was one of those days where its more of the latter.
Just the other day, I had a HDD die in my TrueNAS but only realize…
In our recent trip to Shanghai, our family stayed at Hyatt Regency Wujiaochang throughout our stay and we enjoyed it. This is our review.
Location and Convenience Map of Hyatt Regency Wujiaochang Shanghai
View from hotel room
Located in a area known as Wujiaochang (五角场) or 5-sided market it is in a shopping district that has at least 5 malls within its vicinity which provides a lot of…
I was doing some keyword searching for Ping Everything Everywhere All At Once recently and found a few keywords that might be interesting to compete for SEO wise.
Once such keyword is “lol ping test” and when I went in and took a look at some of the top few results, I was impressed. Not only at some of the implementations, but also that people building such tools for more serious…
When working on a single or bi-directional GRPC stream, the client needs to manage retries themselves when a stream loses connection due to disconnects.
Unlike unary RPC connections which connects/retries on a per-RPC basis, streams once connected do not have a process for retries automatically.
To better manage retries, after some experiments, this is a pattern I landed on that can be…
These are my notes on how I migrated a user account from Cpanel to Directadmin (Reseller/Admin)
Prerequisite Access to your Cpanel Access to your DirectAdmin panel as an Admin/Reseller Backup your Cpanel In Cpanel, search for “Backup” and perform a Full Backup. Unfortunately there are many different skins for Cpanel so this is a very generic instruction. You are looking for a…
Background Kubernetes versions <v1.24 has been deprecated and removed from the official kubernetes pckage repositories.
Hence if you’re < v1.24, you will no longer have the support being able to do your upgrades with the help of the official package repositories.
You will need to upgrade your clusters manually until at least v1.24 before you can continue with the official…
The following might be useful for those who want to disable their systemd services from logging into /var/log/syslog but still maintain the ability to view the logs via journalctl -u <service>
Caveat: Unfortunately journal and syslog are quite coupled together, you can’t configure your specific systemd service to disable syslog logging without disabling it from journald via a…
Just the other day, I had a hardware failure on one of my servers. After fixing the main issue, I brought my VMs up.
But one of my VMs that hosted a development postgresql server would keep entering a crash loop. This can happen when there’s an unexpected or forced shutdown before postgresql can gracefully shutdown.
LOG: starting PostgreSQL 16.2 (Ubuntu 16.2-1.pgdg22.04+1) on…
Weekly Notes An even slower week then the previous week.
There hasn’t been much progress on the projects this week due to IRL life events. Also, I’ve been spending more time with some client work.
Hopefully I can get back to the grind this coming week!
🌈 Changelog NIL
🙀 Picks NIL
Weekly Notes Slower week this week. Spent more time trying to focus on a single project as well as doing some client work. Work has begun on some new features for Relaye.io, including new Outputs. Except to hear about it in the next few weeks!
A friend introduced me to Mom’s Test (in today’s Picks) which I can’t wait to start reading this week.
Lets keep grinding…
Weekly Notes Currently thinking of setting up a GeoDNS service so that I can serve my stateless sites like Ping Everything Everywhere All At Once’s API server closer to the request. This way I can reduce the latency global users are experiencing when accessing our site right now (based in Singapore)
This is a dilemma for me because then I would be embarking on a new side quest that takes…
Just the other day I found out one of my sites newsletters.love was having a bit of a slowdown.
As the site is hosted in Singapore, latency wise I couldn’t feel any issues, but it is apparently a big enough problem that someone mentioned it in a reply to my post on Hacker News
While I was disappointed that I didn’t catch this earlier, this gave me an opportunity to use Ping…
Weekly Notes This week was a relatively slower week as I was down with the flu. Spent most of the week on the bed. But managed to pick up pace towards the end of the week and made some new changes to Relaye.io!
For this coming week, I am looking to clean up and stablise relaye a little bit more and finish up with its landing page as well as spend some time adding a new free tier for Ping…
This guide is tested only up to Grafana 7.5
Background Since Grafana v7.0.0, PhantomJS support has been removed causing the image rendering of dashboard panel and alert image snapshots have been delegated out to [Grafana Image Renderer plugin] or remote rendering services.
In this guide, I am implementing a remote rendering service.
This is my consolidated notes on what you need to…
Weekly Notes This week, I have been toying with a new idea for a SaaS product targeted at solo entrepreneurs and small startup owners who want to quickly open up an email channel to receive feedback but don’t want to spend time (and money) configuring and setting up their email system.
This app would display all incoming emails belonging to the user’s domain name in a web inbox…
Just caught this article When privacy expires: How I got access to tons of sensitive citizen data after buying cheap domains during my morning lobste.rs read.
Reading this article makes me realize just how much I might have accidentally exposed myself by letting my domains expire. It’s a stark reminder that our digital actions can have significant consequences, even if we’re not…
I love how both Slack and Discord has a way to make use of the OAuth2 flow to directly retrieve webhooks urls that we can use to send out their messages.
This makes it really convenient to build integrations that would allow you to request for incoming webhooks on behalf of your users without having to take the extra step to process and retrieve incoming webhook links.
Discord For example,…
ICQ is closing, thank you for the memories.
Just caught the news about ICQ finally shutting down on 26 June.
Its goodbye to an era which has shaped quite a bit part of my online identity. As a tween in the 90s, ICQ was my very first instant messenger.
So many fond memories remembering friends’ ICQ UIN numbers. Heck, I’m surprised I still remember my ICQ number till today,…
As a developer who loves Vim’s efficiency and minimalist approach, I’ve often found myself in situations where I would use emojis to add some color and personality in either my blog posts, or when building outputs in my CLI tools.
Some emojis when they are displayed on vim, take up seemingly more character space that the others, causing vim to inteprete the line cursor a little…
Hello everyone!
Just wanted to say hi again. 👋
Its been quite a while since my last post and an even longer time since I’ve actively write about what’s going on in my life and I thought this might be a good time to re-introduce myself.
Who Am I 👋 Hi, I’m David 🌏 I live in the GMT+8 timezone 🎞️ I’ve been online since the early 90s as a young kid 💼 I’m…
When facing a “Too Many Open Files” error, it is normally a result of a process taking up too many file descriptors.
To diagnose this, find out the PID of the process which gave out the error.
You can do this with a:
$ ps auwx | grep <name_of_process> Verify open file limit per process 📝 Note: Each process that runs have a number of open files limit. By default, most of…
While migrating this blog from Jekyll to Hugo, I found I needed to replace all my old markdown posts which has my title params using single-quotes which Hugo does not appreciate, to double-quotes.
For example, I have the following across multiple files (over 200+ files)
--- title: 'abcdefg' tags: - new - stuff --- And I needed to replace title: 'abcdefgh' with title: "abcdefgh".
To do…
Hello 👋 Hi, I’m David 🌏 I live in the GMT+8 timezone 💼 I’m a 15 YoE backend, infra, SRE, currently taking a short sabbatical to try to build a business 🏡 I run my own Autonomous System (ASN) with most of my projects, running on servers hosted in my homelab 💗 I love everything relating tech infrastructure, cloud-native, building systems, devops, observability and engineer…
Hackerdrinks Friends Andy Emmanuel Zane
Sending your structs across the wire and receiving them on the other side. Using encoding/gob can help ensure your data structures can receive it on the other side.
// server.go package main import ... // Create your custom data struct type Message struct { ID string Data string } func main() { // for purpose of verbosity, I will be removing error handling from this // sample code server, err…
The signal pattern can be used to ensure your long-lived processes (ie. servers) are able to handle signals being sent to the process.
import ... type Handler struct { } func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { w.Write([]byte(`{}`)) } func main() { h := &Handler{} sigCh := make(chan os.Signal, 1) errCh := make(chan error, 1) signal.Notify(sigCh, syscall.SIGTERM,…
Openshift’s S2i (Source-2-image) is an alternative framework to buildpacks which uses Dockerfiles and Openshift to create applications out of reproducible container images from source.
Objective The goal is to be able to create a container image that contains the necessary application dependencies just by uploading the application code without having to write Dockerfiles.
Creating…
One of the coolest stuff I’ve picked up just today is that you can keep environment variables that you want to be loaded into every deployment pod in a neatly configured ConfigMap or Secret which gets injected back into the Pod during deploys.
Lets say you have a Secret that looks like:
apiVersion: v1 kind: Secret metadata: name: mysecret type: Opaque data: username: YWRtaW4=…
Short Answer, you can’t. BUT you can still declare it.
If you have two struct types which is embedded within one another but declared as an unexported type, you will not be able to declare it in a composite type. You can only declare it outside of an composite literal
Came across this while attempting to make use of Kubernetes’ API to declare an Deployment object.
type…
Using Vault’s Audit Backend to send logs to a remote Syslog server.
Objective Send audit_logs from Hashicorp’s Vault to an Graylog instance Prerequisite Setup a Syslog TCP/UDP Input on Graylog (if you’re using graylog) Has a remote syslog server running Steps Important Notice: Vault has a Syslog Audit Backend as part of its suite but it currently does not allow remote…
How to setup Prometheus AlertManager and get a whole alerting pipeline setup.
Objectives and Goals Write and Deploy Prometheus Alert Rules Configure Prometheus to send Alerts to Alert Manager Setup AlertManager to receive Prometheus Alert Send a Slack message on Alert Prerequisite Prometheus is already setup and running Alert Rules To begin writing and deploying alerts, you’ll need to…
When you run out of diskspaces and need to delete files quickly to recover them, most of the time, your deleted files will not free up the diskspace until the process that is using it is restarted or deleted.
To force the filesystem to free up the lock to the file so that the files can be cleared up, you’d need to find the process that is using the file and truncate it.