Recent blog posts on blog.domainmess.org
This year I also attented the EOFS Workshop on Open Source Parallel Filesystems. I wrote about the last one already here on my blog and it was always a pleasure to be there. This year the EOFS organized the workshop in Paris; the first time outside of Germany. In this blogpost I’m gonna summarize the parts the I found interesting the most: GekkoFS GekkoFS is a fast on demand distributed…
When working with networks types other than Ethernet bonding is often not supported or just with a reduced feature set. With eBPF one can reimplement simple distribution of outgoing traffic. When I administered HPC clusters we had Linux machines working as routers between multiple high performance networks. In this particular case, they were routing traffic from multiple hundreds of compute nodes…
Since I work in maintaining a smaller HPC cluster at the moment I had the chance to attend the ISC Conference for the first time. Here is the summary of my notes during the conference and exhibition. The event overall Overall, I found the exhibition booths to be the most interesting part of the event. They provided a great opportunity to connect effortlessly with various manufacturers and…
Who can truly say that they never wished for a way to write files to two places at the same time?! No I’m not talking about setting up a complete Ceph cluster or, god forbids, DRDB. Just something simple that double the writes to two different folders. I stumbled over this problem again a few weeks ago when I was trying to write config files to a distributed filesystem but wanted to always…
In 2024 I first attened the EOFS workshop about opensource parallel filesystems. The EOFS organizes this meeting once a year with different partners from german Univeristies to gather people and talk about the current status of open, as in non-proprietary, filesystems. I found this mini conference quite interesting and inspiring and it was one of the reasons I ended up working in HPC. So I’m…
Just a collection of commands an config snippets I use to work with nixos. Start a shell and install package from local nixpkgs repo: nix-shell -I nixpkgs=./nixpkgs/ -p package1 package2 Hints: https://www.mankier.com/1/nix-shell Create a container from a configuration.nix sudo nixos-container create container-name --config-file /path/configuration.nix Experimental nixos features in nixos-rebuild…
Instead of running through a installer everytime you setup a PC you can create a bootable image with mkosi After my first post about mkosi where I introduced the tool, I tried to use it as often as I could to easy the pain of spontanious image creation. One thing I really don’t like todo is to run an OS installer to setup a PC. Especially if it is a machine which won’t use without…
Through my work at Cosmic Cable Ltd. I had some experience with old analog & rotary telephones. Over the time my partner and I wished for some W48 or W49 telephones at home for internal calls. This blogpost will follow the different stages of this setup as (read if) it progresses. The plan The initial idea was to have 2 telephones one in the living room in the upper floor, the other one somewhere…
I always wanted to get hands-on experience with eBPF as it seems to be an exciting technology. A friend of mine recommended me the BCC Toolchain when I investigated some TCP socket issues and I was astonished how rich the possibilities are and how easy the code is compared to writing kernel patches. When I was working on a test setup for a storage solution, we discovered that the disks images for…
“Use less NAT” is a sentence I really like to hear. For a customer project we should build a high performace server for a webapplication. One of the requirements was that the ingress connections should not go though a loadbalancer or a NAT. Although that the throughput would probably not be throttled by those techniques and of course the application was really old Python software, but…
During a search to make a Linux application suitable for rolling upgrades I search for a way to overtake a already bound TCP port. I realized that some versions of netcat are able to do that and also worked when I was using Dockers port forwarding feature. So I started to investigate how this works and what one could make of it. SO_REUSEPORT Apparently there is a kernel feature in Linux since…
By chance I found the Blog and Videos of Tanel Poder. The content is excellent if you want to get into low-level I/O debugging on Linux, I especially recommend the following blogpost with its corresponding video: https://tanelpoder.com/posts/high-performance-block-io-on-linux/ https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation/ Inspired by the detail of his…
In Octobre 2023 I attended the LAD23 in Bordeaux: the Lustre Administration and Development workshop. I was working on a ~15 PB (raw capacity) storage system for a University which was based on the Lustre filesystem and my colleague and I wanted to get some information since we both didn’t any experience with Lustre yet. It was also my first conference in person since the FOSDEM20.…
Yesterday a friend ask me if it is possible to route the outgoing traffic of a Linux machine with a specific destination port via a wireguard VPN. He used a publicly accessible proxy host to forward SMTP (port TCP/25) via this VPN to a machine in his home network and now he wanted the mailserver to push all the outgoing mail also though this VPN connection to the proxy. However there should no…
When it comes to make code or config publicy available as open source one has always to make sure that the repo doesn’t contain any sensitve information. To remove stuff like passwords from various files in all commits i use bfg. First I clone a single branch from a local repo which should be adjusted for public. Although I use a seperate branch I also use a seperate git directory because…
Together with a friend, I found a old loudspeaker box in the storage of the CCC which contained the material of the Blinkenlights project. Both of us took one of them since they are beautiful and have the perfect format for a bluetooth sound box and include a nice rotary control element for volume. My friend guessed these speakers where used as a monitor system in dressing rooms and he started…
In my new apartment I luckily have a fibre (FTTH) which terminates in the living room. My first excitement was curbed when I investigated about the technology in use. The ISP uses a technique called GPON which is probably smart from a economical point of view but also enables some possible (sniffing) attacks. Additionally the provider seems to do some filtering which is not just DNS blocking. Long…
In my current job we use Proxmox clusters as VM hosts and a lot of Debian VM on them. Most of the VMs are old and handcrafted but for some of them like Jenkins agents and Gitlab runners I created Ansible playbooks to configure them. Of course this is rather useless if there is no way to automatically spawn the whole VM before configuring it and so I started to create a role which creates the VM if…
In 2016 and 2022 I participated in a project called “Cosmic Cable Ldt.”, a fictional company which provides telephone booths at Fusion Festival. This was one of the most varied and beautiful projects I’ve ever done as it combines work with metal and wood, art and decoration with a technical stack from old dial phones to modern VoIP networking. As I love doing all of this at once,…
Installing (Linux) operating systems is a time consuming and, even worse, repetitive task. The whole “burning to disk” era is definitely over and bootable USB sticks are useful in some cases (eg. installing a new workstation) but let’s be honest, most of the time one installs an OS to be used in some kind of VM or container. For this use case live systems with crude installer…
To work without external display I wanted to have a laptop stand which needs less space on the table than the laptop itself. Inspired by DIY notebook stands like this one made from steel pipes this one from cork cork or this one made of wood, I wanted to build one too. Of course when the stand should be smaller than the notebook on top of it, the barycenter has to be placed well to prevent…
Sharing data between VMs is a headache every admin had once in a while. Especially when multiple virtual machines are expected to have read/write access. Besides the oldfashioned ways like NFS and the new shiny stuff like Ceph, the ‘simple’ sharing of a filesystem from host to VM fits in. Although some solution can configure this with one click, I wanted to understand with which…
A while ago I realised, that some of my ubuntu systems, most of them nspawn containers, don’t receive any updates. I didn’t really care about the containers but as I realised that one of my internet facing host is also affected, I began to search. I searched on https://packages.ubuntu.com/ for the newest kernel and locally with apt-cache policy linux-generic for the newest available on…
I had two Arch Linux machines in short time where pacmans database was deleted by accident (it resides under /var/lib/pacman/). Afterwards pacman doesn’t know about any installed packages, which I realized when no updates where available ;-) To verify this situation you can use pacman -Qe to list all installed packages. If you lost the database, of course your system continues working but…
In some setups, mostly big storage arrays, SATA drives are prevented from spinning up automatically. There are many standards and lots of technologies to controll this behaviour. Unfortunately, I learned about all of them. A few years ago I got a new home server which just did not work as I expected. This article will follow the different topics I learned about during this endeavour. I try to keep…
PXE boot concepts are often complicated chains of many stages. Init ramdisks contaning casper 1.445 can load a ISO over http as rootfs. Most Linux distros offer an ISO file which could either be burned on CD or copied to an USB stick. Although [grub2] can handle such ISO files directly, SYSLINUX can’t and it’s part PXELINUX still used in a lot of PXE environments. The idea behind PXE…
Excel files can protect sheets/cells to prevent accidental editing. This protection can be “secured” by a password. This article describes how to remove that protection with basic Linux tools. A friend of mine called me and ask if I know how to remove the write protection of a Excel file. No, I’ve never heard of such a thing so she explained that she was using a template form…
Postgres can get in a sitation where there are corrupt datasets in the DB which are neither readable nor deleteable. This article describes how to identify such blocks and zero them with basic linux utils At work I found a very slow and rather huge Postgres database. Over 750GB of Bareos backup information with such incredible slow I/O that the jobs sometime timeout. So I started to clean the…
A few times I had the need to build Arch Linux AUR packages which took a long time on my notebook. I had access to some faster hosts but they were all running Ubuntu (16.04 and later). To keep this in mind I write down how to create a Arch Linux build chroot inside an Ubuntu host. Bootstrapping Arch Linux First we need to get the Arch Linux base filesystem and extract it to a directory. A few…
We disassembled a large equipment cabinet in our storehouse to use its space and I got a lot of pneumatic gear out of it. Since I don’t have any kind of compressed air systems at home I did what I do most: I used it to build a desk lamp. I removed all the filter stuff from the chassis to conduct a USB cable through and screwed a IKEA LED light arm on top. We disassembled a large equipment…
I had another air filter with a barometer left over. Again I took all the filter things out of it to make room for a cable coming in on one of the intakes and I added the arm of a floor lamp on the top of the filter. For this one I ordered a 12V warm white 1W LED and put it on a heatsink with thermal glue. The heatsink just fit into the round chassis on the end of the arm and is hold in place by a…
The story of struggeling with the always wrong DNS server on my laptop and the behaviour of systemd-resolved with static and dhcp announced DNS servers. This topic bugged me for a very long time! In Germany there a quite a lot ISPs which censor specific content either due law enforcements or to assert their own interests. Since years I wanted to use DNS Servers which I chose myself so I nailed…
A desk lamp made from an old alternator and an industrial shade After my wisdom teeth have been drawed I had two times two weeks of being at home, half the day on painkillers *_* First I wanted to build a light for bringing my balcony plants through the winter so I decided to go through my metal and electronic trash in search for something I could use to make a full lamp out of an old industrial…
LLMNR is a rather easy yet unknown protocol to resolve hostnames on a local link. On most modern Linux systems it can be used with systemd-resolved. This article describes the basic components and configs When I use containers, I don’t want to fumble around with IPs. Especially not when they change each time a container is created from scratch. Of course Docker has its way to address this…
This year I visited Brussels with some colleagues to attend the FOSDEM Conference for the first time https://fosdem.org Conference and Organisation The conference itself takes place in the Université Liberal Brussels (ULB) which is a campus of more or less rotten buildings and can be visited without a ticket or fee. There are many many talks over two days which average on about 30 minutes and are…
I decided to use my second APU2c4 board as my new Nextcloud/Mailserver. Since this thing should be reliable, especially reboot safe because I have to travel a few hundred kilometers to get hardware access in the datacenter, I decided to use Ubuntu as OS. Sadly the Ubuntu standard installer ISO is not configured to be used only via serial console. Although there are some hints in the web how to…
Preamble This is a classic one. I run my own mailserver since I’m legally allowed to rent rootservers and twice a decade this stuff had to be rebuilt somehow. I ran a setup with postfix + dbmail + spamassasin for a long time but since dbmail is completly dead and it’s a hassle to get the share libraries straight on anything newer that Ubuntu 14.04 I must switch to something more…
“I’m a huge fan” With this words on it this totally broken Bladecenter fan was lying for something like a year in our office. Well actually I’m not a fan of Bladecenters at all so I decided to make something useful out of this “I’m a huge fan” With this words on it this totally broken Bladecenter fan was lying for something like a year in our office. Well…
Once I bought a Lenovo Thinkpad Edge, can’t remember the exact number, but I selected the small CPU and it was so slow, that I smashed it one day so that the display was broken. What a relief! Afterwards my flatmate gave me a Dell Latitude E4300 which I used approximately from 2014 to 2018 and besides it was also very old I was very happy with it. This thing was unbreakable and a good…
Storage server with APU2c4 Board, btrfs and systemd-nspawn Homeserver. Standard. Something with a bit of space for backups, entertainment and for sure to be a seedbox. I decided to build on based on an apu2c4 board cause they don’t need much power and have good reliability. Sadly the have only one SATA port so another an additional controller is needed to get a RAID running. Plan was to get…
Introducing netjail Long time ago I felt the need to force a single application to use a specific network connection. Or to be more specific I needed to ensure that my torrent client cannot escape the VPN connection. Today the internet is full of scripts and tools to realize that under any kind of OS and many people just use container virtualization like docker or LXC. Anyway I started to write a…
A friend of mine gave me the case of an old HP Gen8 Microserver, chassis only without the board or power supply. Since the HDD cage has no real backplane but a SAS (SFF-8087) Plug directly attached to the HDD Slots and I didn’t have SAS controller at that time, the case wasn’t useful for me. Cause I want to install 19" rackmountable hardware only at home I finally decided to do…
I usually use systemd user services to get some simple tasks done on my workstations like backups of my homefolder or changing my wallpaper any hour. For some of them I want to get some notifications if they succeed or fail and since I don’t use some kind of Desktop Environment like GNOME I had to choose the components myself. Right now I glued together X.org, LightDM and i3 windowmanager…
After using Networkmanager and Wicd for a few years each I craved for a less complicated networking setup on my laptop. I did some research and found that nothing out there fits my need so I decided just to use the minimal basic setup. systemd_networkd I created a simple network file to use systemd to do the DHCP: [Match] Name=wlan0 [Network] DHCP=yes [DHCP] RouteMetric=10 The Name tells systemd…
blog.domainmess.org This will be a random collection of stuff I do or should not do or already did. I just felt the need for some form of publication which I can control myself without depending on any kind of company. This is a static page proudly served with nginx created with hugo and nofancy theme. blog.domainmess.org This will be a random collection of stuff I do or should not do or already…