Recent content on Blog - MattBits
I am wanting to tackle an issue with best practice in AWS - IAM policies and resource policies. The immediate reaction would be to say: All IAM policies should be scoped to the actions and resources that they interact with All resource policies should be scoped to the principals that interact with them In a distributibuted micro-service architecture, where each application is deployed indepdently,…
Before I get into GitLab functions and components, I wanted to cover how I manage build and deployment artifacts, as it’s a dependency for a lot of what comes later. I generally block internet access across my homelab. Aside from specific external APIs, everything is either pre-downloaded or accessed through a caching proxy. I use Nexus for PyPI, apt, go etc. package pull-through caches. For…
Homelab history I’ve been running some form of home infrastructure for a long time now. Looking back through old notes and photos, it’s interesting seeing how much it’s changed. Some of it was planned, a lot of it wasn’t, and there were more than a few dead ends along the way. Like most long-running homelabs, it grew one project at a time rather than from any grand plan.…
Over the past couple of days, I’ve had a couple of things that have been annoyed me and have resulted in a learning and a success! Dependency OOMs I’ve been doing a bit more work on my virtual machine agent and had been trying to build it and began getting OOM errors. I followed some of the failures (it was a strong assumption that the library it was failing to build was the culprit,…
Having gotten an AMD AI 395+-based machine this month, I’ve been using Lemonade to run models, which takes care of bootstrapping RocM, I now wanted to bring this into a workflow I had been tested a couple of months ago which was toying around with making a game. My goal was to create a point-and-click game, but given that I’m awful at graphics, I had been taking photos and using AI to…
This is a summary of how I built and tuned a local OpenCode-style coding setup on AMD ROCm using Lemonade + llama.cpp, moving through multiple models, context issues, and performance tuning before settling on Gamma 26B A4B Instruct (IT). 1. Base Setup Started with Lemonade running llama.cpp on ROCm via Docker: docker run -d \ --name lemonade-server \ -p 13305:13305 \ -v…
The beginning I’ve been running a Hashicorp stack for a while now - Vault, Consul and Nomad across a small cluster of machines. Over time, I’d built up a pretty reasonable setup, but it was always a bit… scattered. Terraform here, manual config there, some docker-compose files scattered about. The thing is, I’m running all of this on self-hosted hardware. There’s no…
This is a pledge to myself and for any one else interested in following the same journey. I don’t know jack about Kubernetes. I know about docker, integrated with it’s API, cgroups etc. I’ve used nomad and docker swarm heavily and rancher. History I have taken steps in the past to master technologies in various levels, for example Terraform, nomad Terraform Used Terraform for…
My homelab and work life have followed very different trajectories, though often influencing one another. I like to try out interesting thought experiments at home, see how they work out to determine whether they’re worth investing in. Right now, my homelab consists of a load of VMs and my initial goal was to find a new way of monitoring (often these small tasks lead to a big spray of…
What is this? I work in close proximity to a lot of AI/ML/big data engineers and have friends who are of the same variety. I somewhat understand the basic concepts of a neural network and I have some knowledge of vectors etc. but I’m tired of being the one left confused when having conversations… So, my plan is to get stuck in a little.. at least to train something and to work my way…
Homelab storage Over the past 15 years, my homelab has seen a biiiig variety of iterations and solutions, in somewhat of an order: Random desktop machine with a hard drive First actual servers, with local hard drives Home-built NAS with iSCSI for VMs (ESX!) Random array of laptops with iPXE and NFS root drives (running out of money!) Real datacenter with iSCSI to NEXSAN SATABOY! Back home to iSCSI…
In my work life, I’ve moved away from working in a team and into an IC role. As a result, not only do I have a much longer list of tasks, but the scope and variety of those tasks has expanded significantly—more context switching, more stakeholders, more blockers, and more background work to keep track of. I’ve tried using various tools to manage this—Microsoft To Do, Jira boards, etc.—but they’re…
TL;DR: Profiling a Terraform provider revealed a performance bottleneck caused by repeated d.Get("task") calls inside a DiffSuppressFunc. The fix, implemented in PR #44543, introduces a singleTask flag to avoid repeated expensive calls, reducing plan time from ~15 minutes to seconds. A colleague told me she had issues deploying aws_appflow_flow, a Terraform resource for AWS. Every time she ran it,…
History Whilst at school and college, I loved running servers and bits and pieces. During this time, to host anything, I needed a domain name. Since I was also having fun minimicing services I had been working with during college (whilst working part-time at my old school), I decided to create more a brand-name rather than a personal-esque domain name. After various iterations (including maplehat,…
To kick this off, I’ve not had a good way to host binaries generated during CI/CD. At the moment, I have: Gitlab itself - I avoid using this for any binaries/OCI images mostly because the thing is bloaty enough and I do try to have complete backups without added hundreds of GBs of binaries to it. Sonartype Nexus - This is used for apt/yum caching and some scripted public binary mirroring…
For a while, I’ve been trying to get away from my previous VPN - OpenVPN. It was clunky and, somewhat, unreliable. I’d start it up, give it a username, password, OTP and it would connect after about 30 seconds. Then I’d leave my laptop for 2 seconds, the screen would go blank and as soon as I shake the mouse, OpenVPN would have disconnected, removed my routes and be asking for…
History In my homelab setup, the way services are run are not the simplest of approaches and I often find this bites me - but every challenge is an opening for learning, which is always great (unless I just want something to work). The challenge I’m talking about today is client identification with NAT outside of HTTP. What I mean is, in the world of HTTP, load balancers are often used.
The Problem! I’ve ocassionally found Terraform providers that take an attribute and manipulate it in a way that.. isn’t favourable. This means that, following runs see a change in the attribute (compared) to the attribute passed and want to modify it. For example!… The Docker Terraform provider (by kreuzwerker :D…
Dymamic Terraform Provider I wanted to create a proof-of-concept simple method of storing and retrieving data. User X --push--> Database --read--> User Y This would be used as a sort of dictionary for users to query information being published by others. The exact nature and source of the data made Terraform an obvious choice for this. I explored the most basic option: A basic boiler-plated http…
For Christmas last year, I wanted to make a present for my Mum. My parents had recently had solar panels installed with a battery. The system worked, so I was told, that the solar power would: primarily power the house; then charge the battery; once filled heat water for the hot water tank She wanted to be more energy efficient, but knowing when to use the dishwasher and when to have a shower…
Traefik’s ConsulCatalog plugin provides a defaultRule parameter, which is applied by default to exposed services. The example from the docs (https://doc.traefik.io/traefik/providers/consul-catalog/#defaultrule), suggests: For a given service, if no routing rule was defined by a tag, it is defined by this defaultRule instead. The defaultRule must be set to a valid Go template, and can include…
Whilst implementing basic end-to-end tests for, an open source Terraform-cloud alternative, Terrarun (https://github.com/matthewJohn/terrarun), I need to deploy the Hashicorp Terraform cloud agent. The agent (as well as the Terraform-cloud Terraform provider) require a trusted SSL certificate to correctly interact with the server. I’ve deployed various solutions in the past, such as…
Gitlab Pipeline Templates For a while, I have built custom pipelines for my Gitlab projects, each starting from scratch and facing the same issues. With the introduction of a new platform for running services, I needed to: Use custom versions of Terraform Inject CA certificates into each container Use a replacement docker registry, which now used authentication, provided by Vault Authenticate to…
Homelab monitoring I currently use Icinga2 (with Icinga Director) to monitoring internal homelab machines, services, as well as external VPSes and external services. This has worked reasonably well, but as I’ve been recently migrating to a new Hashicorp cluster, I’ve realised how manual the configuration is. I want to find a new solution that can perform this monitoring, but configured…
Pre-amble This blog post was written in-flight during a quest to create a secure deployment mechanism for Terraform projects to Vault, Consul and Nomad. The beginning portion was written whilst attempting to use a technique that ended up failing. Feel free to skip this portion and jump to “Using JWT authentication” Intro For the Hashicorp stack of my homelab, I have: Vault cluster…
Background I maintain a handful of open sources projects - most of which are of no interest to anyone. There are one or two, however, that have a small handful of users - and also a small number of contributors. Because of this, I spent a very inconsistent amount of time on each of these, occasionally fixing bugs and occasionally spending an hour or two each for a week to get a feature done.
AWS offers a broad variety of services - some of which are essential and unavoidable when using Amazon as a cloud provider. Others, on the other hand, provide solutions that, whilst on the surface are great as a quick start, do not scale. By scale, I don’t mean in the usual compute-power way, I mean in cost. If you use a service that provides scalability with a steady and predictable…
Goals I started off with a basic goal - host a small python website, which uses a MySQL-like database on a highly available cluster. After recently moving from a rack in a datacenter (and fortunately saving ~£250/month), I started looking at hosting some web applications on VPSs from a couple of providers. Of course, this means: Every instance costs money - I can no longer spin up 10 extra VMs…
Tinc appears to be one of the few open source mesh VPNs and, in my expierience, once working, performs incredible well. That said, the configuration of tinc is a little clunky and repetitive, nor does their documentation give much of a clue as to what is required for a minimal setup. Example The following example will connect 4 machines in a tinc mesh network; one machine has a direct internet…
Obligatory About Me Section My name’s Matt, I’m a senior systems engineer working at The Stepstone Group. I’ve been a full-time systems engineer since 2012, though worked part-time from 2010 and ran my first NT server in 2006/7 (though, of course, things swiftly moved to Linux ;) ). My full-time role entails mostly Devops work, working with AWS. But my home-life contains many…