Proxmox and code reviews
How to corrupt cluster configuration without doing anything. When a data consistency related bug goes undiscovered for well over a decade, it's time for a second look at code review practices.
free-pmx: On Proxmox free software
A compendium of independent insights, guides and solutions for free users of Proxmox suite.
Latest posts
The pitfalls of some post-install scripts
Be aware of security implications that come with third-party setup tools, such as those supplied by the wider community.
Proxmox VE Live System build
Build a live system that boots the same kernel and provides necessary compatible tooling as a regular install - with a compact footprint. Use it as a rescue system, custom installer springboard and much more - including running full PVE node disk-less.
Fragile Proxmox cluster management
Proxmox Cluster filesystem builds on Corosync, but there are oddities in the PVE configuration. The support for cluster management is imperfect and can bring inexplicable situations which deserve a better user approach.
SSH certificates setup
PKI SSH setups for complex clusters or virtual guests should be a norm, one which improves security, but also manageability. With a scripted setup, automated key rotations come as a bonus.
Public Key Infrastructure with Secure Shell
Secure your SSH infrastructure from the very first boot. Rotate keys and never trust a previously unknown machine. Never pass through a key-not-known prompt and do not get used to the identification-changed warning with a remote host.
ERROR: dpkg processing archive during apt install
Conflicts in files as packaged by Proxmox and what finds its way into underlying Debian install do arise. Pass proper options to the apt command for remedy.
ZFSBootMenu setup for Proxmox VE
A complete feature-set bootloader for ZFS on root install. It allows booting off multiple datasets, selecting kernels, creating snapshots and clones, rollbacks and much more - as much as a rescue system would.
Taking advantage of ZFS on root
A look at limited support of ZFS by Proxmox VE stock install. A primer on ZFS basics insofar ZFS as a root filesystem setups - snapshots and clones, with examples. Preparation for ZFS bootloader install with offline backups all-in-one guide.
Restore Proxmox VE host from backup
Restore a full root filesystem of a backed up Proxmox node - use case with ZFS as an example - simple to adjust for other, less complex setups. Standard Debian tooling: tar, sgdisk and chroot.
Rescue boot Proxmox VE host
Access PVE host root filesystem when booting off Proxmox installer ISO. A non-intuitive case of ZFS install not supported by regular Live Debian.
Backup entire Proxmox VE host
Fast full host backup (no guests) demonstration resulting in 1G archive that is sent out over SSH. This will allow for flexible redeployment in a follow-up guide. Only regular Debian tooling.
Getting rid of systemd-boot
Ditch the unexpected bootloader from ZFS install on a UEFI system without SecureBoot. Replace it with the more common GRUB and remove superfluous BIOS boot partition.
Making sense of Proxmox bootloaders
What is the bootloader setup determined by and why? What is the role of the Proxmox boot tool? Explore the quirks behind the approach of supporting everything.
Why Proxmox offer full feature set for free
Everything has its cost. Running off repositories that only went through limited internal testing takes its toll on the user. Be aware of the implications.
A glimpse at Proxmox Quality Assurance
What kind of testing procedures do they use at Proxmox and how does your bug-reporting fit into it? How consistent and thorough is regression testing before users get hold of a public package?
Proxmox software repositories
What exactly do you get from the non-subscriber and freely available package repository? And what are the differences to the one meant for public testing only. Almost none.
Upgrades and repositories
Set up necessary APT repositories upon fresh Proxmox VE install without any subscription license. Explainer on apt, apt-get, upgrade, dist-upgrade and full-upgrade.
How Proxmox VE shreds your SSDs
Debug-level look at what exactly is wrong with the crucial component of every single Proxmox node, including non-clustered ones. History of regressions tracked to decisions made during increase of size limits.
The pmxcfs mountpoint of /etc/pve
Understand the setup of virtual filesystem that holds cluster-wide configurations and has a not-so-usual behaviour - unlike any other regular filesystem.
Some lesser known quorum options
When considering a Quorum Device for small clusters, be aware of other valid alternatives that were taken off the list only due to High Availability stack concerns.
The Proxmox cluster filesystem build
The bespoke filesystem that is the heart of Proxmox stack compiles from its sources in C. Necessary when changing hardcoded defaults or debugging unexplained quirks.
The Proxmox cluster probe
Experimental setup that can in fact serve as a probe to the health of a cluster. Unlike e.g. Quorum Device, it mimics an actual full fledged node without the hardware or architecture requirements.
WARNING: Setting locale failed
Common Perl warning during upgrades regarding locale settings lies in AcceptEnv directive of SSH config. A better default for any Proxmox VE install, or any Debian-based server in fact.
Container shell with no password
Do not set passwords on container users, get shell with native LXC tooling taking advantage of the host authentication. Reduce attack surfaces of exposed services.
The Proxmox Corosync fallacy
Distinguish the role of Corosync in Proxmox clusters from the rest of the stack and appreciate the actual reasons behind unexpected reboots or failed quorums.
How to disable HA permanently
Do away with the High Availability stack in a safe manner. Avoid the risk of reboots when troubleshooting suspected hardware woes. Particularly useful for frequent loss-of-quorum scenarios.
How to disable HA for maintenance
Avoid unexpected non-suspect node reboot during maintenance in any High Availability cluster. No need to wait for any grace periods until it becomes inactive by itself, no uncertainties.
Improved SSH with hidden regressions
Over 10 years old bug finally got fixed. What changes did it bring and what undocumented regressions to expect? How to check your current install and whether it is affected?
Turning a cluster member into a standalone node
Making a node that was once part of a cluster standalone again can be counter-intuitive compared to simply removing nodes from list of cluster members.
Why Proxmox VE shreds your SSDs
Quantify the idle writes of every single Proxmox node that contribute to premature failure of some SSDs despite their high declared endurance.
ERROR: failed to load local private key
Misleading error message during failed boot-up of a cluster node that can send you chasing a red herring. Recognise it and rectify the actual underlying issue.
Verbose boot with GRUB
Most PVE boots are entirely quiet. Avoid issues with troubleshooting non-booting system later by setting verbose boots. If you are already in trouble, there is a remedy as well.
Extract Cluster configuration - /etc/pve
Recover individual cluster configuration without need for a bogus node. Both from previous backup or a corrupt database. The source does not need to originate from the same cluster.
DHCP setup of a single node
Set up your sole node Proxmox VE install as any other server - with DHCP assigned IP address. Useful when IPs are managed as static reservations or dynamic environments. No pesky scripting involved.
Passwordless SSH can lock you out
Not so intuitive cascade of events may lead to inability to SSH connect to a node with otherwise healthy networking setup due to inaccessible keys location.
DHCP setup of a cluster
Keep control of the entire cluster pool of IPs from your networking plane. Avoid potential IP conflicts and streamline automated deployments with DHCP managed, albeit statically reserved assignments.
Backup Cluster configuration - /etc/pve
Backup cluster-wide configuration virtual filesystem in a safe manner, plan for disaster recovery for the case of corrupt database. A situation more common than anticipated.
The Proxmox time bomb watchdog
The unexpected reboot you have encountered might have had nothing to do with any hardware problem. Details on specific Proxmox watchdog setup missing from official documentation.
Proxmox subscription nag removal
What exactly brings up the 'No valid subscription' notice and how to eliminate it, including with automation. No obscure regular expressions that might corrupt the system in the future.