Posts

A lot has changed since my last “Homelab Architecture” post. No longer am I on 1 host, but 3 hosts with distributed storage using Ceph and (to a lesser degree) Longhorn. Even better, most of my services now run on K8s not for it’s high availability features but for it’s operational management; which is significantly more comfy than traditional VMs. I intended to write something after things were stable, and they have been for ~5-6 months as of this writing. ...

The stack Longhorn: Persistent storage for Kubernetes Velero: Backup tool for Kubernetes and Longhorn Uptime Kuma: Alerting for failed backup jobs Longhorn I’ve been using longhorn as an easy foray into K8s storage that provides some redundancy without the complexities of Ceph. I will soon dive into Ceph but Longhorn has less strict hardware requirements (e.g. can use virtual disks on VMs). Backups Many of my workloads have data I care to backup. I tried K10/Kasten but found it excessively complex and finicky. The last straw is being proprietary software which I cannot rely on (I avoid using proprietary software wherever possible). ...

Moving the blog to K8s My blog is pretty simple, running hugo with the content stored in a git repo. My go-to stack for many years has been a full Fedora VM on Proxmox using Podman, which is pretty wasteful for a tiny static site. I’ve been playing around with K8s for nearly a year getting comfortable with it and have finally decided to move a “production workload” to K8s to encourage treating it like I treat Proxmox workloads (“internal production”) with little downtime, backups, well-understood operational processes, etc. ...

I moved a VM to my NixOS workstation and used libvirt to run it, but ran into an issue a few weeks later after rebooting post-updates. Turns out that by default the virt-manager GUI will hardcode some paths for configuration items, and of course, when using NixOS these will change after updates. Example: <os firmware="efi"> <type arch="x86_64" machine="pc-q35-9.1">hvm</type> <firmware> <feature enabled="no" name="enrolled-keys"/> <feature enabled="yes" name="secure-boot"/> </firmware> <loader readonly="yes" secure="yes" type="pflash">/nix/store/ddf1vssb23g4jbdivhfgwi0iaimpz7my-qemu-9.1.3/share/qemu/edk2-x86_64-secure-code.fd</loader> <nvram template="/nix/store/ddf1vssb23g4jbdivhfgwi0iaimpz7my-qemu-9.1.3/share/qemu/edk2-i386-vars.fd">/var/lib/libvirt/qemu/nvram/vmname_VARS.fd</nvram> <boot dev="hd"/> </os> I wish I had a better fix- but this seems to work: ...

I found it a bit fiddly to get a Xbox controller I had used for years on Fedora working on NixOS. I recall all sorts of incantations to get it working on Fedora and thankfully it was a bit easier on NixOS, but I had to piece together some reddit posts and other content to get it working. TL;DR Add this to your configuration: hardware.steam-hardware.enable = true; hardware.xone.enable = true; environment.systemPackages = with pkgs; [ linuxKernel.packages.linux_zen.xone ]; Reboot Plug in adapter Hold sync on controller Hold sync on adapter That seemed to do the trick to get it working on Steam. ...

I started this post in April 2024 but never committed it to the repository. I’ve recently dug into K8s and am strongly considering migrating my lab over- so I figured I should finish this up as gitops upgrades for virtual machines has been reducing time I spend on updates for about 1 year. Reliability A core principle in my lab has always been reliability. Many design choices may appear outdated, inefficient, expensive, or bespoke, but are intentional in my desire to maintain reliability and simplicity in updates, backup restoration, availability, and ongoing maintenance. There are a single points of failure (many, actually) but fixing them is trivial save for a full non-disk hardware failure, which I hope to better solve with K8s and ceph (or OpenEBS- TBD). ...

Landing Headfirst The foray into NixOS was painful, and while there wasn’t much I found for newcomer guidance here’s what would have helped me: Start small, use the installer on a real system or VM to build on Said differently, don’t start with a beefy configuration that will inevitably require troubleshooting Yes, this is obvious advice, but I could not resist the allure of creating a “perfect” configuration As of 2023-11, the installer did not play well with LUKS It is worth learning how to manually partition disks properly Make /boot large (5GB+), or don’t put it on a dedicated partition I ended up needing to wipe and re-partition due to a full boot partition; a well documented issue that shockingly isn’t resolved in the installer When using -p to name generations, don’t use spaces nixos-rebuild will happily give you the rope, invalid generations won’t be cleaned out by garbage collection and you’ll find yourself unable to upgrade with a full /boot The following are a bit tongue-in-cheek that I have yet to do them as of this rambling: ...

I recently had a painful migration from Fedora -> Debian (temporary) -> NixOS on my primary workstation. My git/Ansible workflow is currently executed locally; it turns out Ansible/Python/NixOS can be a painful combination. There are a few others with similar issues. PEBCAK is not entirely ruled out in this scenario. From what I could tell, the core issue was getting Ansible to recognize the Python binary who could import the correct packages (installed via Nix config). ...

Background I run 2 Proxmox hosts, one being a primary and the secondary being a Proxmox Backup Server target. As is too common the primary is far different from the secondary including the boot configuration, which was the affected component in the latest Proxmox major upgrade (v7 to v8). The situation was further exacerbated by a “temporary” implementation of virtualized OPNsense after a failure of pfsense hardware in 2021. This makes the primary Proxmox host a rather juicy single point of failure. ...

16 data disks and counting I have a lot of data- the 12 3.5" slots on the storage primary host are all filled with the exception of a cold spare and a 20TB disk I connect quarterly to update the offline backup. Most of the data is fairly unique- personal pictures/videos, backups of old websites, images of prior computers (one day I’ll virtualize them for fun), tons of (actual) Linux ISOs (I keep a copy of almost all I use), and more. I’m a frequent visitor to the Internet Archive and like to keep personal copies of things. ...