A lot has changed since my last “Homelab Architecture” post. No longer am I on 1 host, but 3 hosts with distributed storage using Ceph and (to a lesser degree) Longhorn. Even better, most of my services now run on K8s not for it’s high availability features but for it’s operational management; which is significantly more comfy than traditional VMs. I intended to write something after things were stable, and they have been for ~5-6 months as of this writing.
The stack
- 3x Dell Optiplex 7000 SFF desktops
- Intel SSDSC2BX800G4 800GB SSDs
- WD_BLACK SN850X 1TB NVMe SSDs
- 10gb NICs
- 1gb NICs
Software
- Hypervisor: Proxmox
- Storage: Ceph
- Storage: Longhorn
Connecting it all together
- 10gb NICs: Ceph and Proxmox cluster traffic over a Ubiquiti switch
- 1gb NICs: application traffic
- SSDSC2BX800G4: Ceph OSDs
- SN850X: Proxmox boot disk & Longhorn volumes
Reaping the value
It’s a significant jump from the old:
1 beefy Dell PowerEdge with a bunch of mirrored ZFS pools
To the new:
3x Dell Optiplex desktops (easy to procure) in a cluster with replicated storage primarily on Kubernetes (k8s) with some VMs I haven’t finished migrating to the cluster.
Beyond the improvement in hardware redundancy I’ve seen significant performance improvements with Ceph/Longhorn being spread out to so many disks, rather than a bunch of VMs on a few pools. That said, I haven’t benchmarked it, but some I/O heavy workloads are certainly faster.
Kubernetes
What actually drove the project was investing in a real K8s cluster. It’s not bare metal; it’s rke2 on Proxmox (with that awesome Ceph integration). I realized roughly a year or so ago that I was trying to remake many of K8s scheduling/operation features with Ansible (and doing it poorly). Making the jump has been fantastic.
Forgejo + Renovate + ArgoCD
Not much to say on Forgejo, Renovate, and ArgoCD beyond they are fantastic projects.
Forgejo has been an excellent replacement for the now-bloated mess that is GitLab. I did not need the array of features it provided and Forgejo is a snappy replacement with everything I need.
Renovate is phenomenal; it required a bit of configuration to get it in a place where I was happy but automating update checks with automatic PRs makes patching the homelab significantly easier.
ArgoCD fixed a big problem I had in the past with identifying and reconciling the state of an application. It requires a bit of configuration with more complex/dynamic software (e.g. Browsertrix) but is generally easy to setup and a solid pairing to Renovate/Forgejo.
What’s next
Things are somewhat stable and I’m happy with that but the big items in the near future include:
- Finishing Grafana/Loki/etc. and integrating everything including Proxmox
- Multiple clusters (management cluster + DMZ clsuter)
- Finish moving VMs to K8s