Kubernetes and Nix Cluster

About this time last year, I purchased 3 mini computers with the aim of making a Kubernetes cluster. I followed an excellent youtube video: This homelab setup is my favorite one yet - Dreams of Autonomy (Github Repository) which described the machines to buy (Beelink EQ 13 Mini), how to upgrade the SSD and RAM, how to provision them with NixOS, setup K3S, and connect using kubectl to manage the cluster. You will also have to modify your router’s DHCP settings so you can give the cluster static IP addresses.

Things deployed on it (using Helm/helmfile and kustomize):

• Longhorn - for storage
• MetalLB - for load-balancing and IP assignment
• Nginx - as an ingress controller
• Pihole - for ad blocking and DNS service
• externaldns - able to update Pihole DNS records when a new service is added

These are all in the Github repo linked above. I’ve since experimented with other services on the cluster such as:

• Grafana and Prometheus - for dashboards and monitoring

• Flux - CI/CD deployment from config direct to the cluster

• Livebook - for live coding, and interacting with services

• Redis - as a data store (maybe I should have use Valkey? )

• Postgres - database

• Kafka - message bus

• and more I’m currently trying..

Things that have surprised me are:

• There was a small configuration value missing from the video, which I created a pull request to address
• the lack of readily available and easily usable Helm charts on ArtifactHub - Bitnami filled this role somewhat but has converted to a commerical model, disappointingly
• I tried to use LLMs to give me correct helm values but it always got confused (should it not be simple?)
• the sheer amount of YAML config
• Troubleshooting needs a defined workflow of what to check (but this is useful to practice)
• Nix running out of space on /boot while trying to upgrade the Linux kernel

Things that worked well

• The hardware was very easy to use and upgrade
• Nix has been solid in upgrades (I expected things to break - only one was the /boot issue above)
• Using internal or external domains was easier than I thought, I’ve exposed some on .lan and some on a public domain
• kubectl is a nice tool, the information density is good
• Helmfile is a nice way to deploy
• Livebooks were brilliant to test other services, e.g. I wrote Elixir to interact with Postgres, Redis and other services on the cluster

Would I do this again?

I’m happy I did this as it’s been a great learning opportunity for me to break things and fix them again. Kubernetes can be a bit off-putting with concepts, how they are linked and how to just deploy something quickly to verify it works. There’s a lot to understand before something works.

I really would only recommend using Kubernetes an Enterprise context where you have more than one workload. For simpler apps, I would stick to docker, docker-compose and things like Dokku. Or of course doing it just for fun, learning and experimenting!