I Can't Open The Garage and Other Lessons Not Learned Running A Homelab
September 12, 2026
We live a car free life, so our garage keeps the one mode of transportation we have. Our Bikes. Unlike other garages, ours doesn't have a keypad on the outside for us to open. A problem easily solved, however why solve it with an off-the-shelf, purchased, solution when you can wire the existing remote into Home Assistant. So I built my own solution which is wired into Home Assistant and will open the garage if you key in the right code at our side door.
Enter the homelab! As I have mentioned in my Immutable Infrastructure Journey , I've worked to setting up a homelab which I can quickly recover if a server loses its mind for any reason. Services automatically bounce around to any other available node, including services which back this website! I also ensure that my VMs are both always up-to-date and constantly test my terraform by automatically re-provisioning each virtual machine randomly at least once a month. This process leads to some exciting revelations like what happens if the node that needs to be re-provisioned is setup to re-provision itself. Something that I annoyingly overlooked the first time I automated re-provisioning.
This time is different though, unlike the other times where a design decision I had made was the source of an outage, this outage was mostly external.
Because my VMs automatically re-provision themselves, the VMs are at the whim of any external dependencies that are required to set it up from scratch. I woke to a bunch of alerts on my phone of various services being offline because the re-provision job failed to complete successfully and the cluster couldn't start a few jobs that were pinned to the node that was being re-provisioned. Not a problem, check the Terraform workspace isn't locked and rerun the job. Usually after about 5 minutes the node is back online and registered in the cluster. Not this time however. In my case a packaging key rotation ground everything to a halt and packages like Consul, Nomad, and Vault were unable to be installed as they had been signed with a different key that hadn't been published yet. The job always expects packaging to be available, and this time it wasn't. The end result, one of Home Assistant's dependencies was unable to start as it is pinned to the node that had just deleted itself from the cluster leading to a funny realization that one issue in a far off land made it so opening my garage became a little more inconvenient.
This naturally led me to think what actions could I take to prevent a similar outage from occurring in the future. Running my own apt mirror for the packages I need? It is definitely something that I can maintain, but is it something I want to maintain? Given the frequency of key changes, I think this is an acceptable amount of risk I'm willing to accept in my system.