Enter the homelab! As I have mentioned in my Immutable Infrastructure Journey , I've worked to setting up a homelab which I can quickly recover if a server loses its mind for any reason. Services automatically bounce around to any other available node, including services which back this website! I also ensure that my VMs are both always up-to-date and constantly test my terraform by automatically re-provisioning each virtual machine randomly at least once a month. This process leads to some exciting revelations like what happens if the node that needs to be re-provisioned is setup to re-provision itself. Something that I annoyingly overlooked the first time I automated re-provisioning.

This time is different though, unlike the other times where a design decision I had made was the source of an outage, this outage was mostly external.

Because my VMs automatically re-provision themselves, the VMs are at the whim of any external dependencies that are required to set it up from scratch. I woke to a bunch of alerts on my phone of various services being offline because the re-provision job failed to complete successfully and the cluster couldn't start a few jobs that were pinned to the node that was being re-provisioned. Not a problem, check the Terraform workspace isn't locked and rerun the job. Usually after about 5 minutes the node is back online and registered in the cluster. Not this time however. In my case a packaging key rotation ground everything to a halt and packages like Consul, Nomad, and Vault were unable to be installed as they had been signed with a different key that hadn't been published yet. The job always expects packaging to be available, and this time it wasn't. The end result, one of Home Assistant's dependencies was unable to start as it is pinned to the node that had just deleted itself from the cluster leading to a funny realization that one issue in a far off land made it so opening my garage became a little more inconvenient.

This naturally led me to think what actions could I take to prevent a similar outage from occurring in the future. Running my own apt mirror for the packages I need? It is definitely something that I can maintain, but is it something I want to maintain? Given the frequency of key changes, I think this is an acceptable amount of risk I'm willing to accept in my system.

See, I'm also a Proton Mail user and they let you write your own custom sieve filters which allow you to super charge email filters with a set of rules and functions that run on every incoming message. I am sharing my filter just in case anyone else has a similar system.

This script will strip the local part (the part before the @) of an incoming email's to address and make sure it is within the domain of the from address (or the part after the @). If they match, great, it's a normal marketing email and we can apply normal marketing rules. If they don't match, accelerate its expiry because I likely don't care for it since a company sold my email off to someone else.

require ["fileinto", "envelope", "regex", "variables", "vnd.proton.expire"];

# Bypass filtering on my special email address
if envelope :all "to" "<redacted>@brittg.com" { stop; }

# Get Recipent's local part
if envelope :localpart :regex "to" "(.*)" {
    set "expected_domain" "${1}";

    # Get Sender's domain name
    if envelope :domain :regex "from" "(.*)" {
        set "actual_domain" "${1}";

        # If the local part shows up in the domain, great! Let them in.
        if string :contains "${actual_domain}" "${expected_domain}" {
            fileinto "Marketing";
            expire "day" "180";

        # If there is a mismatch, woof! Throw them aside.
        } else {
            fileinto "Marketing/Noncompliant";
            expire "day" "30";
        }
    }
}

You will note that I don't outright delete emails, because even legitimate emails can get caught in this system, so I want to at least give myself a chance to flag emails as good before they are reaped. One common case is Square receipts, which seem to always prefer the last email address that is put into one of their terminals and tie that to the card being used in the transaction. An annoyance but I can live with that.

Combine this with the fact that only a special email address actually gets through to notifying me and my inbox has come back into being manageable, even when the bad actors start signing me up for other marketing lists.

Big shoutout to https://sieve-tester.com/ for testing the filters before I put them into action.

Both my partner and I got used to the rules and craved a little bit more weirdness in our games which is where my project Wild Chess comes in. So one of the things to switch things up even more is adding a second king to defend. This usually has the nice side effect of making games a bit quicker, so we can fit more games in during our lunch break together. It is also a fun challenge as one move can sometimes place both kings into check. But that wasn't enough for us. We also added in a morphing mechanic where if you capture a piece, it morphs into the piece that it just captured. This adds an entirely new dimension to the game where you threaten pieces instead of capturing with your strongest piece as your strong pieces usually get downgraded when they capture. In the same vein it incentivizes lesser pieces to play more offensively with the potential reward they become a better piece by capturing it.

The added layers of complexity add just enough flavor to a game to keep us entertained. If you are at all interested in a twist on a classic, definitely check out Wild Chess

I also have more and more friends I'm getting to know in the Portland music scene who I want to succeed. Not only because I consider them friends but because their music has been some of my favorite to listen to this year!

In my research I've tested Apple Music, Tidal, YouTube Music, and Qobuz. Each has it's pros and cons. For Apple Music, it was a tough sell as I'm an android user and when I installed their app it crashed my phone whenever I wanted to listen to music. Somehow their persistent music notification was buggy and caused massive stability issues with my phone even though their UI was preferred over the other options. I had Tidal for a bit, but their music selection didn't quite hit for me, especially when it came to the more local artists I like to listen to. It wasn't because the local artists were not on the platform, they certainly were. But Tidal's artist disambiguation left a lot to be desired, often merging completely unrelated artists with the same name together making a challenging listening experience. YouTube Music definitely has a lot of artists. It's UI is a little.. odd? I kinda just want a music player and not have videos in my experience at all. I've also learned from past experiences to not trust Google keep any, even popular, service around for long. Rest in peace Google Music. I didn't want to put my eggs in that basket again.

I've ultimately settled on Qobuz for a few reasons important to me. They have better than most payouts for artists, although lets be honest, it's no where close to buying a record or merch at a show. But Qobuz does let you purchase DRM-free songs for your collection which I've been making it a point to do when I am able to. If you sign up for their Sublime subscription tier, you even get a discount on your song purchases which has saved me some money collecting my favorite artists music. They also have a partnership with Soundiiz at the time of writing which lets you import all of your playlists to Qobuz for free.

In addition to Qobuz, I'm also a fan of supporting artists via Bandcamp. Many artists have a pay what you like model for their music with the option to pay more if you want to cover some of the fees that Bandcamp has. Like Qobuz, you also get DRM-free downloads of your purchases. One thing I appreciate about Bandcamp is that they will let you stream songs for a limited number of times before you actually decide to buy them, giving you a taste of the music before going ahead with your purchase.

Anyhow, support your local artists, especially in person, and if not in person at least through better companies than Spotify.

This post will go through what Terraform is and using Terraform to manage Proxmox VMs automatically.

Prerequisites

Before we get started there are a few things we will need to kick off this journey. Each blog post will build upon the last so if there is a tool mentioned that you don't recognize, check out the #ImmutableInfrastructureJourney tag on this blog.

Since the last post I've upgraded Proxmox to version 8.0.3 so everything mentioned here should work with that version. You can find instructions to install Proxmox at Proxmox's getting started page.

You will also need to download Terraform from HashiCorp's Terraform download page.

For posterity, everything that I am doing will be done within Windows WSL but should work on any system supported by Terraform. The Proxmox node I'll be working on is installed on an Intel NUC with 32GB of memory although can be done on a smaller or larger machine as well.

Creating a new VM with Terraform

Now, why would one want to do this in the first place? Virtual machine templates are already doing a ton of heavy lifting to no longer need to run through the ubuntu install from scratch. You can even install all of the packages, configuration, and services to your hearts content with one of these golden images and use that, why do you want to manage this with yet another tool? I believe the power with Terraform doesn't truly show until you start managing multiple components of a home lab in concert with each other. For example, we've all needed to set up a DNS record for a newly deployed service or create a credential that we then have to give to a config deep within a VM to ensure that it spins up correctly. Even a task such as opening a port on a home router is often tied to some infrastructure in a home lab. Terraform can manage all of these resources together and more in a simple, infrastructure as code way. All while building on the things we have already learned in setting up a cloud ready image from the last post.

Once you have a cloud ready image, either by downloading an existing image from Ubuntu's site or building one from scratch using the tutorial in the previous post, you are ready to start composing your entire home lab as code instead of the resources on a single machine with Terraform.

Start a Terraform project

Our first step is to get a Terraform project setup with the providers we will need.

What is a Terraform Provider?

Terraform itself can manage very little by itself. You can pretty much only use the built-in functions and nothing more. In order to start using Terraform to manage things you must pull in Terraform providers which let Terraform know how to interact with various services APIs.

In a new directory, create a file called main.tf and lets pull in the Proxmox provider, telmate/proxmox, by defining it as a required provider.

terraform {
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "2.9.14"
    }
  }
}

From here we need to configure the provider to let it know how to connect to our Proxmox server. Terraform providers can usually be configured in one of two ways. Either within HCL, the configuration language that Terraform uses, in a provider block or within environment variables which providers will look for when you are running Terraform.

I'll show the HCL with the environment variables as comments aside. I highly recommend storing these in environment variables as much of these configurations are considered secret and should never be committed to a repo although I recognize that getting something working quick is higher priority than making it work securely.

Below your terraform block within the main.tf file we created, we will add in the following provider block.

provider "proxmox" {
  pm_api_url      = "https://proxmox-server:8006/api2/json" // $PM_API_URL
  pm_user         = "terraform@pve"                         // $PM_USER
  pm_password     = "changeMe!123"                          // $PM_USER

  pm_tls_insecure = "true"  // Required for self-signed certificates
}

If you don't have a user dedicated to managing Proxmox with Terraform, the docs on the Proxmox provider list instructions on how to generate them with the correct permissions for managing Proxmox.

Run a simple terraform init which will tell Terraform to download all of the required providers from the Terraform Registry and that is all we need to do in order to setup Terraform.

$ terraform init
Initializing the backend...

Initializing provider plugins...
- Finding telmate/proxmox versions matching "2.9.11"...
- Installing telmate/proxmox v2.9.11...
- Installed telmate/proxmox v2.9.11 (self-signed, key ID A9EBBE091B35AFCE)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Create a VM Resource with Terraform

Now that we have configured terraform to talk to Proxmox, lets start managing resources!

What is a Terraform Resource?

In Terraform, any single unit of infrastructure that can be managed is a resource. Resources can be created, updated, or deleted depending on what you change within the resources parameters. These resources are defined in HCL as resource blocks with the type that the resource is suppose to be and a unique name identifying the resource. Within the block are the parameters of the resource. Terraform providers let Terraform know which parameters of a resource are safe to update and which parameters require a resource to be destroyed and created from scratch.

For example, a Proxmox VM has a description attached to it. This description is more a human helper for us to help identify a VM's function and has no real impact to what is contained on the VM. Updating this parameter does not cause the resource to be "tainted" or marked for deletion and recreation. Other parameters, such as updating the base image cannot be done cleanly without "tainting" the resource. In this case the resource would be marked as tainted and cleaned up when running terraform.

Lets create a file specifically for a new Proxmox virtual machine using a cloud ready image. I'll name this new file proxmox.tf and add a resource for a new VM named "lab".

resource "proxmox_vm_qemu" "lab" {
}

In this stanza we need to tell Proxmox what image to clone when creating this VM. My cloud ready image has the name ubuntu-ci-base so we will use that as our clone name. We also need to let the provider know the name of the proxmox node we would like to spin this VM up on. This is useful for setups where you have a clustered Proxmox setup. Even if you don't have a Proxmox cluster, the API still needs to know.

resource "proxmox_vm_qemu" "lab" {
  clone       = "ubuntu-ci-base"
  target_node = "nuc"
}

And now we can give this VM a name, 4 vCPUs, and 2GB of memory.

resource "proxmox_vm_qemu" "lab" {
  clone       = "ubuntu-ci-base"
  target_node = "nuc"

  name    = "node-1"
  sockets = 1
  cores   = 4
  memory  = 2048
}

Next lets add in 100GB of disk for our machine to use. We also need to attach a network interface so the VM can reach the internet.

resource "proxmox_vm_qemu" "lab" {
  clone       = "ubuntu-ci-base"
  target_node = "nuc"

  name    = "node-1"
  sockets = 1
  cores   = 4
  memory  = 2048

  disk {
    type    = "scsi"
    storage = "local-lvm"
    size    = "100G"
  }

  network {
    bridge = "vmbr0"
    model  = "virtio"
  }
}

And there we have it, a basic VM that we can fully manage via Terraform! Now we run terraform plan to make sure that everything looks correct.

$ terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # proxmox_vm_qemu.lab will be created
  + resource "proxmox_vm_qemu" "lab" {
    [...]
  }

  Plan: 1 to add, 0 to change, 0 to destroy.

Perfect! Looks like all of the resources we defined are accounted for in this plan. Lets go ahead and apply it with terraform apply.

$ terraform apply

[... Terraform plan output ...]

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

proxmox_vm_qemu.lab: Creating...
proxmox_vm_qemu.lab: Still creating... [10s elapsed]
[...]
proxmox_vm_qemu.lab: Still creating... [1m40s elapsed]
proxmox_vm_qemu.lab: Creation complete after 1m41s [id=nuc/qemu/110]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Now if we log into Proxmox we can see that Terraform created a VM and started it, all entirely defined within the code that we just wrote. If you try and modify anything within the stanza notice that depending on what is modified, Terraform will either update the VM in place or destroy and recreate the VM.

Updating things like CPU, memory, disk, and even network can be updated by the provider without first destroying it. The provider is smart enough to know that it first needs to stop the VM, modify the hardware settings of the VM, and then start the VM back up. However when updating the base image, there is no way to cleanly migrate from one base image to another base image so Terraform instead deletes the old resource and recreates a new one in its place.

Terraform State

So how the heck does Terraform keep track of all of this stuff? Enter Terraform state files, found in the terraform.tfstate file that was created after running terraform apply. This state file is what Terraform thinks is running out in the infrastructure it is managing. It contains things like what specific vm instance maps to what resource in your code and all the features that are associated with the resources.

This file provides Terraform a way to know if someone or something modified the VM outside of Terraform and the next time you run Terraform, it will notice the change and revert it back to what is defined within your code. This is the beauty of infrastructure as code as it forces people to codify changes within code that can be peer reviewed by others.

To check in or not to check in terraform.tfstate

It may be tempting to check in the terraform.tfstate file into git. I would advise against this for a few reasons. First, this file contains the state of every resource that Terraform is managing, not just the properties you define on a resource. Some of these resources may have private values which would be bad to leak. Second, storing this file within git implies a workflow where multiple users may want to make changes on their own branches. The state file is suppose to reflect the current desired state of your infrastructure so if person on feature branch branch-a runs terraform apply from there branch and then later someone on the main branch runs terraform apply, there will be changes that have never been seen in main which could cause mayhem as Terraform attempts to remedy the resources it thinks has been tainted. And lastly, even if you were to keep everyone from using Terraform in their own branches, Terraform does not have a locking mechanism with the default state backend that would prevent multiple instances of Terraform from running at the same time. Potentially causing duplicate and conflicting state files being generated, both being accurate and true but talking about different infrastructure.

If in doubt, store the terraform.tfstate file in a single location centralized location, preferable using a state backend that has locking like S3 or Terraform Cloud. This ensures that one and only ever one instance of Terraform may be operating on the state file at any given time.

Clean up

Lets say you are finished with the VM that you created with Terraform. Instead of deleting all of the code you wrote and running terraform apply again, simply tell Terraform to destroy all the defined resources with terraform destroy.

proxmox_vm_qemu.lab: Refreshing state... [id=nuc/qemu/110]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # proxmox_vm_qemu.lab will be destroyed
  - resource "proxmox_vm_qemu" "lab" {
  [...]
  }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

proxmox_vm_qemu.lab: Destroying... [id=nuc/qemu/110]
proxmox_vm_qemu.lab: Destruction complete after 4s

Destroy complete! Resources: 1 destroyed.

And that's a crash course on using Terraform to start to manage VMs in Proxmox.

In the next post we will talk about how to add in our own customizations to an instance that Terraform manages by using the full power of seeding our cloud ready images with configs using cloud-init.

Prerequisites

Before we get started there are a few things we will need to kick off this journey. Each blog post will build upon the last so if there is a tool mentioned that you don't recognize, check out the #ImmutableInfrastructureJourney tag on this blog.

This lab depends on a single Proxmox server which should be installed on its own dedicated hardware. Instructions on how to install Proxmox can be found at Proxmox's getting started page. I am currently using version 7.2-11 so that version should work here.

You will also need to download HashiCorp's Packer from HashiCorp's getting started page.

For posterity, everything that I am doing will be done within Windows WSL but should work on any system supported by Packer. The Proxmox node i'll be working on is installed on an Intel NUC with 32GB of memory although can be done on a smaller machine as well.

Building a VM template with Packer

One of the pieces of that I often found myself working on more than I liked was installing Ubuntu from scratch and getting all of my tools and services first installed and then configured. It was honestly more headache than was worth and thus I found myself never updating to the latest version of Ubuntu until it was a year or two EOL. When update time did come, there was always some detail that I had left behind and was lost to time.

This is where Packer comes into play. Packer is an open source tool by HashiCorp that aims to build VM images automatically for me from a single source file. Packer essentially automates the process of creating my golden VM templates which will then be used to spin up subsequent VMs from boot to template by interacting directly with the VM as if a user were seated at the console. It's pretty magical to watch all of this automation in front of my eyes do what I had spent so much time configuring on my own.

Create a Packer file

We will create a file which defines the VM we would like to use as our template for the rest of the VMs we will be using. This will have all of our boot instructions as well as any configurations specific to my environment. Today I'll be using "Ubuntu 22.04 Server" as my base image.

In a new folder create a file named ubuntu-2204-server.pkr.hcl. Open this file in an editor of choice and lets work on defining everything we need starting with the plugins we will be using. See Packer doesn't come out of the box knowing how to interact with Proxmox but it does know how to download a plugin that does.

# ubuntu-2204-server.pkr.hcl

packer {
  required_plugins {
    proxmox = {
      version = ">= 1.1.1"
      source  = "github.com/hashicorp/proxmox"
    }
  }
}

Now below the packer stanza I'll add my NUC as a source for the proxmox-iso builder. This block basically contains the information on how to connect to Proxmox, where to store our ISO image, hardware details when provisioning the VM, and the keys to press as the VM boots for the first time.

Lets start by first adding the connection details.

# ubuntu-2204-server.pkr.hcl

variable "proxmox_username" {
  type = string
}
variable "proxmox_password" {
  type = string
  sensitive = true
}

source "proxmox-iso" "intel-nuc" {
  proxmox_url              = "https://nuc-proxmox.local:8006/api2/json"
  insecure_skip_tls_verify = true
  username                 = var.proxmox_username
  password                 = var.proxmox_password
  node                     = "intel-nuc"
  task_timeout             = "10m"
}

I'll next add some information on what ISO I want to use and where to store that ISO just below task_timeout as well as define how much CPU and memory to use when creating the template. This doesn't have to be a ton as you can always change the CPU and memory when you create instances of this VM template later. I'll also add some additional hardware devices like network and storage here.

# ubuntu-2204-server.pkr.hcl

source "proxmox-iso" "intel-nuc" {
  # [...]

  iso_url          = "https://releases.ubuntu.com/22.04.1/ubuntu-22.04.1-live-server-amd64.iso"
  iso_checksum     = "sha256:10f19c5b2b8d6db711582e0e27f5116296c34fe4b313ba45f9b201a5007056cb"
  iso_storage_pool = "local"
  unmount_iso      = true  # Unmount's the ISO after installation is completed

  memory = 2048
  cores  = 4
  os     = "l26"

  network_adapters {
    model  = "virtio"
    bridge = "vmbr0"
  }
  disks {
    type              = "scsi"
    disk_size         = "64G"
    storage_pool      = "local-lvm"
    storage_pool_type = "lvm"
  }
}

Cloud-init

Ubuntu, and many more OSes designed to run in cloud environments, can take advantage of unattended installs, meaning I don't actually have to be at the computer to finish the initial installation. Since we want this process to be as hands off as possible so that we can easily repeat it when needed, we are going to make use of cloud-init here. The way that it works is when ubnutu boots up for the first time a user, or Packer in this case, can type a few commands into the bootloader and tell Ubuntu, "I would like you to install using this script I'm going to provide on a CD named cidata" and Ubuntu will load all of the configuration from that CD which the user provides and performs all sorts of tasks on behalf of the user. These can be as straight forward as install ubuntu and nothing else or they can be very customized to automatically install and run entire playbooks with no intervention from the user.

For our use case we only want to upgrade all of our packages to the latest versions and install a few packages which will help proxmox be aware of what is going on inside of the VM when it launches.

Create a new folder named cidata and create two files called user-data and meta-data. Open up user-data in an editor and add the cloud-config to the file.

#cloud-config
autoinstall:
  version: 1
  refresh-installer:
  locale: en_US.UTF-8
  keyboard:
    layout: us
  identity:
    hostname: ubuntu-server
    password: "$6$QnLQVguRtS.t38.F$qybKEFbxqQRvbntU5dhLgVauc/FIAw4RUqpL7RcP5WHRrjOurA40M.24FzGB9hPgtD28B93cxkrYe6.ky63I8."
    username: ubuntu
  ssh:
    allow-pw: false
    install-server: true
    authorized-keys:
      - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ6z4PxtPlUS4aOvY8/XBJDCUr0juwIKVXrClcJkI6QH britt@Helios"

  package_update: true
  package_upgrade: true
  packages:
    - qemu-guest-agent
    - cloud-init

Walking through the file from the top down we have the following. #cloud-config lets cloud-init that this isn't a shell script, which can also be used to provision but is extremely brittle and not recommended. An autoinstall key that shows the version of autoinstall, refresh-installer ensures we use the latest version of the ubuntu installer before continuing. The next three lines are all about keyboard layout. Then we get to identity which is the default computer name, username, and password for the install. You will want to make sure that you use a password generated with openssl passwd -6. The next section is for defining what keys you would like to use for ssh with the default user. This should be a key you have access to since Packer will use this user later on to finish the provisioning steps and reset the VM for cloud-init to run when this template is cloned later.

This last part of the file is for all things packages. I recommend leaving this list as small as possible to make this template as versatile as possible. For Proxmox you will want to at least install qemu-guest-agent to make sure that Proxmox knows what IP the VM has when it boots. This also helps out with automatically provisioning VMs in the future.

Unattended Installations Using Packer

Now that we have the contents for the cloud-init CD created we need to mount this CD to the template using the additional_iso_files stanza.

# ubuntu-2204-server.pkr.hcl

source "proxmox-iso" "intel-nuc" {
  # [...]

  additional_iso_files {
    cd_files         = ["./cidata/*"]
    cd_label         = "cidata"
    unmount          = true
    iso_storage_pool = "local"
  }
  qemu_agent   = true

  ssh_username = "ubuntu"
  ssh_private_key_file = "~/.ssh/id_ed25519"
  ssh_timeout = "25m"
}

Also in this code is qemu_agent which lets proxmox know that this VM will have qemu-guest-agent installed on it and we can use that to discover the IP address of the node for provisioning with the ssh details in the following ssh fields.

This next part takes some time to get right, its best to find the proper commands to give GRUB, the bootloader, online. This will let Ubuntu know that you want to automatically perform an unattended installation.

# ubuntu-2204-server.pkr.hcl

source "proxmox-iso" "intel-nuc" {
  # [...]

  boot_wait = "15s"
  boot_command = [
    "<esc><wait>",
    "<esc><wait>",
    "c<wait>",
    "set gfxpayload=keep",
    "<enter><wait>",
    "linux /casper/vmlinuz quiet<wait>",
    " autoinstall<wait>",
    " ds=nocloud;<wait>",
    "<enter><wait>",
    "initrd /casper/initrd",
    "<enter><wait>",
    "boot<enter><wait>",
  ]
}

Finally we give a name to the template on this Proxmox node.

# ubuntu-2204-server.pkr.hcl

source "proxmox-iso" "intel-nuc" {
  # [...]

  template_name = join("-", [
    "ubuntu",
    "2204",
    "base",
    formatdate("YYYYMMDD-hhmm", timestamp()),
  ])
}

Build and Prepare to Templatize

Now we've used cloud-init to automatically install and configure the basics of Ubuntu. The thing with cloud-init is that many of the modules only run when the VM first boots. This is not ideal for our VMs if we create clones from a template that has already ran cloud-init so we have to run some additional steps after the initial install to reset cloud-init's state so that cloud-init thinks that its being ran for the first time.

Add an additional stanza to Packer to target the source to build on as well as some additional steps that will tell Packer to ssh into the VM and run a cleanup script.

# ubuntu-2204-server.pkr.hcl

source "proxmox-iso" "intel-nuc" {
  # [...]
}

build {
  name = "ubuntu-x86_64"
  sources = [
    "source.proxmox-iso.intel-nuc",
  ]

  # Clean up the machine for cloud-init
  provisioner "shell" {
    execute_command = "echo 'ubuntu' | {{ .Vars }} sudo -S -E sh -eux '{{ .Path }}'"
    inline = [
      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done",
      "sudo rm /etc/ssh/ssh_host_*",
      "sudo truncate -s 0 /etc/machine-id",
      "sudo apt -y autoremove --purge",
      "sudo apt -y clean",
      "sudo apt -y autoclean",
      "sudo cloud-init clean",
      "sudo rm -f /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg",
      "sudo sync"
    ]
  }
}

At this point we are ready to run Packer and build a template VM. The entire Packer file is below.

# ubuntu-2204-server.pkr.hcl

packer {
  required_plugins {
    proxmox = {
      version = ">= 1.1.1"
      source  = "github.com/hashicorp/proxmox"
    }
  }
}

variable "proxmox_username" {
  type = string
}
variable "proxmox_password" {
  type = string
  sensitive = true
}

source "proxmox-iso" "intel-nuc" {
  proxmox_url              = "https://nuc-proxmox.local:8006/api2/json"
  insecure_skip_tls_verify = true
  username                 = var.proxmox_username
  password                 = var.proxmox_password
  node                     = "intel-nuc"
  task_timeout             = "10m"

  iso_url          = "https://releases.ubuntu.com/22.04.1/ubuntu-22.04.1-live-server-amd64.iso"
  iso_checksum     = "sha256:10f19c5b2b8d6db711582e0e27f5116296c34fe4b313ba45f9b201a5007056cb"
  iso_storage_pool = "local"
  unmount_iso      = true  # Unmount's the ISO after installation is completed

  memory = 2048
  cores  = 4
  os     = "l26"

  network_adapters {
    model  = "virtio"
    bridge = "vmbr0"
  }
  disks {
    type              = "scsi"
    disk_size         = "64G"
    storage_pool      = "local-lvm"
    storage_pool_type = "lvm"
  }

  additional_iso_files {
    cd_files         = ["./cidata/*"]
    cd_label         = "cidata"
    unmount          = true
    iso_storage_pool = "local"
  }
  qemu_agent   = true

  ssh_username = "ubuntu"
  ssh_private_key_file = "~/.ssh/id_ed25519"
  ssh_timeout = "25m"

  boot_wait = "15s"
  boot_command = [
    "<esc><wait>",
    "<esc><wait>",
    "c<wait>",
    "set gfxpayload=keep",
    "<enter><wait>",
    "linux /casper/vmlinuz quiet<wait>",
    " autoinstall<wait>",
    " ds=nocloud;<wait>",
    "<enter><wait>",
    "initrd /casper/initrd",
    "<enter><wait>",
    "boot<enter><wait>",
  ]

  template_name = join("-", [
    "ubuntu",
    "2204",
    "base",
    formatdate("YYYYMMDD-hhmm", timestamp()),
  ])
}

build {
  name = "ubuntu-x86_64"
  sources = [
    "source.proxmox-iso.intel-nuc",
  ]

  # Clean up the machine for cloud-init
  provisioner "shell" {
    execute_command = "echo 'ubuntu' | {{ .Vars }} sudo -S -E sh -eux '{{ .Path }}'"
    inline = [
      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done",
      "sudo rm /etc/ssh/ssh_host_*",
      "sudo truncate -s 0 /etc/machine-id",
      "sudo apt -y autoremove --purge",
      "sudo apt -y clean",
      "sudo apt -y autoclean",
      "sudo cloud-init clean",
      "sudo rm -f /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg",
      "sudo sync"
    ]
  }
}

Run packer build ./ubuntu-2204-server.pkr.hcl and packer will start the process of installing Ubuntu within Proxmox. If you pop on over to Proxmox you will see a brand new VM spin up which will start typing in all of the keys that were specified in boot_command. This will type the magic commands into grub and send ubuntu into unattended installation mode. After about 5-10 minutes Packer will ssh into the VM and reset cloud-init's state so that the VM can be turned into a template which can be used again with cloud-init for your more specialized configs.

Now we have an entirely reproducible VM template that doesn't take up a whole VMs worth of space. And this can all be stored in code and even updated automatically within a CI pipeline.

In the next post we will talk about how to use this new VM template and the power of Terraform to automatically create VMs configured with the tools and programs you need.

Home-manager

So in my exploring of what Nix is I found a tool that explicitly is for managing dotfiles called Home-manager. Installing this tool and running through the Nix Pills to start learning how to write my own Nix files really started to open my eyes to the power of how Nix manages files. My current dotfiles git repo structure is meant to be an overlay on an existing home directory. It's installation is almost intentionally messy, but a familiar kind of messy because when I say overlay, it isn't the nice way that docker layers overlay on each other in the overlay2 filesystem. The installation script iterates over every file in the dotfile repo and creates a symlink from your home directory to the repo itself. Sometimes clobbering your existing configurations in the process.

Now how home-manager handles installation is fascinating. Remember how I said that Nix is declarative? Well when you write your configuration for your dotfiles, you are not writing a .vimrc file or a .bashrc file. Nix expects you to define your configurations in Nix and it will build out a .vimrc and .bashrc for you. Each configuration file generated based on what is know in Nix as a derivation or builder. Since everything in Nix is built to be immutable, Nix keeps every version of your generated dotfiles in /nix/store and symlinks to the versions you actually want to use. This makes rolling back to a previous version of your configurations super simple as its just moving some symlinks around. Home-manager acts as a profile manager for these generated configuration files. It sees what you have declared in your own home.nix file and makes sure that what you want exists in your version of home.nix. It then modifies your PATH or symlinks config files into place in your home directory (if it can do so without destroying anything) and you have a configured environment.

Dotfiles are not the only thing that Home-manager knows how to handle. It can also handle packages and with that you can also get the same benefits of Nix's declarative and immutable powers. Lets take git for instance which I have defined in my home.nix file.

{ config, pkgs, ... }:
{
  # ...
  home.packages = {
    pkgs.git
  }
  # ...
}

A version of the package git is downloaded, built, and stored within /nix/store with a hash, sometimes referencing dependent libraries by their unique hashes within the derivation for git. git is then symlinked into your PATH so that specific version is made available to you. Not only does this allow you to ensure that the same version of a package exists for you down to the libraries that package is dependant on, it also allows you to have different versions of the same package installed simultaneously which other package managers are definitely not good at handling. This may not be a big deal for most, but for developers out there who bounce between multiple projects, this is huge.

Anyhow, there is a lot more to explore here and I'm only at my dotfiles and am getting some use out of it so far. Needless to say, I'm excited to see where this path takes me.

Removing single points of failure

One of the reasons why this system failed so spectacularly so often can be attributed to the fact that all of my critical infrastructure was running on a single machine which I constantly used as a test bed for new projects.

It had always been in my head that I wanted some way to run Home Assistant on another machine if I needed to do maintenance on my primary server. A few key things stood in my way to achieving this goal. I needed my service to be available no matter where it was, I needed its state to be up-to-date everywhere so it can move at a moments notice, and I needed the migration of the services to be entirely hands-off. Because of these requirements I decided to use <N> different services paired together with each other.

• Tailscale
• Consul
• Traefik
• GlusterFS
• And lastly Nomad.

Keeping a single point of entry for end user devices

Tailscale

Now in my particular setup I have a DigitalOcean droplet I am running publicly to run my website, It also serves as a way to access services I run at home without a stable public IP address. I am using a VPN service called Tailscale which provides me with a very minimal setup private network that I plan on having all the nodes in my cluster live on so they can all communicate with each other without needing to worry about network problems which may get in the way. Having each node on the same network also drastically simplifies routing requests to a publicly facing node for ease of accessing services.

Now that everything is connected I wanted to make sure that I didn't have to maintain a list of what node Home Assistant could possibly be running on. The last thing I want is to try a bunch of IP addresses and ports to hunt for where Home Assistant is currently running. I also didn't want to be in the business of constantly updating a DNS entry to point to the real live instance of Home Assistant either.

Consul

Luckily this is a problem that is solved today by using Consul, a service discovery platform which pairs nicely with the rest of my planned stack. Its sole job is to keep track of what services are running where and the health of each individual service. I went ahead and installed a Consul agent on each node I wanted to have in my cluster and made sure to bootstrap an odd number of them as voting members so they could find quorum with each other. This attribute is very important in clustered computing as if you lose quorum and have an even number of authoritative members in a cluster they could split and each have different copies of entirely valid state with no one to break the tie.

Consul needed to be setup to advertise only the ip addresses assigned to Tailscale which I accomplished by adding bind_addr to my Consul config and setting it to a go-sockaddr template which grabs the ip address for a given interface name, in this case tailscale0.

bind_addr = "{{ GetInterfaceIP \"tailscale0\" }}"

Now services returned by Consul will automatically be routable to every other node in this cluster.

Installed with Consul is a DNS server which can be used to get at services without having to know where the service actually is. For example, if you had MongoDB as a service you would find it by querying mongodb.service.consul to Consul's DNS server and it would return a list of IP addresses which expose the mongodb service. There are a few other primitives as well but the one we care about at the moment is node for finding a specific node in a cluster.

Traefik

Lastly to tie together the public part of this problem I need a way to expose the services. This is where I pulled in Traefik to automatically add and remove services to its catalog of routers. Conveniently Traefik knows how to talk to Consul to get a list of services and more importantly where they are located in order to act as a reverse proxy to route incoming requests to the service seemlessly.

Maintaining application state across multiple nodes

GlusterFS

Next I had to find a way to maintain Home assistant's configuration across multiple nodes at the same time. I have a whole slew of configuration files which need to be kept up-to-date in order for automations to continue to work. Home assistant also keeps a few key files which it uses to maintain its own internal state of entities it knows about. I didn't want to rely on a single NAS since the whole reason for going down this path was to remove single points of failure.

I eventually found GlusterFS, a self-hosted distributed file store. I installed this onto my primary server, an Intel NUC, and a few Raspberry Pis I had left around from a few other projects. I peered them together with gluster peer probe <consul node hostname> and setup a replicated volume to ensure that each node had all the files it needed locally and would sync file operations to the cluster if any one of them changed. It's important to note that I am not going for a highly available installation of Home Assistant but instead a highly resilient installation of Home Assistant. At least on a per-node basis.

I mounted the replicated volumes to the same places on each node and copied over the required Home Assistant configuration into a service specific directory. I went ahead and logged into the other machines just to make sure that everything was looking golden and sure enough all the files were there.

Rescheduling workloads on nodes which are no longer available

Nomad

Enter Nomad, the task scheduler I didn't know I needed. Nomad's entire role in this setup is to know how much capacity each node has, and pack workloads efficiently onto the cluster as a whole. It also notices when workloads are not running on nodes and can reschedule them to run else where.

Setup for nomad looks a lot like the setup for Consul, there is a few things I thought I'd call out. Now since I have a DigitalOcean instance and that obviously doesn't have direct access to the devices within my home, it is connected via Tailscale to expose the service publicly. If you are familiar with Cloudflare tunnel, this is essentially a way of making a private version of that entire service. I then use this to expose any of the services that I want to be made available publicly.

I set up a Home Assistant nomad file that constrains the job to only run on nodes tagged with the "home" region. I then expose a service as "homeassistant" which lets Consul know about the service and in turn lets Traefik know about the service to expose.

job "homeassistant" {
    # ...
    group "homeassistant" {
        count = 1
        constraint {
          attribute = "${meta.region}"
          value = "home"
        }
        network {
            port "homeassistant" {
                to = 8123
                static = 8123
            }
        }
        task "homeassistant" {
            driver = "docker"
            config {
                network_mode = "host"
                image = "homeassistant/home-assistant:latest"
                privileged = true
            }
            service {
                name = "${TASK}"
                port = "homeassistant"
                tags = [
                  "traefik.enable=true"
                ]
            }
        }
    }
    # ...
}

Now I am well aware that maintaining a whole cluster of machines for a few the pleasure of turning a few light bulbs on and off but if it was about turning on and off lights then I would have stopped this whole endeavour long ago. I'll have a few other posts which go into more detail about each aspect of the setup considering Home assistant has some hardware constrains within my installation like Zigbee radios which always need to be available.