Automating VM creation in Proxmox VE

linux_benchmarking

Automating virtual machine creation in Proxmox Virtual Environment

In this article, I’m going to cover the process of creating a virtual machine in PVE in a purely automated way. The tool we’re going to be using throughout this process is going to be Terraform and more specifically, the Telmate Terraform provider. The provider seems to be mature and being worked on an actively enough basis to be reliable.

But before we start creating VMs, there is something else we need to take care of first.

Images in proxmox

By default, proxmox provides a way to manually upload ISO images and then power on a vm using said images, proceed through a regular installation which can be quite a time consuming, and then you have your virtual machine.

Once you’ve been through this process once, you can avoid having to do it again by converting this Virtual machine into a template which you can then quite easily clone into as many virtual machines as you want.

This doesn’t sound so bad, but imagine having to go through this process for every single version of every operating system or distribution you want to use. It doesn’t sound very scalable to me.

So before creating our first virtual machine, let us first automate the creation of said template without having to go through any manual steps.

The process of doing this is not that difficult nor is it complicated, it can be summed up in the following steps:

Step #1: Select and download an image of your Linux distro of choice but in a cloudimg format.
Step #2: Use this image to create a virtual machine in proxmox.
Step #3: Convert this virtual machine into a template.

Let’s start automating:

As an example here, let’s try to create a template for a Debian 11 (“bullseye”) image.

The first thing we need to do is find a compatible image, our only requirement regarding the image is for it to be a cloud image. With Debian being quite mainstream and available, finding cloud images is not [much of a problem]{:target=”_blank”}.

qcow2_

As we can see above, it seems like every image is provided in either a Raw or a QCow2 format, and since both of them could be imported as volumes in qemu we need to make a decision, it seems that on average Raw images are much smaller in size but on the other hand QCow2 seem to have a reputation of adding some overhead especially if used inefficiently, whereas Raw images provide better performance than QCow2 at the expense of disk space and some missing features. For now, this is enough to convince me of sticking to raw format.

Now we can finally get started with the fun stuff, creating virtual machines:

wget https://cdimage.debian.org/cdimage/cloud/bullseye/latest/debian-11-generic-amd64.raw # Downloading Raw image.


# Creating virtual machine
qm create 9999 --memory 2048 --net0 virtio,bridge=vmbr0 # Configuring the memory and networking for the virtual machine.
qm importdisk 9999  debian-11-generic-amd64.raw ssd_storage_thin_pool # importing the local raw file as a disk.
qm set 9999 --scsihw virtio-scsi-pci --scsi0 ssd_storage_thin_pool:vm-9999-disk-0 # Attaching the imported disk as a scsi device.
qm set 9999 --agent enabled=1,fstrim_cloned_disks=1 # Enabling Qemu Guest Agent and enabling the execution of fstrim after moving a disk or migrating the VM.
qm set 9999 --name debian-11-template # Naming the template.

# Creating and attaching the cloudinit volume
qm set 9999 --ide2 ssd_storage_thin_pool:cloudinit 
qm set 9999 --boot c --bootdisk scsi0
qm set 9999 --serial0 socket --vga serial0

qm template 9999 # Converting the VM 9999 into a template

And there it is, the template has been created:

qcow2_

Now let’s try cloning it and see how it goes:

qcow2_

The cloning operation went perfectly fine. A few things to note:

No credentials were set so the VM was not in a usable state but that was expected since we didn’t provide one.
The clone virtual machine inherits the clone’s specifications including the memory, disk size, and CPU settings.

Now it is time to try using the template we just created to spin up a virtual machine using Terraform, thankfully as a place to start we have this great example provided by Telmate as well as this well-written piece of documentation.

But before we can do that we first need to set up our local terraform environment. We start by installing the terraform CLI locally, there is already an existing walkthrough for that so there is no need for me to expand on that.

The next step would be to create a provider.tf file locally. This file is supposed to contain a reference to the provider we’re using to connect to proxmox as well as its configuration, it should look something like this:

terraform {
  required_version = ">= 1.0.3"
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "== 2.7.4"
    }
  }
}


provider "proxmox" {
  pm_api_url        = "https://IP_ADDRESS/api2/json"
  pm_user           = "...."
  pm_password       = "...."
  pm_parallel       = 1
  pm_tls_insecure   = true
  agent             = 1
}

I could at this point initialize my local terraform environment and start deploying VMs but there is one improvement that could be made here. We have a few values here which might be better off in a separate file dedicated for terraform variables. That way provider configuration and environment configuration will be stored in two separate files, the setup will end up looking like this:

variables.tf
----

variable "pm_api_url" {
  default = "https://x.x.x.x:8006/api2/json"
}

variable "pm_user" {
  default = "...."
}

variable "pm_password" {
  default = "...."
}

---

provider.tf
----

terraform {
  required_version = ">= 1.0.3"
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = ">= 2.7.4"
    }
  }
}

provider "proxmox" {
  pm_api_url        = var.pm_api_url
  pm_password       = var.pm_password
  pm_user           = var.pm_user
  pm_parallel       = 1
  pm_tls_insecure   = true
}

---

Well doesn’t that look much neater?

Now if we run ‘terraform init’ in the same directory, we should be able to see that the initialization was successful:

└─ $ ▶ terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of telmate/proxmox from the dependency lock file
- Using previously-installed telmate/proxmox v2.7.4

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

We are finally ready to create our first virtual machine:

resource "proxmox_vm_qemu" "test-debian-vm" {
    name = "test-debian-vm"
    desc = "A test for using terraform and cloudinit"

    # Node name has to be the same name as within the cluster
    # this might not include the FQDN
    target_node = "ghima-node01"

    # The template name to clone this vm from
    clone = "debian-11-template"

    # Activate QEMU agent for this VM
    agent = 1

    os_type = "cloud-init"
    cores = 2
    sockets = 1
    vcpus = 0
    cpu = "host"
    memory = 4096
    scsihw  = "virtio-scsi-pci"
    bootdisk = "scsi0"

    disk {
        size = "10G"
        type = "scsi"
        storage = "ssd_storage_thin_pool"
        ssd = "1"
    }

    
    network {
        model = "virtio"
        bridge = "vmbr0"
        tag = -1
    }

    
    ipconfig0 = "ip=192.168.5.112/24,gw=192.168.5.1"
    ciuser = "john"
    cipassword = "doe"
    nameserver = "8.8.8.8"
    sshkeys = "....."
}

Now it’s just a matter of seeing what resources will be created/updated by the deployment of these changes using terraform plan and then deploying them using terraform apply.

One thing that seems a little bit odd is the time it is taking for the creation to be over 11 minutes, which is much slower than the time it takes for the machine to come up. This is probably due to the machine being unable to report back to Qemu and then Terraform that the creation is complete.

proxmox_vm_qemu.test-debian-vm: Still creating... [10m50s elapsed]
proxmox_vm_qemu.test-debian-vm: Still creating... [11m0s elapsed]
proxmox_vm_qemu.test-debian-vm: Still creating... [11m10s elapsed]
proxmox_vm_qemu.test-debian-vm: Creation complete after 11m16s [id=ghima-node01/qemu/106]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

This seems to be a known issue: https://github.com/Telmate/terraform-provider-proxmox/issues/325

This is probably because the qemu guest agent is absent from the VM. There are clear instructions here for how to install it, Although they do seem to require recreating the VM template.

reference: https://registry.terraform.io/modules/sdhibit/cloud-init-vm/proxmox/latest/examples/ubuntu_single_vm#install-qemu-guest-agent-on-ubuntu-cloud-image

Now I run the following on the proxmox machine:

apt-get install libguestfs-tools -y
virt-customize -a debian-11-generic-amd64.raw --install qemu-guest-agent

And then run terraform apply one more time to see how it goes:

proxmox_vm_qemu.test-debian-vm: Creating...
proxmox_vm_qemu.test-debian-vm: Still creating... [10s elapsed]
proxmox_vm_qemu.test-debian-vm: Still creating... [20s elapsed]
proxmox_vm_qemu.test-debian-vm: Still creating... [30s elapsed]
proxmox_vm_qemu.test-debian-vm: Still creating... [40s elapsed]
proxmox_vm_qemu.test-debian-vm: Still creating... [50s elapsed]
proxmox_vm_qemu.test-debian-vm: Still creating... [1m0s elapsed]
proxmox_vm_qemu.test-debian-vm: Still creating... [1m10s elapsed]
proxmox_vm_qemu.test-debian-vm: Creation complete after 1m19s [id=ghima-node01/qemu/104]

There it is! the new template solved the qemu agent issue and the VM now can report its status back to terraform in 1m19s.

Now that we found a way to properly create a template for a linux VM, let’s automate that into a small bash script:

#!/bin/bash
set -e

if [[ $# -ne 3 ]]; then
    echo "Usage: ./$0 PATH_TO_CLOUD_IMAGE PROXMOX_TEMPLATE_NAME PROXMOX_TEMPLATE_ID"
    exit
else
    export IMAGE_FILE=$1
    export TEMPLATE_NAME=$2
    export TEMPLATE_ID=$3
fi

export STORAGE_POOL="ssd_storage_thin_pool"

if ! command -v virt-customize &> /dev/null
then
    echo "virt-customize could not be found"
    exit
fi

apt install libguestfs-tools -y 

virt-customize -a $IMAGE_FILE --install qemu-guest-agent


qm create $TEMPLATE_ID --memory 2048 --net0 virtio,bridge=vmbr0
qm importdisk $TEMPLATE_ID $IMAGE_FILE $STORAGE_POOL
qm set $TEMPLATE_ID --scsihw virtio-scsi-pci --scsi0 $STORAGE_POOL:vm-$TEMPLATE_ID-disk-0
qm set $TEMPLATE_ID --agent enabled=1,fstrim_cloned_disks=1
qm set $TEMPLATE_ID --name $TEMPLATE_NAME


qm set $TEMPLATE_ID --ide2 $STORAGE_POOL:cloudinit
qm set $TEMPLATE_ID --boot c --bootdisk scsi0
qm set $TEMPLATE_ID --serial0 socket --vga serial0

qm template $TEMPLATE_ID

I think we’ve covered enough in this post. We have not only automated the process of creating a re-usable operating system template in Proxmox in a bash script, but we have also automated the creation of a virtual machine using terraform.

The next blog post is going to involve less practice and more design, I’m going to try to take a step back and make a sketch of how I want the target state of this homelab to look like and maybe get started on the networking part of things.