Kestra’s Proxmox Plugin: Create a Workflow to Easily Spin-Up VMs

Manually spinning up VMs in Proxmox gets old fast. In this video, I’m using Kestra (an open-source workflow automation platform) to automatically clone a VM template and start it, all without touching the Proxmox UI! You’ll see the process of buiding the workflow, setting up the Proxmox API – and more! In this step-by-step video, you’ll be able to build this for yourself.

Thanks to Kestra for supporting this tutorial! You can self-host it and use it to automate entire workflows , so check it out!

Video-specific Links

Check out Kestra
Official blog post for this video
Kestra’s Proxmox VE Plugin
Get up to speed with Proxmox
Haven’t installed Kestra yet? If not, check out the Getting Started Tutorial

What is Kestra?

First, let’s do a brief overview of what Kestra is. I won’t spend long on this, since my previous video on Kestra covered it in full – but just in case you haven’t seen that one, here’s a quick rundown to set the foundation for the rest of this video.

Kestra is a tool for automating entire workflows. Normally, when we think automation, tools like Chef, Puppet, and Ansible come to mind – server configuration management. Kestra can handle that too, as we’ll see shortly, but it’s built around something bigger: orchestrating entire workflows, not just configuring servers.

Today, I’ll show you Kestra spinning up Proxmox servers – but that’s just the start. Later on, I might want it to automatically tear servers down when I’m done, send me an email when one’s created, trigger backups, and more. But it’s not just about servers.

Kestra has a huge library of plugins, so you can automate just about anything. Maybe later I’ll want it to automatically post on social media every time I publish a new video. Or build a workflow to handle spam emails, another for VPS backups, and so on.

The idea is you start small – automate one thing – and over time add more workflows that cover more of your needs. Bit by bit, you’ll find yourself less overwhelmed by things that constantly need your attention.

Kestra is also open-source and easy to set up. I covered the whole process in the previous video, but the short version is: run the Kestra container, and that’s about it.

What will you need to follow along?

Let’s go over what you’ll need if you want to follow along. The requirements are pretty minimal, but it’s worth understanding the foundation before we dive in.

The first is a Proxmox server, since that’s what we’ll be integrating with Kestra. That kind of goes without saying, but worth mentioning anyway.

You’ll also need a Kestra server. As I mentioned earlier, I covered the full installation process in the previous video, so I won’t repeat it here – feel free to watch that one and come back once you’ve got it set up. My preferred method is Docker Compose, so the process is fairly straightforward.

As for me, I set Kestra up a bit differently – I run mine as a VPS instance on Linode. Since my Proxmox server is physically in my office, that meant I needed some kind of tunnel between my home network and Linode, so I used Tailscale for that. Setting up that kind of tunnel to a cloud provider is beyond the scope of this video, but it’s something I could cover separately. None of that hybrid setup is required here, though – you can just as easily run Kestra on a local machine or a Proxmox VM. I went with the cloud route because I eventually want Kestra managing cloud resources too, so it made sense to host it there from the start.

Overall, though, you don’t need much. A Proxmox server and Kestra installed somewhere – that’s really it.

Project Setup

Like I mentioned earlier, we’re about to start a hands-on lab where you’ll build a Kestra workflow to automate VM deployment on Proxmox. Before we dive in, though, I want to give you a quick overview of what we’ll be doing – think of this section as a syllabus for the hands-on part of the video.

At this point, you should have a Proxmox server and Kestra installed somewhere that can reach it on your network. If applicable, make sure you’ve set up firewall rules ahead of time so the two can communicate. If you haven’t installed Kestra yet, you’ll find a link in the description below to the video that walks through it – just follow the installation section there and you’ll be in good shape.

From there, here’s how it’ll play out. We’ll start in Proxmox, laying the groundwork the automation needs – creating a template Kestra can use, and setting up the authentication and permissions that let Kestra talk to Proxmox in the first place.

After that, we’ll generate and apply the variables Kestra needs for the automation, which means editing our Kestra container a bit. It’s quick, though – this part should only take a few minutes.

Finally, we’ll start writing our flow within Kestra and test it out. If all goes well, you should see a virtual machine spring to life on Proxmox as soon as you run it.

As for difficulty, I’d call this one fairly easy. If you get stuck, double-check you’ve added everything I show you – most issues come down to a missing permission somewhere. Take your time, and you should have the whole thing done in about 20 minutes.

Let’s dive in!

Proxmox Setup

We’ll start our project in Proxmox, where we’ll create the foundational resources Kestra will use later – authentication, permissions, and a template. Let’s start with the template.

As you probably already know, a template in Proxmox lets you set up an operating system ahead of time – install your favorite Linux distro, tweak a few settings, and then use that as the basis for any future VMs you create.

The first decision is which distro to use, and this is actually one of the fun parts – it really doesn’t matter, so feel free to get creative. Go simple with Ubuntu or Fedora, or go wild and set up CachyOS. The distro genuinely makes no difference here.

Once you’ve picked a distro, grab the URL to its ISO file – just track it down on the distro’s site and copy it.

Back in Proxmox, we can download the ISO directly: click the storage pool for your chosen node, click ISO Images, then click Download from URL. Paste the URL in, click Query, and then click Download – and go grab a snack. (Or two, if your internet connection is slow.)

That covers the first thing our template needs. Next, we’ll use that ISO to create a new VM – the same way you’d create any other VM. You can choose whatever settings make sense for you, but I’ll give you a few suggestions.

First, turn on the “Start at boot” option – otherwise any server created from the template will default to shut down. Also use a low amount of memory and CPU cores; since a template can spawn VMs with all sorts of different resource needs, it’s better to start low. I’ll set memory to around 2GB and give it 2 CPU cores – very modest specs, but you can always bump those up on any VM you create from here. If you set the numbers too high instead, it’s easy to forget to scale them back down on a VM that doesn’t need them, so it’s best to start small and increase as needed while you tune things.

Also consider setting the CPU type to “host” – this exposes the underlying CPU’s features to the VM and generally improves performance. Skipping this can be especially problematic for workloads like hardware encoding, but in general, it’s the way to go.

Just a few more: for storage, set the SCSI controller to VirtIO SCSI single, which also helps performance. Also enable the Qemu Agent option – we’ll be installing the actual agent inside the VM later, but it only works if this checkbox is enabled here too. The agent lets Proxmox communicate more effectively with the VM. Finally, if your storage pool is on SSDs, consider enabling TRIM support.

That covers the settings. At this point, set the VM to boot from the ISO you downloaded, and go through the setup process for your chosen distro.

Once the install finishes, shut the VM down and remove the virtual CD drive – you likely won’t need it again unless you plan to attach another CD image down the road. If not, just remove it.

Now we’ll begin converting the VM into a template. Start it back up, and the first thing I’d recommend is running through some best practices. I have a video on this channel covering the first things you should do on any new Linux server build – installing updates, setting the hostname, and more. Apply those best practices in the template now, so you won’t have to repeat them every time you spin up a new VM from it.

Optionally, you can also set up automatic updates in the template. The process varies by distro, but once it’s in place, every VM cloned from the template will start updating itself automatically as soon as it comes online. I’ve got videos on this channel covering the process for a few distros – let me know in the comments if you’d like to see one for another.

We’re almost done with the template. Next, we’ll clear the machine ID. A full explanation of what that is would be beyond the scope of this video, but think of it as an identification number – every new Linux install gets one, randomly generated automatically in the background during setup.

The problem is, since we’re creating a template, every VM cloned from it would end up with the identical machine ID. The fix is simple: wipe it out now, and a fresh one gets generated automatically the next time the instance boots. So we’ll clear the machine ID before converting to a template, leaving it blank in every VM created from it. When a VM boots with a blank machine ID, it just silently generates a new one in the background – and you’ll never have to think about it again.

Of course, there’s more we could tweak in this template instance, but that’s an entire video on its own – and one I’ll likely produce at some point. For now, let’s finalize this one. We’ll shut it down:

sudo poweroff

Then all we have to do is right-click the VM and convert it to a template. Write down the exact name you gave it – we’ll need that name when we set things up in Kestra later.

With that, we’ve finished the template – which is actually the longest part of this whole project. Next, we’ll create a dedicated user for Kestra to use, since it’s always better to set up a service account per app rather than letting everything share the same access.

To do that, click Datacenter, then under Permissions you’ll find a Users option. Create the user there – it’s as easy as that.

By default, though, a user in Proxmox can do absolutely nothing – what we’ve created so far is just an identity, nothing more. To actually let this user perform tasks, we need to build out a set of permissions and scopes that control exactly what it’s allowed to do.

For that, we’ll create a Role – the container that holds the permissions we need. You’ll find a Roles option also under Permissions; create a new one there, give it a name, and start selecting privileges. In particular, you’ll need these:

Datastore.AllocateSpace
Datastore.AllocateSpace
SDN.Use
VM.Allocate
VM.Audit
VM.Clone
VM.PowerMgmt

Proxmox’s permission system can be a bit confusing, so let me recap what we’ve done so far. We created a user, which by itself gives us nothing but a name. The Role is where we add the actual privileges, so between the two we’ve now answered “who” and “what.”

Create resource pool

The next question is “where” – meaning, where exactly these privileges should apply. In Proxmox, that’s answered by creating what’s called a user permission. To add one, click Permissions, then create a new User Permission.

Here, we’ll select the Kestra user we created earlier, and use “/” for the path, and we’ll do the exact same thing again (but with an API User permission instead).

With that, we’ve covered two of the three things we need to set up in Proxmox. The next one is quick and simple – generating an API token.

To do that, navigate to API Tokens and click Add. Be sure to select the Kestra user, and uncheck Privilege Separation – leave that box checked, and the permissions we just set up won’t apply at all, even though we built them correctly.

Once you create the token, copy it somewhere secure – and that’s it, we’re done with Proxmox for now. All that’s left is configuring Kestra, but first let’s make sure you’ve got everything you need. You should have noted the username for the user we created, the role name, the template name, and the IP address or fully qualified domain name for your Proxmox server.

With Proxmox out of the way, let’s move over to Kestra and put all of this to use.

Configuring Kestra

At this point, Proxmox is ready to go, so let’s shift focus to Kestra. First, we’ll set up the secrets containing the values Kestra needs to communicate with Proxmox. Each of these needs to be a base64-encoded string – here’s how we’ll generate them.

The first one is for the IP address or FQDN of your Proxmox server. The command to base64-encode it looks like this:

echo -n "172.16.250.66" | base64

Take that output and keep it somewhere safe. Next, we’ll generate the encoded value for the username:

echo -n "kestra@pve" | base64

We’ll do the same for the token name (note: use full user path and single quotes this time):

echo -n 'kestra@pve!kestra_token' | base64

And finally, we’ll encode the token secret itself:

echo -n "<token secret>" | base64

Now that we’ve got all of those, let’s hand them over to Kestra. As you may recall from my previous video, I set Kestra up via Docker Compose, so all we need to do is open that compose file and add these values under the environment section, like this:

environment:
  SECRET_PROXMOX_HOST: "<Base64 value for the IP/FQDN>"
  SECRET_PROXMOX_USERNAME: "<Base64 value for the Kestra user we created>"
  SECRET_PROXMOX_TOKEN_ID: "<Base64 of the API ID>"
  SECRET_PROXMOX_TOKEN_SECRET: "<base64 of the secret key>"

Once that’s saved, we can have Kestra pick up the new configuration:

docker compose up -d

No need to worry about wiping anything out – relaunching the container simply applies the new configuration on top of what’s already running.

And now, we’re ready to test it out!

Testing it Out

Now it’s time to see Kestra actually do its thing – we’re going to build the flow that instructs it to create a new Proxmox VM whenever we run it. To do that, we’ll navigate to the Flows section and create a new one. Give it a name, and we’ll start building.

Note: the code below is untested and theoretical for now – I’ll fine-tune it and film the working version as I talk through it.

id: proxmox_clone_from_template
namespace: learnlinuxtv
tasks:
  - id: clone_vm_from_template
    type: io.kestra.plugin.proxmox.vm.Clone
    host: "{{ secret('PROXMOX_HOST') }}"
    node: pve-demo
    username: "{{ secret('PROXMOX_USERNAME') }}"
    tokenId: "{{ secret('PROXMOX_TOKEN_ID') }}"
    tokenSecret: "{{ secret('PROXMOX_TOKEN_SECRET') }}"
    vmName: ubuntu-template
    newId: 201
    newName: my-tutorial-vm
    full: true
  - id: start_vm
    type: io.kestra.plugin.proxmox.vm.Start
    host: "{{ secret('PROXMOX_HOST') }}"
    node: pve-demo
    username: "{{ secret('PROXMOX_USERNAME') }}"
    tokenId: "{{ secret('PROXMOX_TOKEN_ID') }}"
    tokenSecret: "{{ secret('PROXMOX_TOKEN_SECRET') }}"
    vmName: my-tutorial-vm

And there it is – as soon as I ran it, a new instance sprang to life. This is going to help me out a great deal: all I have to do is tell Kestra to spin up a new Proxmox VM anytime I’m starting a tutorial, and it will. Once I get this fully into production, I’ll build out templates for each of the distros I cover, preconfigured with everything I need before I even start recording. I’ll also add Inputs, so instead of a separate flow for every distro, I’ll just choose one from a single flow.

This is a good example of what life looks like with Kestra: you start small, add things over time, and before long you’re automating more than you expected. Automation is always an ongoing project, but the more you build, the more time you get back to focus on what matters most – for me, that’s making more videos!