Proxmox Security Updates Made Easy – Unattended Upgrades Tutorial

Learn how to set up Proxmox unattended upgrades to automatically install security updates and patches on your Proxmox VE server. This complete tutorial covers everything from installation to configuration, ensuring your virtualization environment stays secure without manual intervention.

YouTube player

Important Considerations

In this first section, I’m going to go over some important considerations to keep in mind before we actually set up unattended-upgrades (which is the service that we’ll be using to automate updates).

The first of these is that you may want to take a moment and audit which virtual machines and containers are set up to automatically start. Reason being, as part of the process of setting up unattended-upgrades, we’ll be setting a time for our server to automatically reboot to apply updates. If you have any VMs or containers that aren’t configured to automatically start, you’ll end up waking up to some of your resources not being available.

Also, while we’re on the subject of configuring our Proxmox resources to automatically start, we should also consider the order that they start in. Sometimes, a server might need to be fully booted in order for another server to function. A common example of this is hosting a website – perhaps you’ll have two servers, one for the database, and another for the actual web site. If your web server ends up starting before your database server, it’ll generate a bunch of errors in your logs, complaining about the database server being down. In that scenario, it would be better if the database server was instructed to start before the web server.

A confusing aspect of these settings I want to make sure you understand is startup delay. This is an option that many people get stuck on. What you’re doing here is setting a particular number of seconds. And the number of seconds you enter here will force Proxmox to wait that number of seconds before it starts another VM. This means that this setting doesn’t impact the server you’re configuring it on, it has an effect on the next resource that Proxmox starts. For example, if I configure my debian test machine to have a startup delay of 30 seconds, that means that once this VM starts, Proxmox will wait 30 seconds before starting something else. Many people might think that the startup delay applies to the VM you’re configuring it on, but in actuality it only effects how long until Proxmox starts the next thing.

Thankfully, the Shutdown timeout is a lot easier to explain, and this has to do with how long Proxmox is willing to wait for the VM or container to finish shutting down before it gives up and kills it. The default is 180 seconds. So basically, whenever you reboot your Proxmox server, it’s going to send a signal to all VMs and containers to shut down. Once they all finish shutting down, then Proxmox will then reboot itself.

Now imagine what happens if one of your VMs freezes and totally locks up during its shutdown process, and it begins to hang forever. In that case, Proxmox itself would never reboot, because it would be waiting for an eternity for that VM to finish powering down. To protect against that, Proxmox will wait up to 180 seconds for a resource to finish shutting down when it’s instructed to do so. If 180 seconds passes without a resource powering down, Proxmox will assume that it’s locked up and will force it to turn off. Going back to this setting here, if you want to override the default of 180 seconds, you can enter a different number here.

All in all, everything I’ve gone over so far in this session has to do with ensuring your VMs and containers automatically start, and start in the right order. But before we dive in, there’s a few more things you might want to consider, that are completely optional but recommended.

And one of these is that you should probably set up outbound email, so that way you can get an email report anytime updates happen. These reports will give you piece of mind when they report success, as well as alert you to issues if the process fails for some reason. This isn’t required or anything, but it’s a good idea. Also, not everyone is going to have an outbound email service to use, so you can skip this if it doesn’t pertain to you.

If you do decide to enable outbound email, you’ll do that by going go Data Center > Notifications (toward the very bottom of the list) > and then you’ll add a notification target. You can enter the details here for your email provider here, which gives you the ability to receive reports from your server.

My final tip for this section is to test whether or not your server is able to actually reboot. I know this sounds strange, but I’ve seen issues where there’s a Proxmox server that hasn’t been rebooted in a long time, and its boot configuration ended up being incorrectly configured at some point. If there’s any misconfiguration on your server, it’s a good idea to be aware of it now rather than waking up to finding that it never came back on. I’ve also seen issues with updates causing odd behavior, so believe it or not – it happens. So, make sure you reboot your Proxmox server at some point to test it out, and be sure to let your team know that you plan on doing this if you are working for a company.

Anyway, with all of that out of the way, let’s get unnattended-upgrades installed and configured.

Installing and Configuring unattended-upgrades

So now we’re going to get started and set this up.

The first thing we’ll do is access the shell for the server we’re setting this up on. If you have access to the server via SSH, you can do that. If not, you can access your servers shell by clicking on the Proxmox node itself on the left-hand menu, and then clicking on “Shell”. And now that you have access to run commands, I’ll go over what you’ll need to enter.

The first command will update our package repository index, just to make sure the package cache is up to date:

apt update

Next, we’ll install the package that will give us access to automatic updates. So, we’ll install the unattended-upgrades package:

apt install unattended-upgrades

After that, we’ll run the command you see on the screen right now to enable it:

sudo dpkg-reconfigure --priority=low unattended-upgrades

Finally, to ensure that the service is running properly, we’ll check its status:

systemctl status unattended-upgrades

Believe it or not, after running just those few commands we’re done with the process of enabling automatic updates. However, in order to get the most out of this service, we should tune and configure it to meet our needs. So, what I’ll do know is show you some useful tweaks you can make.

To configure it, we’ll navigate to the following directory:

cd /etc/apt/apt.conf.d
ls -l

First, we’re going to alter the origins pattern, which determine which repositories updates will be installed from. The thing is, when it comes to unattended-upgrades its recommended that you automate the installation of only packages that fix security problems, instead of having it update everything. The reason why this is important is because if you enable all repositories, then eventually a new version of Proxmox itself will be released, and if you enable all origins then you’ll suddenly find yourself on a completely new version of Proxmox. And this can be a real problem, because sometimes newer major versions of Proxmox will require manual intervention during the upgrade process, and without you being present to perform those steps, then you may end up experiencing issues. Whenever Proxmox does release a major new version, they’ll include a list of steps you might need to do manually in order to complete the process. Instead of dealing with that, we can instruct unattended-upgrades to only install security updates, and focusing on just those will prevent your server from being automatically updated to a newer major version.

The result here is that security updates from Debian will be applied automatically, while updates to Proxmox itself will not. Again, we want to avoid surprise major upgrades to Proxmox itself.

Continuing, there’s a few more options I recommend you tweak. The next one we’ll look at is Unattended-Upgrade::AutoFixInterruptedDpkg which we want set to true. Technically, it already is – but it’s commented out. Even though this is on by default, I prefer to be explicit so I’ll uncomment this option.

Next, let’s get a few optional settings out of the way that pertain to email alerts. Earlier, I mentioned it was recommended to set up an email provider so your Proxmox server can send you reports. If you decided to do that, then we need to configure unattended-upgrades where to send the reports to, and also how frequently. We’ll uncomment the Unattended-Upgrade::Mail setting, and within the double quotes you’ll add whatever email address you want reports sent to.

After that, the very next option is where we tell unattended-upgrades when we’d like to receive these reports. By default, it’s set to on-change – which means you’ll receive reports of both successes and failures. If you prefer to only be bothered when there’s a problem, you can set this to only-on-error. However, I think it’s better to always get reports even when there’s no problem – that way the successful reports will at least confirm that email is working.

Continuing, let’s look at the Automatic Reboot setting. This is set to false by default, but I highly recommend that you configure the server to automatically reboot after updates. Reason being, some security updates require a reboot of the server in order for the changes to take effect. If you don’t reboot your server every so often, then it’ll remain vulnerable to threat actors. I also recommend that you enable the option to reboot even when users are logged in, so stuck sessions don’t prevent the server from rebooting if it needs to.

Related, you can also choose when the server reboots after being updated. And this one is fairly important, because you’ll want to make sure the server is scheduled to reboot whenever it’s being used the least. If you’re running Proxmox in a Homelab, then just set this to a time when you’re normally sleeping. If you’re managing a Proxmox server at a company, then set the time to be within an acceptable maintenance window.

And that’s about it for the settings in this file, so we’ll save it and exit the editor.

The next thing we’re going to do is set the time when automatic updates happen. We’ve already changed the time when the server will reboot when we edited the previous file, but when it comes to actually kicking off the update process, we’ll configure that in a different place. To do that, we’ll run the following command:

systemctl edit apt-daily-upgrade.timer

What we have here is what’s called a systemd-timer. Systemd timers aren’t specific to Proxmox, it’s a part of the underlying Linux system that Proxmox runs on. I’ve covered systemd timers in other videos, so I won’t go into too much detail about them here. But the basic idea is that systemd timers are basically alarms, and when they go off they’ll execute a command. With the command we just entered, we are editing a systemd timer, this one in particular being the alarm (so to speak) that kicks off the update process.

Anyway, all we have to do here is update the time. I recommend setting the time to around 1 hour before the reboot time, so that way the server has more than plenty of time to complete the update process before it reboots. And although it goes outside the scope of this video, we can configure this to trigger during certain days as well, by editing the entire string. If you’re curious to learn more, then check out my systemd timer video. But in case you want to see examples of configuring the OnCalendar setting further, I’ll leave some helpful cards on the screen.

Override.conf:

[Timer]
OnCalendar=
OnCalendar=--* 03:30
RandomizedDelaySec=0

We’re almost done! There’s a few quick things we’ll do to ensure everything is working, and then we’ll be all set.

So, to wrap it up, we’re going to instruct the underlying Linux system that we’ve made changes to the files we updated, and we can do that by running this:

systemctl daemon-reload

Next, let’s restart unattended-upgrades itself to make sure the service picks up all the changes:

systemctl restart unattended-upgrades

After that, let’s check the status of unattended-upgrades:

systemctl status unattended-upgrades

As long as we didn’t make any mistakes with our config files, the service should be running. If not, just double-check everything from the previous steps, and then try restarting the service again.

Lastly, we’re going to run a special command to test the unattended-upgrades service even further. What we’re going to do is basically run a simulation, that will go through the entire update process – but it won’t actually install updates. You can think of this as a dry run, and in fact that’s part of the command:

sudo unattended-upgrade --dry-run --debug

Your output here won’t be quite the same as the real thing, but it does check things a bit and might give you an idea of what’s wrong if you encounter an issue.

Commands used in this video

Ensure the package cache is up to date

Run the following command to make sure APT has a fully updated cache:

apt update

Install the unattended-upgrades package

Use this command to install unattended-upgrades, the package that makes all of this work:

apt install unattended-upgrades

Enable unattended-upgrades

After installing the package, run this command to enable unattended-upgrades:

sudo dpkg-reconfigure --priority=low unattended-upgrades

Check the status of unattended-upgrades

Check the status of the unattended-upgrades service to ensure that it’s enabled and running:

systemctl status unattended-upgrades

Start and enable the unattended-upgrades service

If for some reason the previous command shows that unattended-upgrades is either disabled or stopped, this command will take care of both:

systemctl enable --now unattended-upgrades

Change into the apt.conf.d directory

The /etc/apt/apt.conf.d directory contains most of the files we’ll need to edit, so navigate to that directory:

cd /etc/apt/apt.conf.d

Edit the 20auto-upgrades file

Next, edit the 20auto-upgrades file to ensure that everything is enabled (each option should show 1):

nano 20auto-upgrades

Copy 50unattended-upgrades to a new file

Most of the configuration we’ll need can be found in the 50unattended-upgrades file, but it could be overwritten by an update at some point. Instead of editing that file directly, make a copy of it:

cp 50unattended-upgrades 99unattended-upgrades

Edit the 99unattended-upgrades file

Considering that 99unattended-upgrades has a higher number than 50unattended-upgrades, it will “override” it. We’ll edit that file to adjust the configuration to meet our needs:

nano 99unattended-upgrades

Edit the apt-daily-upgrade timer

A systemd timer (apt-daily-upgrade.timer) kicks off the update process. Run this command to adjust when it starts:

systemctl edit apt-daily-upgrade.timer

Reload systemd

Run the following command to “reload” systemd, which will pick up changes we’ve made to some of the config files:

systemctl daemon-reload

Restart the unattended-upgrades service

Restart unattended-upgrades itself in order to ensure it loads all of our changes:

systemctl restart unattended-upgrades

Check the status of unattended-upgrades

Check the status of unattended-upgrades and ensure that it’s still running:

systemctl status unattended-upgrades

Perform a “Dry Run” to test unattended-upgrades

To make sure everything is working properly, the following command will perform a “Dry Run” to simulate what will happen once an actual update triggers:

sudo unattended-upgrade --dry-run --debug