In my previous years of operating a small homelab, I made many mistakes and neglected some aspects which resulted in a very fun homelab although very far from the standards of what I saw and came to expect in professional environments.

A fine example of past mistakes is the way I handled storage.

hard_drive


Storage

I have always treated storage like a very secondary part of my work. While building my previous homelab I operated according to the following philosophy: Just plug in a couple of HDDs with random sizes as you go and make sure you never run out. what could go wrong? Fast forward two years, I was managing a herd of virtual machines and containers, they worked just fine on most days, but they had several issues which became more and more apparent with time.

One of these main issues was, as you would expect related to performance, booting a single virtual machine would sometimes take several minutes, this started to become quite intolerable. Add to that the fact that I had a set of different HDDs (1x4TB + 1x2TB + 1x1TB) configured in quite a messy manner, and having to resize certain VM disks was becoming a hellish endeavor.

So when building this homelab I figured I should invest some time looking into how to set up storage properly before even installing Proxmon VE on the machine. The setup I wanted to go for and which made the most sense to me was one where I had two different classes of storage, that way I could save some money but still have slightly better performance when needed.


Planning phase

My plan was to have the following:

  • Some SSDs for running Operating Systems and anything else I expected to require high I/O performance, for this part I did not care about redundancy but cared quite a bit about performance.

  • Some HDDs for storing data which did not require fast access, but I did need at least some basic redundancy since this might contain some personal info, some backups, etc…

First tier: SSD storage

The plan was to have 2 SSDs on RAID0 which was exactly what I was looking for, good performance even if at the expense of redundancy, of course before coming to this decision I had to review some concepts I hadn’t dealt with since college. This article was a good way to refresh my memory.

ssd_first

I went ahead and bought 2 mid-range 480GB SSDs. Having received both SSDs, I started looking into how to set up RAID0 in BIOS on my machine and how it would work with Proxmox.

I was all excited about setting my first semi-proper storage configuration when I found these PVE forum posts:

Now that RAID0 no longer was an option, I had to look for an alternative and after reviewing the storage options/possibilities in PVE. Having done that, the best I could hope for without investing too much time would be to have both SSDs accessible through a single LVM volume group and then exposed as an LVM thin pool which would look something like this:

ssd_second


Having figured this part out (at least theoretically), let’s move on to the next one.

Second tier: HDD storage

For the same reasons as previously mentioned for the SSD storage, we cannot make use of BIOS RAID. This means now we have to figure out another way to set RAID 1 configuration for these two disks. The most convenient way of setting this up would be using ZFS. ZFS provides a convenient way of setting up a “soft” RAID

hdd_second


Now having made a plan for a two-tier storage system, let’s try to implement that and hope that no issues will pop up along the way.


Implentation phase

First tier: SSD storage

The first thing we have to do here is of course to install the base image of PVE. For that purpose, we will slice a 150 GB partition from /dev/sda (/dev/sda1) and we will worry about the rest later on. I am not going to cover the base installation of a Proxmox virtual environment since that’s pretty straight forward and I plan on using all the default options anyway.

disks


As shown above, we have our operating system functioning and all storage devices connected properly:

partitions


SSD disks: /dev/sda and /dev/sdb

HDD disks: /dev/sdc and /dev/sdd

Now the next thing we need to do is to create the partitions needed for the LVM volume group, as seen above we now have /dev/sda4 and /dev/sdb1 which make up most of our SSD storage, we still need to make them accessible in a unified way to Proxmox.

We can do that by creating a volume group out of these two volumes. Doing this is pretty straightforward, we just need to use the vgcreate command and provide a name to the volume group we want to create and a list of physical volumes we want to use:

vgcreate ssd_storage /dev/sda4 /dev/sdb1

vgdisplay


We now have an LVM Volume group and would like to create an LVM Thin pool to use for our future VMs. This pool can be created by a single command:

lvcreate -l 100%FREE --poolmetadatasize 1024M --chunksize 256 -T -n ssd_storage_thin ssd_storage

This command will create a pool named ssd_storage_thin with spans over 100% of the space available in the ssd_storage volume group. Using the pvesm (Proxmox VE Storage Manager) tool, we can list all LVM volume groups and then scan the returned volume group for thin pool, and voila:

pvesm


And this how the final result looks like:

ssd_third


Second tier: HDD storage

Now when it comes to creating a Raid 1 ZFS pool, it is much easier than it sounds like. All I had to do was run the following command to create the pool from disks sdc and sdd:

zpool create hdd_storage mirror /dev/sdc /dev/sdd

zpool_mirror


And this is what the final state looks like:

hdd_third


And there it is. We now have both the fast and the slow storage providers configured in our Proxmox server as an LVM thin pool and a ZFS pool respectively:

pvesm1


The same state is also visible from the UI:

storage_ui


Wrapping up


Considerations

There are a few things here I can think of which can come back to haunt me a few months down the line:

  • The partition where the main distro is installed is quite small (~50 GB), I have a feeling this is gonna make it back in the future as a blog post on its own.
  • I have no idea how easy extending these storage pools will be, and that is something else for my future self to worry about.
  • For now, all storage used by the homelab is contained within the same machine, which will probably limit my ability to extend storage down the line.
  • One cannot think they’ve done a good job at setting up storage without actually having backup and restoration plans, but of course, it is too early to consider this.
  • This walkthrough is in no way constitute a set of best practices or industry standards. I am no expert at managing infra and even less of a storage expert, so the content of this blog post will change as I move forward and test the provisioning times and performance in the future. Depending on the relative size of the issues faced, I might either make adjustments to this existing post or make a new one for refactoring/ correction purposes.

Immediate next steps

Having set up a decent storage backend, the next step I can think of here is doing some Disk I/O performance tests to make sure it reflects what is expected of it and also to have a baseline to compare to in case of future bottlenecks/ issues.