Using LVM For Proxmox (2024)

This post describes everything you need to know about using LVM in Proxmox.

Thursday, January 19, 2023

Tags:

proxmox

If you are trying to use Proxmox to set up a homelab for the first time, it must be very confusing to see all the storage related terms, e.g., LVM, LVM-Thin. This post talks about everything you need to know to understand how Proxmox uses LVM as its storage infrastructure.

LVM Basics

LVM (Logical Volume Management) is actually a Linux technology. It is not Proxmox proprietary. Proxmox can leverage LVM as its storage infrastructure, but LVM itself is not part of Proxmox.

Let’s try to understand the basics of LVM, and then we try to understand how Proxmox uses it.

What Is LVM?

The overall goal of LVM is to create a virtual storage layer on top of the physical storage layer, so that the storage usage can be more flexible.

For example, if you have two 1TB disks, but you want to use them together as your root directory. LVM provides a solution to this.

How Does LVM Work?

The following image presents the overall architecture of LVM. (The image comes from wikipedia)

There are a couple of layers involved (from bottom to top):

On the bottom layer, we have the physical disk itself. These are your hard drives.
Above the hard drives, we have partitions. Each hard drive is splitted into a few partitions.
PV (physical volume) is a LVM concept. It is a representation of a physical volume. It is mapped to a single disk partition.
VG (volume group) is also a LVM concept. A VG contains a couple of PVs.
LV (logical volume) is a logical volume, which is basically a virtual volume.
On top of each LV, a file system can be created, and the LV device can be mounted to an accessible directory.

In this design, logical volume does not have to tie to a single physical disk. Logical volume can be created over multiple physical disks.

Installation LVM

On debian and ubuntu, you can do

sudo apt install lvm2

How To Use LVM?

Here are some commands to manage PV, VG and LV.

# Create a PVpvcreate <disk-device-name># Remove a PVpvremove <disk-device-name># List all PVspvs# Create a VGvgcreate <vg-name> <disk-device-name># Remove a VGvgremove <vg-name># List all VGsvgs# Create a LVlvcreate -L <lv-size> -n <lv-name> <vg-name># Remove a LVlvremove <vg-name>/<lv-name># List all LVslvs

An Example

For example, I have a empty /dev/sdb device,

swe@ubuntu-server:~$ lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTloop0 7:0 0 63.3M 1 loop /snap/core20/1778loop1 7:1 0 141.4M 1 loop /snap/docker/2285loop2 7:2 0 55.6M 1 loop /snap/core18/2667loop3 7:3 0 55.4M 1 loop /snap/core18/2066loop5 7:5 0 67.6M 1 loop /snap/lxd/20326loop6 7:6 0 49.6M 1 loop /snap/snapd/17883loop7 7:7 0 49.8M 1 loop /snap/snapd/17950loop8 7:8 0 91.9M 1 loop /snap/lxd/24061loop9 7:9 0 139.4M 1 loop /snap/docker/2343sda 8:0 0 32G 0 disk├─sda1 8:1 0 1M 0 part└─sda2 8:2 0 32G 0 part /sdb 8:16 0 32G 0 disksr0 11:0 1 4M 0 romsr1 11:1 1 1024M 0 romswe@ubuntu-server:~$

let’s create a PV on it

swe@ubuntu-server:~$ sudo pvcreate /dev/sdb Physical volume "/dev/sdb" successfully created.swe@ubuntu-server:~$ sudo pvs PV VG Fmt Attr PSize PFree /dev/sdb lvm2 --- 32.00g 32.00gswe@ubuntu-server:~$

and then create a VG on the PV

swe@ubuntu-server:~$ sudo vgcreate test-vg /dev/sdb Volume group "test-vg" successfully createdswe@ubuntu-server:~$ sudo vgs VG #PV #LV #SN Attr VSize VFree test-vg 1 0 0 wz--n- <32.00g <32.00gswe@ubuntu-server:~$

and then create a LV on the VG

swe@ubuntu-server:~$ sudo lvcreate -L 10G -n test-lv test-vg Logical volume "test-lv" created.swe@ubuntu-server:~$ sudo lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert test-lv test-vg -wi-a----- 10.00gswe@ubuntu-server:~$

Now we can create a file system on the LG, and mount it for real usage:

swe@ubuntu-server:~$ sudo mkfs.ext4 /dev/test-vg/test-lvmke2fs 1.45.5 (07-Jan-2020)Discarding device blocks: doneCreating filesystem with 2621440 4k blocks and 655360 inodesFilesystem UUID: 5c329fcf-76ec-450a-8d96-dfb816538e3eSuperblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632Allocating group tables: doneWriting inode tables: doneCreating journal (16384 blocks): doneWriting superblocks and filesystem accounting information: doneswe@ubuntu-server:~$ cd /swe@ubuntu-server:/$ sudo mkdir mountswe@ubuntu-server:/$ sudo mount /dev/test-vg/test-lv /mountswe@ubuntu-server:/$ cd mount/swe@ubuntu-server:/mount$ lslost+foundswe@ubuntu-server:/mount$

Now, we have an ext4 filesystem on the LV:

swe@ubuntu-server:/mount$ lsblk -fNAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINTloop0 squashfs 0 100% /snap/core20/1778loop1 squashfs 0 100% /snap/docker/2285loop2 squashfs 0 100% /snap/core18/2667loop3 squashfs 0 100% /snap/core18/2066loop5 squashfs 0 100% /snap/lxd/20326loop6 squashfs 0 100% /snap/snapd/17883loop7 squashfs 0 100% /snap/snapd/17950loop8 squashfs 0 100% /snap/lxd/24061loop9 squashfs 0 100% /snap/docker/2343sda├─sda1└─sda2 ext4 761b4178-9020-4642-ac79-d172a2a0a06d 21G 28% /sdb LVM2_member biMIK2-WI2D-ypLl-tVGw-p8hZ-CynW-vo8swy└─test--vg-test--lv ext4 5c329fcf-76ec-450a-8d96-dfb816538e3e 9.2G 0% /mountsr0 iso9660 cidata 2023-01-12-13-13-29-00sr1

How Does Proxmox Use LVM?

Now you have a basic understanding of LVM. So how does Proxmox use LVM as its storage infrastructure?

In a few words, Proxmox uses LVs for two purposes:

Use LVs as the client OS disks.
Use a LV as the host OS root directory.

For example,

root@host3:~$ pvs PV VG Fmt Attr PSize PFree /dev/nvme0n1p3 ssd lvm2 a-- <1.82t 738.50groot@host3:~$root@host3:~$ vgs VG #PV #LV #SN Attr VSize VFree ssd 1 13 0 wz--n- <1.82t 738.50groot@host3:~$root@host3:~$ lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root ssd -wi-ao---- 96.00g vm-1001-disk-0 ssd -wi-ao---- 16.00g vm-2000-cloudinit ssd -wi-ao---- 4.00m vm-2000-disk-0 ssd -wi-ao---- 16.00g vm-2002-cloudinit ssd -wi-ao---- 4.00m vm-2002-disk-0 ssd -wi-ao---- 16.00g vm-5000-disk-0 ssd -wi-ao---- 200.00g vm-7000-disk-0 ssd -wi-a----- 16.00g vm-7000-disk-1 ssd -wi-a----- 500.00g vm-8000-cloudinit ssd -wi-a----- 4.00m vm-8000-disk-0 ssd -wi-a----- 32.00g vm-8000-disk-1 ssd -wi-a----- 200.00g vm-8001-disk-0 ssd -wi-a----- 32.00groot@host3:~$

The above commands shows that on one of my Proxmox servers, I have a PV, /dev/nvme0n1p3, a VG, ssd, and a few LVs. Each of the client OS has its own LVs.

Using LV as the host OS storage device makes a lot of sense, which is the whole point of LVM.

LVM-Thin

By default, Proxmox does not use LVM. Instead, it uses LVM-thin. The difference between LVM-thin and LVM is that:

LVM allocates your storage blocks when you create LVs. With LVM, your LV’s storage is guaranteed for your use.
LVM-thin allocates your storage blocks when you write data to LVs. LVM-thin provides a higher disk utilization, but it can’t guarantee you can use all your LV storage.

←Previous
Next→