LVM2 Extent Shuffle
Say you have yourself a large mdadm device /dev/md0
, and you've made it into a big LVM volume. You partition it into a couple of modestly-sized LVM logical volumes. Because you're drunk on the power of large disk storage, you start growing and shrinking your logical volumes with reckless abandon.
You're looking to shrink your RAID device to free up some drives for other purposes, so you do something like
sudo pvresize -t -v --setphysicalvolumesize 6T /dev/md0
to test a resize, and whammo:
Resizing physical volume /dev/md0 from 2141183 to 1572863 extents.
/dev/md0: cannot resize to 1572863 extents as later ones are allocated.
But, you say, I totally have only 5T of that LVM volume allocated! There's a whole 1TB of slack space!
Ah, but then you run pvdisplay -m
and weep, as the last of the physical extents are allocated, but there's plenty of free space in the middle of your LVM physical volume:
--- Physical Segments ---
Physical extent 0 to 114688:
Logical volume /dev/filebox/timemachine
Logical extents 534527 to 649215
Physical extent 114689 to 524287:
FREE
Physical extent 524288 to 589823:
Logical volume /dev/filebox/filerecovery
Logical extents 786432 to 851967
Physical extent 589824 to 820223:
FREE
Physical extent 820224 to 1606655:
Logical volume /dev/filebox/filerecovery
Logical extents 0 to 786431
Physical extent 1606656 to 2141182:
Logical volume /dev/filebox/timemachine
Logical extents 0 to 534526
Normally, the solution would be to use pvmove /dev/md0 -n timemachine
to evict all of the physical extents for timemachine
to another physical volume, but you don't have any more volumes, which is why you were trying to shrink your raid array in the first place!
Luckily, a great blog post I found describes how to shuffle your extents on the same physical volume:
sudo pvmove /dev/md0:1606656-2016254 /dev/md0:114689-524287 --alloc anywhere
This command allows you to move arbitrary extents from one location to another (assuming the destination is free). Repeated applications of this command will allow me to open up the end of the physical extents, which will then allow me to shrink the RAID device, then I can reshape the RAID device to free up a couple of devices.