LVM2 Extent Shuffle

Say you have yourself a large mdadm device /dev/md0, and you've made it into a big LVM volume. You partition it into a couple of modestly-sized LVM logical volumes. Because you're drunk on the power of large disk storage, you start growing and shrinking your logical volumes with reckless abandon.

You're looking to shrink your RAID device to free up some drives for other purposes, so you do something like sudo pvresize -t -v --setphysicalvolumesize 6T /dev/md0 to test a resize, and whammo:

Resizing physical volume /dev/md0 from 2141183 to 1572863 extents. /dev/md0: cannot resize to 1572863 extents as later ones are allocated.

But, you say, I totally have only 5T of that LVM volume allocated! There's a whole 1TB of slack space!

Ah, but then you run pvdisplay -m and weep, as the last of the physical extents are allocated, but there's plenty of free space in the middle of your LVM physical volume: --- Physical Segments --- Physical extent 0 to 114688: Logical volume /dev/filebox/timemachine Logical extents 534527 to 649215 Physical extent 114689 to 524287: FREE Physical extent 524288 to 589823: Logical volume /dev/filebox/filerecovery Logical extents 786432 to 851967 Physical extent 589824 to 820223: FREE Physical extent 820224 to 1606655: Logical volume /dev/filebox/filerecovery Logical extents 0 to 786431 Physical extent 1606656 to 2141182: Logical volume /dev/filebox/timemachine Logical extents 0 to 534526

Normally, the solution would be to use pvmove /dev/md0 -n timemachine to evict all of the physical extents for timemachine to another physical volume, but you don't have any more volumes, which is why you were trying to shrink your raid array in the first place!

Luckily, a great blog post I found describes how to shuffle your extents on the same physical volume:

sudo pvmove /dev/md0:1606656-2016254 /dev/md0:114689-524287 --alloc anywhere

This command allows you to move arbitrary extents from one location to another (assuming the destination is free). Repeated applications of this command will allow me to open up the end of the physical extents, which will then allow me to shrink the RAID device, then I can reshape the RAID device to free up a couple of devices.

FileVault on secondary drives

With OSX Lion, Apple introduced their Whole Disk Encryption technology known as FileVault 2. Not to be confused with the old FileVault, which basically just wrapped up a home directory into a single encrypted disk image, this FileVault is a transparent encryption layer at the filesystem level.

More details on its use can be found in John Siracusa’s great Lion review, but my favorite feature is a rolling encryption setup, where an active and running volume can be reconfigured as an encrypted volume while the system still runs (as long as the volume you’re encrypting isn’t the boot volume).

To do so, use diskutil list on the terminal to view your active volumes. You’ll see something like:

/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *80.0 GB    disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:                  Apple_HFS Lion                    79.7 GB    disk0s2

All it takes to turn that pesky unencrypted volume into an encrypted one is:

diskutil cs convert disk0s2 -passphrase somethinglongandcomplicated

Lion will then convert the volume in the background. Hoorah! Now, you may wonder, when is this background encryption done? Luckily diskutil cs list will show you a nice healthy volume of information about all the volumes on your system. Instead of trolling through all the data, just run this guy:

diskutil cs list | egrep -B1 '(Converted).*B' | awk '{ print $3;}' | perl -e 'my $total = <STDIN>; die "Conversion done!\n" unless $total; chomp($total); my $done = <STDIN>; chomp($done); printf("%0.2f percent\n", 100 * $done/$total);'

and you’ll see something like:

26.93 percent

You can leave drop that command into something like GeekTool and have that on your desktop so you can do a happy dance when the disk encryption process is completed.

PS: For those of you who have installed watch using homebrew, here’s the watchified version of that command:

watch -n 20 "diskutil cs list | gwatch -n 20 "diskutil cs list | egrep -B1 '(Converted.*B)' | awk '{ print \$3;}' | perl -e 'my \$total = <STDIN>; die \"Conversion done\!\\n\" unless \$total; chomp(\$total); my \$done = <STDIN>; chomp(\$done); printf(\"%0.2f percent\", 100 * \$done/\$total);'"

Hints: Migrating GPG Keys from one machine to another

GPG is a great utility used to encrypt and decrypt ... anything really. It's a public/private key infrastructure, and like most of these infrastructures, it's generally accepted that you share your public key with everyone, and you keep your private key very secret. Most operations involve both the private and public keys in some way. The operations enabled by your private key (decryption of messages sent to you, and signing of messages sent from you) are guaranteed to have actually come from you, as opposed to someone else attempting to impersonate you. Conversely, the operations enabled by the public key (encrypting a message and verifying the signature of a signed message) can be done by anyone. This works out in practice quite well: Anyone in the world (with the help of my public key) can send me an encrypted message that can be safely read by only myself (by using my private key). Additionally, whenever I send a message, I sign it with my private key, so that anyone receiving a message from me can verify it with my public key. This is all great and wonderful, and I fully support anyone who wishes to use it.

However, things get hairy when you, the owner of a brand spankin new private key, want to protect it.. You don't want to lose your private key in the madhouse that is hard drive failure, but you don't want to leave that private key hanging around somewhere that someone nefarious could pick it up and start impersonating you. You may, like I do, want to share a private key between multiple computers that you use on a regular basis. For this, we need a quick and secure method of transferring private and public keys between computers.

After you download and install GPG on your platforms of choice, and assuming you have SSH installed on those machines, you can run the following command on your source machine (the one that has the goods) to export your private or public keys between your machines without too much fear of interception in the middle.

gpg --export email@domain.com | ssh user@remote.host '/usr/local/bin/gpg --import' gpg --export-secret-keys email@domain.com | ssh user@remote.host '/usr/local/bin/gpg --import'

replacing as appropriate the email address, user and hostname of the remote host, and the path to gpg on the remote server (use which gpg on the remote machine to find that out). You should have your private and public keys exported over in no time!

EDIT - 5/1/11: It has come to my attention that you can export your entire private or public keychain in one go, saving you the trouble of sending each key individually. Just run...

gpg --export | ssh user@remote.host '/usr/local/bin/gpg --import' gpg --export-secret-keys | ssh user@remote.host '/usr/local/bin/gpg --import'

and watch all your keys come over. If you've already imported some, gpg will just skip over it.