Hacker's ramblings

This is part 2 of my hard drive upgrade. My previous part was about failure to partition a replaced hard drive with GNU Parted: It was just emitting an error of: "The resulting partition is not properly aligned for best performance"

When I had the drive partitioned properly, I failed to proceed with my setup in a yet another mysterious error. My drives are always using LVM, so that I get more control over the filesystem sizes. To get the new partition into LVM, it needs to be associated with a Volume Group (VG). First step is to inform LVM about new physical drive:

# pvcreate /dev/sda1
  Can't open /dev/sda1 exclusively.  Mounted filesystem?

Oh really? It's definitely not mounted, but ... somebody is stealing my resource. The root of this problem is obviously on the fact, that there used to be a PV on that partition, but I replaced the drive and partitioned it. It is entirely possible, that LVM likes to fiddle with my new partition somehow.

The device mapper knows about the partition:

# dmsetup ls
Box_vg1-LogVol_wrk2   (253:9)

That's kind of bad. I guess it likes to hold on into it. Further check of:

# pvdisplay

... indicates, that LVM doesn't know about the partition (yet), but Linux kernel does.

An attempt to fix:

# dmsetup remove Box_vg1-LogVol_wrk2

And new attempt:

# pvcreate /dev/sda1
  Can't open /dev/sda1 exclusively.  Mounted filesystem?

No change. Perhaps a strace will provide helpful details of the problem:

# strace pvcreate /dev/sda1
...
stat("/dev/sda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 1), ...}) = 0
stat("/dev/sda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 1), ...}) = 0
open("/dev/sda1", O_RDWR|O_EXCL|O_DIRECT|O_NOATIME) = -1 EBUSY (Device or resource busy)
...

Reading a fragment of OPEN(2) man page:

OPEN(2)
open, openat, creat - open and possibly create a file

       O_EXCL Ensure that this call creates the file: if this flag is
              specified in conjunction with O_CREAT, and pathname already
              exists, then open() will fail.

              In general, the behavior of O_EXCL is undefined if it is used
              without O_CREAT.  There is one exception: on Linux 2.6 and
              later, O_EXCL can be used without O_CREAT if pathname refers
              to a block device.  If the block device is in use by the
              system (e.g., mounted), open() fails with the error EBUSY.

... confirms the suspicion, that somebody is holding a handle to the block device. Running lsof(8) or fuser(1) yield nothing. It's not a file-handle, when kernel has your block device as hostage.

My only idea at this point was to do a wimpy Windows-style reboot. The thing is: Linux-men don't reboot on anything, but this time I was out of ideas. I'm sure somewhere there is an IOCTL-call to release the handle, but I couldn't find it easily. So, a reboot was in order.

After the reboot: yes results:

# pvcreate /dev/sda1
  Physical volume "/dev/sda1" successfully created

Then I could proceed with my build sequence. Next, associate a Volume Group with the new Pysical Volume. The options would be to to add the drive into an existing VG, or create a new one. I chose the latter:

# vgcreate Box_vg1 /dev/sda1
  Volume group "Box_vg1" successfully created

Then create a logical partition, or Logical Volume in LVM-lingo on the newly created VG:

# lvcreate -L 800G -n LogVol_wrk2 Box_vg1
  Logical volume "LogVol_wrk2" created

As a physical partition also a LV needs to have a filesystem on it, to be usable for the operating system:

# mkfs.ext4 /dev/Box_vg1/LogVol_wrk2
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 209715200 4k blocks and 52428800 inodes
Filesystem UUID: 93be6c97-3ade-4a62-9403-789f64ef73d0
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Now the drive was ready to be mounted and I had plenty of completely empty space waiting for data to be stored on it.

I plugged in a SATA-USB -dock and started looking for my old data. I intentionally had created a VG with precisely the same name as the old drive had, so there was an obvious collision. My syslog had entries about the pvscan:

Nov  8 16:03:21 pvscan: device-mapper: create ioctl on Box_vg1-LogVol_wrk2 failed: Device or resource busy
Nov  8 16:03:21 pvscan: 0 logical volume(s) in volume group "Box_vg1" now active
Nov  8 16:03:21 pvscan: Box_vg1: autoactivation failed.

Yes, that one I had coming. No autoactivation, as VG names collided. A check:

# vgdisplay
  --- Volume group ---
  VG Name               Box_vg0
...
  --- Volume group ---
  VG Name               Box_vg1
...
  --- Volume group ---
  VG Name               Box_vg1
...
  VG UUID               trx8sq-2Mtf-2tfa-2m1P-YPGq-cVzA-6fWflU

No surprises there, there were two Volume Groups with exactly same name. To address them, there are unique identifiers or UUIDs. With UUID, it is possible to rename the VG. Like this:

# vgrename trx8sq-2Mtf-2tfa-2m1P-YPGq-cVzA-6fWflU Box_vgold
  Volume group "Box_vg1" successfully renamed to "Box_vgold"

Now it would be possible to activate and it would appear on udev:

# vgchange -ay Box_vgold
  1 logical volume(s) in volume group "Box_vgold" now active

Now the old data was available at /dev/Box_vgold/LogVol_wrk2 and ready to be mounted and files copied out of it.

Done and mission accomplished! Now I had much more space on a fast drive.