Fix proxmox ZFS boot issues – GRUB: device not found

There is an issue floating around since a year that affects proxmox users with a ZFS root partition, namely after an upgrade it sometimes won’t boot. See here or here for example.

After it happening to me too (with two different servers) and familiarizing myself with the issue more closely, I came to the conclusion that the best course of action would be to create a separate ext /boot partition. Read on in case your server wont boot either and you want to save yourself the headache and time of finding the proper tools and figuring out the step by step proceedings.

Things you’ll need:

• proxmox server that refuses to boot
• temporary USB stick (to boot sysresccd from)
• permanent USB stick (or other block device – I was using an SD card) where the new boot partition will live

sysresccd

It is a pretty nifty tool, but what makes it even more nifty is that zfsonlinux.org provides one that has ZFS support built in. Download it from here and “burn” it to the temporary USB stick (I was using rufus). Plug it in your server and boot from it.

zfs mount

Proxmox always names it’s (first) zfs pool rpool so let us import that:

zpool import rpool

go to /mnt:

cd /mnt

make a new directory to mount your old system to, in this case proxmox:

mkdir proxmox

set the mount point to the path we just created above:

zfs set mountpoint=/mnt/proxmox /rpool/ROOT/pve-1

mount it:

zfs mount rpool/ROOT/pve-1

we also need to mount couple of other things:

mount -t proc /proc /mnt/proxmox/proc
mount --rbind /dev /mnt/proxmox/dev
mount --rbind /sys /mnt/proxmox/sys

chroot into proxmox

chroot /mnt/proxmox /bin/bash
source /etc/profile

make new partition

find out what block device you wanna sacrifice:

fdisk -l

double check if its the right one (in my case it was sdh):

fdisk -l /dev/sdh

Create a partition on it:

fdisk /dev/sdh

while in fdisk use these commands to create a new primary partition:

o (MBR partition table)
n (new partition)
p (primary partition)
1 (partition number: 1)
just press enter (leave default for first cylinder)
just press enter (leave default for last cylinder)
w (write changes to disk)

Notes:

• GPT would also work, but im using a 8GB sd card here… mbr is old, but not dead
• by default fdisk will always make a partition of type “Linux”, no need to play around with that

Format the newly created partition to ext4:

mkfs.ext4 /dev/sdh1

we need to modify our /etc/fstab and add our not-yet-created /boot partition, to do that we gonna add it by it’s UUID run the following:

blkid

and find your partitions (in my case sdh1) UUID. Once thats written down on paper open fstab for editing:

nano /etc/fstab

Add a new TEMPORARY entry according to the UUID you got earlier, something similar to:

UUID=afdc2cf9-c06e-452a-8e42-2b25ad9dac55 /media ext4 defaults 0 0

Test it out my trying to mount it:

mount /media

move /boot

Copy your old boot to the newly mounted media and unmount it:

cp -Rpvf /boot/* /media/
sync
umount /media

edit your previously made temporary fstab entry so that it will be mount as /boot:

UUID=afdc2cf9-c06e-452a-8e42-2b25ad9dac55  /boot  ext4  defaults  0  0

mount it for real now:

mount /boot

run the following (note it is sdh not shd1):

update-grub
grub-install --recheck /dev/sdh
sync
umount /boot

Now you can remove the old /boot:

rm -rf /boot/*

and finally exit the chroot enviroment:

exit

zfs umount

umount proc:

umount /mnt/proxmox/proc

umount sys:

umount /mnt/proxmox/sys

If it gives you lip, you probably need to unmount something else first, find out what by running:

mount | grep /mnt/proxmox/sys

umount dev (same procedure as with sys):

umount /mnt/proxmox/dev

umount zfs:

umount /mnt/proxmox

set the mount-point back to where it should be:

zfs set mountpoint=/ rpool/ROOT/pve-1

Reboot, change the boot device in the bios, and if all goes well it should work!