by auburndaletech

Installing Ubuntu 16.04 on a ZFS root filesystem

Re-publishing this from my old blog since it is bound to be useful for Plex servers.

One of the major new pieces of functionality in Ubuntu 16.04 (Xenial) is built in support for ZFS filesystems.  I was disappointed to learn that ZFS support is not actually built into the installer itself, leaving you to piece things together yourself.  I did find a few good tutorials online for this, but they all seem to be missing a few pieces.  I’m hoping that this guide will be a bit more complete for people.

It is based on a combination of the following two guides:

Booting Into the Install Environment

Setting up a ZFS filesystem requires a full set of userspace tools rather than the limited set included within the actual installer.  Due to this, we’re going to boot a Ubuntu Desktop live CD, and do a manual installation within its root filesystem.  I found that the easiest way to do this was to boot the live CD, set a password for the ubuntu user, and SSH into the machine from a remote box.  This way it is much easier to copy/paste commands from the webpage.
To start off, we need to install the ZFS tools and debootstrap which we will use for actually installing the operating system:
apt-add-repository universe
apt-get update
apt-get install –yes zfsutils-linux debootstrap

Partitioning the Disks

Next, we partition the disks.  For now we’re going to assume a two disk system which is just doing striping, but you can change as desired:

parted — /dev/sda mklabel msdos Y mkpart primary zfs 0% 100%
parted  /dev/sdb mklabel msdos Y mkpart primary zfs 0% 100%

Device naming during bootup on Linux isn’t always static, so we want to reference the disks through a static identifier like disk ID (/dev/disk/by-id), unfortunately Grub fails to properly identify the devices from the zfs command output, so we need this hack for now to make sure that update-grub will correctly identify the disks.  Specifically, it forces creation of symlinks in /dev which map to the device names in /dev/disk/by-id.  Some additional details about the issue can be found in this grub2 launchpad bug.

echo ‘KERNEL==”sd*[!0-9]”, IMPORT{parent}==”ID_*”, SYMLINK+=”$env{ID_BUS}-$env{ID_SERIAL}”
> KERNEL==”sd*[0-9]”, IMPORT{parent}==”ID_*”, SYMLINK+=”$env{ID_BUS}-$env{ID_SERIAL}-part%n”‘ > /etc/udev/rules.d/90-zfs.rules
udevadm trigger

Create Your Zpool

Now we actually create the zpool and import it to /mnt:

zpool create -m none -o ashift=12 -O compression=lz4 rpool /dev/sda1 /dev/sdb1
zfs create -o mountpoint=/ rpool/root
zpool export rpool
zpool import -d /dev/disk/by-id -R /mnt rpool

So here we create the zpool by device name, and then re-import it by device ID while mounting at /mnt.

And then create various partitions off of the root filesystem:

zfs create -o setuid=off rpool/root/home
zfs create -o mountpoint=/root rpool/root/home/root
zfs create -o canmount=off -o setuid=off -o exec=off rpool/root/var
zfs create -o com.sun:auto-snapshot=false rpool/root/var/cache
zfs create rpool/root/var/log
zfs create rpool/root/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/root/var/tmp

We break out these various subdirectories so we have the ability to optionally enable/disable compression, support for setuid/exec, and various other options.  We can also choose to limit the maximum size of each mount independently.

Install Ubuntu

The install of Ubuntu itself is pretty straightforward:

debootstrap the OS

debootstrap xenial /mnt
zfs set devices=off rpool
grep -v cdrom /etc/apt/sources.list > /mnt/etc/apt/sources.list
cp /etc/udev/rules.d/90-zfs.rules /mnt/etc/udev/rules.d/90-zfs.rules

You’ll notice at the end here we copy our hacky udev rule to the new filesystem.  The “devices=off” option we set here disables the ability to create device nodes in the filesystem.  This works since /dev is actually a devtmpfs partition of its own.

Configure the network interface

export INTERFACE=$(ip addr list | grep ^[0-9]: | grep -v “lo” | awk {‘print $2’} | cut -d “:” -f 1)
echo test > /mnt/etc/hostname
echo 127.0.1.1  >> /mnt/etc/hosts
echo “auto $INTERFACE
iface $INTERFACE inet dhcp” > /mnt/etc/network/interfaces.d/$INTERFACE

Enter the chroot for some final setup

mount –rbind /dev /mnt/dev
mount –rbind /proc /mnt/proc
mount –rbind /sys /mnt/sys
chroot /mnt /bin/bash –login
 
locale-gen en_US.UTF-8
echo ‘LANG=”en_US.UTF-8″‘ > /etc/default/locale
apt-get update
apt-get install –yes zfsutils-linux zfs-initramfs grub-pc linux-image-generic ssh
dpkg-reconfigure tzdata
update-initramfs -c -k all

Set Up Grub’s configuration

sed -i ‘s/^\(GRUB_CMDLINE_LINUX_DEFAULT\)=.*/\1=””/g’ /etc/default/grub
sed -i ‘s|^\(GRUB_HIDDEN_TIMEOUT=.*\)|#\1|g’ /etc/default/grub
sed -i ‘s/^\(GRUB_CMDLINE_LINUX\)=”\(.*\)”/\1=”boot=zfs \2″/g’ /etc/default/grub
ln -s /proc/mounts  /etc/mtab
update-grub
rm /etc/mtab

Set a root password and exit the chroot

passwd root
exit

Install the grub bootloader

grub-probe /mnt
grub-install –root-directory=/mnt /dev/sda
grub-install –root-directory=/mnt /dev/sdb

Reboot into the environment

reboot