Installing Ubuntu 16.04 on a ZFS root filesystem

Re-publishing this from my old blog since it is bound to be useful for Plex servers.

One of the major new pieces of functionality in Ubuntu 16.04 (Xenial) is built in support for ZFS filesystems.  I was disappointed to learn that ZFS support is not actually built into the installer itself, leaving you to piece things together yourself.  I did find a few good tutorials online for this, but they all seem to be missing a few pieces.  I’m hoping that this guide will be a bit more complete for people.

It is based on a combination of the following two guides:

Booting Into the Install Environment

Setting up a ZFS filesystem requires a full set of userspace tools rather than the limited set included within the actual installer.  Due to this, we’re going to boot a Ubuntu Desktop live CD, and do a manual installation within its root filesystem.  I found that the easiest way to do this was to boot the live CD, set a password for the ubuntu user, and SSH into the machine from a remote box.  This way it is much easier to copy/paste commands from the webpage.
To start off, we need to install the ZFS tools and debootstrap which we will use for actually installing the operating system:
apt-add-repository universe
apt-get update
apt-get install –yes zfsutils-linux debootstrap

Partitioning the Disks

Next, we partition the disks.  For now we’re going to assume a two disk system which is just doing striping, but you can change as desired:

parted — /dev/sda mklabel msdos Y mkpart primary zfs 0% 100%
parted  /dev/sdb mklabel msdos Y mkpart primary zfs 0% 100%

Device naming during bootup on Linux isn’t always static, so we want to reference the disks through a static identifier like disk ID (/dev/disk/by-id), unfortunately Grub fails to properly identify the devices from the zfs command output, so we need this hack for now to make sure that update-grub will correctly identify the disks.  Specifically, it forces creation of symlinks in /dev which map to the device names in /dev/disk/by-id.  Some additional details about the issue can be found in this grub2 launchpad bug.

echo ‘KERNEL==”sd*[!0-9]”, IMPORT{parent}==”ID_*”, SYMLINK+=”$env{ID_BUS}-$env{ID_SERIAL}”
> KERNEL==”sd*[0-9]”, IMPORT{parent}==”ID_*”, SYMLINK+=”$env{ID_BUS}-$env{ID_SERIAL}-part%n”‘ > /etc/udev/rules.d/90-zfs.rules
udevadm trigger

Create Your Zpool

Now we actually create the zpool and import it to /mnt:

zpool create -m none -o ashift=12 -O compression=lz4 rpool /dev/sda1 /dev/sdb1
zfs create -o mountpoint=/ rpool/root
zpool export rpool
zpool import -d /dev/disk/by-id -R /mnt rpool

So here we create the zpool by device name, and then re-import it by device ID while mounting at /mnt.

And then create various partitions off of the root filesystem:

zfs create -o setuid=off rpool/root/home
zfs create -o mountpoint=/root rpool/root/home/root
zfs create -o canmount=off -o setuid=off -o exec=off rpool/root/var
zfs create -o com.sun:auto-snapshot=false rpool/root/var/cache
zfs create rpool/root/var/log
zfs create rpool/root/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/root/var/tmp

We break out these various subdirectories so we have the ability to optionally enable/disable compression, support for setuid/exec, and various other options.  We can also choose to limit the maximum size of each mount independently.

Install Ubuntu

The install of Ubuntu itself is pretty straightforward:

debootstrap the OS

debootstrap xenial /mnt
zfs set devices=off rpool
grep -v cdrom /etc/apt/sources.list > /mnt/etc/apt/sources.list
cp /etc/udev/rules.d/90-zfs.rules /mnt/etc/udev/rules.d/90-zfs.rules

You’ll notice at the end here we copy our hacky udev rule to the new filesystem.  The “devices=off” option we set here disables the ability to create device nodes in the filesystem.  This works since /dev is actually a devtmpfs partition of its own.

Configure the network interface

export INTERFACE=$(ip addr list | grep ^[0-9]: | grep -v “lo” | awk {‘print $2’} | cut -d “:” -f 1)
echo test > /mnt/etc/hostname
echo 127.0.1.1  >> /mnt/etc/hosts
echo “auto $INTERFACE
iface $INTERFACE inet dhcp” > /mnt/etc/network/interfaces.d/$INTERFACE

Enter the chroot for some final setup

mount –rbind /dev /mnt/dev
mount –rbind /proc /mnt/proc
mount –rbind /sys /mnt/sys
chroot /mnt /bin/bash –login
 
locale-gen en_US.UTF-8
echo ‘LANG=”en_US.UTF-8″‘ > /etc/default/locale
apt-get update
apt-get install –yes zfsutils-linux zfs-initramfs grub-pc linux-image-generic ssh
dpkg-reconfigure tzdata
update-initramfs -c -k all

Set Up Grub’s configuration

sed -i ‘s/^\(GRUB_CMDLINE_LINUX_DEFAULT\)=.*/\1=””/g’ /etc/default/grub
sed -i ‘s|^\(GRUB_HIDDEN_TIMEOUT=.*\)|#\1|g’ /etc/default/grub
sed -i ‘s/^\(GRUB_CMDLINE_LINUX\)=”\(.*\)”/\1=”boot=zfs \2″/g’ /etc/default/grub
ln -s /proc/mounts  /etc/mtab
update-grub
rm /etc/mtab

Set a root password and exit the chroot

passwd root
exit

Install the grub bootloader

grub-probe /mnt
grub-install –root-directory=/mnt /dev/sda
grub-install –root-directory=/mnt /dev/sdb

Reboot into the environment

reboot

Plex space usage

One item to consider when setting up a Plex server for the first time is space allocation.  Plex servers can use up a lot of disk space and not necessarily where you expect, so it’s better to be educated than blind sided.

The vast majority of space you can expect to use on your server is going to be for movie storage.  The location where you store your movies is completely up to you and is configurable when you first set up your libraries.  In my case, I created a path called /home/media on the server which contains directories for movies, tv shows, music, etc.  Each of those directories results in its own Library.

The less expected space is under your /var partition, which can be a problem if you allocated a small amount of space for / or /var and put the majority of your disk space under /home.  (As an aside, consider using LVM or ZFS for your disks in order to be able to reallocate space on the fly.  I’ll have to write up more about this in the future).

The space in /var will be found in /var/lib/plexmediaserver/Library/Plex Media Server on Ubuntu.  I suspect other Linux distributions will be the same, although there could be some variance if you’re using something like FreeNAS or installing Plex via a container.

This space contains a lot of really useful things, for example your installed plugins, logs, the metadata for your server, and transcoded media files.  The last one is where the bulk of your space will go to.  On my server this space in /var takes about 20GB (compared to 2TB in /home/media).  Plex does however support preemptively optimizing video files for playback.  This feature is useful on slower hardware that may not be able to transcode on the fly, but the resulting media files will also be stored in /var.