Installing Ubuntu 16.04 on a ZFS root filesystem

Re-publishing this from my old blog since it is bound to be useful for Plex servers.

One of the major new pieces of functionality in Ubuntu 16.04 (Xenial) is built in support for ZFS filesystems.  I was disappointed to learn that ZFS support is not actually built into the installer itself, leaving you to piece things together yourself.  I did find a few good tutorials online for this, but they all seem to be missing a few pieces.  I’m hoping that this guide will be a bit more complete for people.

It is based on a combination of the following two guides:

Booting Into the Install Environment

Setting up a ZFS filesystem requires a full set of userspace tools rather than the limited set included within the actual installer.  Due to this, we’re going to boot a Ubuntu Desktop live CD, and do a manual installation within its root filesystem.  I found that the easiest way to do this was to boot the live CD, set a password for the ubuntu user, and SSH into the machine from a remote box.  This way it is much easier to copy/paste commands from the webpage.
To start off, we need to install the ZFS tools and debootstrap which we will use for actually installing the operating system:
apt-add-repository universe
apt-get update
apt-get install –yes zfsutils-linux debootstrap

Partitioning the Disks

Next, we partition the disks.  For now we’re going to assume a two disk system which is just doing striping, but you can change as desired:

parted — /dev/sda mklabel msdos Y mkpart primary zfs 0% 100%
parted  /dev/sdb mklabel msdos Y mkpart primary zfs 0% 100%

Device naming during bootup on Linux isn’t always static, so we want to reference the disks through a static identifier like disk ID (/dev/disk/by-id), unfortunately Grub fails to properly identify the devices from the zfs command output, so we need this hack for now to make sure that update-grub will correctly identify the disks.  Specifically, it forces creation of symlinks in /dev which map to the device names in /dev/disk/by-id.  Some additional details about the issue can be found in this grub2 launchpad bug.

echo ‘KERNEL==”sd*[!0-9]”, IMPORT{parent}==”ID_*”, SYMLINK+=”$env{ID_BUS}-$env{ID_SERIAL}”
> KERNEL==”sd*[0-9]”, IMPORT{parent}==”ID_*”, SYMLINK+=”$env{ID_BUS}-$env{ID_SERIAL}-part%n”‘ > /etc/udev/rules.d/90-zfs.rules
udevadm trigger

Create Your Zpool

Now we actually create the zpool and import it to /mnt:

zpool create -m none -o ashift=12 -O compression=lz4 rpool /dev/sda1 /dev/sdb1
zfs create -o mountpoint=/ rpool/root
zpool export rpool
zpool import -d /dev/disk/by-id -R /mnt rpool

So here we create the zpool by device name, and then re-import it by device ID while mounting at /mnt.

And then create various partitions off of the root filesystem:

zfs create -o setuid=off rpool/root/home
zfs create -o mountpoint=/root rpool/root/home/root
zfs create -o canmount=off -o setuid=off -o exec=off rpool/root/var
zfs create -o com.sun:auto-snapshot=false rpool/root/var/cache
zfs create rpool/root/var/log
zfs create rpool/root/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/root/var/tmp

We break out these various subdirectories so we have the ability to optionally enable/disable compression, support for setuid/exec, and various other options.  We can also choose to limit the maximum size of each mount independently.

Install Ubuntu

The install of Ubuntu itself is pretty straightforward:

debootstrap the OS

debootstrap xenial /mnt
zfs set devices=off rpool
grep -v cdrom /etc/apt/sources.list > /mnt/etc/apt/sources.list
cp /etc/udev/rules.d/90-zfs.rules /mnt/etc/udev/rules.d/90-zfs.rules

You’ll notice at the end here we copy our hacky udev rule to the new filesystem.  The “devices=off” option we set here disables the ability to create device nodes in the filesystem.  This works since /dev is actually a devtmpfs partition of its own.

Configure the network interface

export INTERFACE=$(ip addr list | grep ^[0-9]: | grep -v “lo” | awk {‘print $2’} | cut -d “:” -f 1)
echo test > /mnt/etc/hostname
echo 127.0.1.1  >> /mnt/etc/hosts
echo “auto $INTERFACE
iface $INTERFACE inet dhcp” > /mnt/etc/network/interfaces.d/$INTERFACE

Enter the chroot for some final setup

mount –rbind /dev /mnt/dev
mount –rbind /proc /mnt/proc
mount –rbind /sys /mnt/sys
chroot /mnt /bin/bash –login
 
locale-gen en_US.UTF-8
echo ‘LANG=”en_US.UTF-8″‘ > /etc/default/locale
apt-get update
apt-get install –yes zfsutils-linux zfs-initramfs grub-pc linux-image-generic ssh
dpkg-reconfigure tzdata
update-initramfs -c -k all

Set Up Grub’s configuration

sed -i ‘s/^\(GRUB_CMDLINE_LINUX_DEFAULT\)=.*/\1=””/g’ /etc/default/grub
sed -i ‘s|^\(GRUB_HIDDEN_TIMEOUT=.*\)|#\1|g’ /etc/default/grub
sed -i ‘s/^\(GRUB_CMDLINE_LINUX\)=”\(.*\)”/\1=”boot=zfs \2″/g’ /etc/default/grub
ln -s /proc/mounts  /etc/mtab
update-grub
rm /etc/mtab

Set a root password and exit the chroot

passwd root
exit

Install the grub bootloader

grub-probe /mnt
grub-install –root-directory=/mnt /dev/sda
grub-install –root-directory=/mnt /dev/sdb

Reboot into the environment

reboot

Protecting Your Content

Assuming you’re ripping and decoding your own media, you’ll find that the process is slow.  How slow depends on your specific hardware.  You can expect a DVD rip and encode to take at least 10 minutes or so though.  Even longer for a Bluray rip.

This isn’t too much of a hassle when buying and then ripping discs which you have purchased, but it can be extremely painful when you’re first ripping all of your existing discs.  Due to this, you’ll want to take the right steps in order to ensure that you do not need to rip the discs more than once.

Backup

The most important thing to do in order to protect your content is to come up with a backup solution of some sort.  There’s no hard and fast rule for how you should back up your content.  Instead you’ll need find the balance between cost, hassle and effectiveness that makes sense for your needs.

The simplest backup solution would be to have an additional external hard drive of an appropriate size for you to copy your data to on some regular basis.  This approach tends to be fairly fast and not very expensive and will help protect against loss caused by mistakes made on the server or a failure of your disk.  This approach however will not help if something was to happen to your home.  A fire could easily take out both your primary server and your backup disk.  Perhaps this is an acceptable risk for you, or at least the least of your worries in the event of a house fire.

Another inexpensive method requires that you have a friend with a server set up as well.  You can choose to mirror each other’s content over the internet in order to provide both of you with a backup plan.

Another option is using an online backup solution.  Personally I use Crashplan.  Crashplan has a free option which provides for encrypted backup to other people’s computers for no cost.  Alternatively they offer backup to their own servers for a pretty reasonable monthly cost.

Finally, you can roll your own solution using one of the various cloud providers out there.  Box.net, Dropbox, Google Drive, Rsync.net and others all offer large amounts of file storage for purchase.  The costs, compatibility and performance will vary from service to service.

Storage Redundancy

While offline backup is critical for ensuring your data, recovering can be time consuming and painful, so any steps you can take in order to avoid needing to do it is useful.  The simplest step is to protect yourself against disk failure by using some form of redundancy.

The most common form of this sort of redundancy is a Redundant Array of Inexpensive Disks, or RAID.  RAID comes in different levels which provide for different amounts of protection.  They range from RAID1 which completely mirrors your content between two disks, and RAID5/RAID6 which allow you to combine a number of disks together to increase your storage capacity while putting aside one or two disks for parity.  These parity disks allow you to survive losing a disk or two in your array without losing your data.

The other solution is ZFS, which is a newer system developed by Sun Microsystems (now part of Oracle), and open sourced as part of OpenSolaris.  It was then made available for FreeBSD and now a port is available for Linux as well (although it does have some licensing issues).

ZFS works a lot like RAID, but provides a lot of advanced features too including snapshotting, transparent compression, and remote replication.  It’s a good choice for NAS devices or Linux systems if you’re an advanced Linux admin.