NVidia Shield: Another Option for Plex

For father’s day this year, I was given an NVidia Shield.  While I’ve only had it for a few days so far, it is already my favorite media platform.

Under the covers it’s just Android TV, which I’ve had before using the Nexus Player, but unlike the Nexus the Shield has the Amazon video app.  In fact, it supports all of the major media services that we try to use.  And on top of that, there are games!

The Plex client works great.  It can additionally act as a Plex server, either with local storage or by attaching additional USB storage.  Additionally, the Plex client for the Shield has built in support for the relatively new Plex DVR functionality which allows you to record live television by using a network enabled TV tuner.

At $199 with 4k video output support, it’s a fairly good price.  You can also potentially get the older model for cheaper as well.

Enabling Closed Captioning

After sharing out my movie to a friend who is hearing impaired, I decided to look into how I would go about making sure that my movies would be accessible to her.

Closed captioning for movies is supported by embedding the subtitles in the movie files themselves, or through the use of external files.

My first thought was to attempt to embed the files as provided by the DVDs or Blu-Ray discs.  As it turns out, this approach is a bit fraught with peril as I’ve found that getting Handbrake to do the right thing was a bit hit or miss, but you can find some good instructions online.  I’ve occasionally ended up with permanently turned on closed captioning for specific movies, or just ending up with foreign subtitles rather than English closed captioning.  Not to mention that this approach would require re-ripping all existing media files which were not ripped with them enabled.

The easier approach is through the use of an external file.  These closed captioning files can be found online on a number of websites like OpenSubtitles or Subscene.  Even better, the operation can be automated by various Plex plugins.  I use Sub-Zero which is itself a fork of Subliminal.  Installation of the plugin itself is probably best left for another post.

One item to note regarding using these plugins for downloading subtitles is that it will require the Plex server to have write access to your content directories.  Typically Plex only requires read-only access, so you may not already have this set up.  This can be done by simply adding the plex user to the group which owns the content directory and ensuring that group write permissions are set.

Installing Ubuntu 16.04 on a ZFS root filesystem

Re-publishing this from my old blog since it is bound to be useful for Plex servers.

One of the major new pieces of functionality in Ubuntu 16.04 (Xenial) is built in support for ZFS filesystems.  I was disappointed to learn that ZFS support is not actually built into the installer itself, leaving you to piece things together yourself.  I did find a few good tutorials online for this, but they all seem to be missing a few pieces.  I’m hoping that this guide will be a bit more complete for people.

It is based on a combination of the following two guides:

Booting Into the Install Environment

Setting up a ZFS filesystem requires a full set of userspace tools rather than the limited set included within the actual installer.  Due to this, we’re going to boot a Ubuntu Desktop live CD, and do a manual installation within its root filesystem.  I found that the easiest way to do this was to boot the live CD, set a password for the ubuntu user, and SSH into the machine from a remote box.  This way it is much easier to copy/paste commands from the webpage.
To start off, we need to install the ZFS tools and debootstrap which we will use for actually installing the operating system:
apt-add-repository universe
apt-get update
apt-get install –yes zfsutils-linux debootstrap

Partitioning the Disks

Next, we partition the disks.  For now we’re going to assume a two disk system which is just doing striping, but you can change as desired:

parted — /dev/sda mklabel msdos Y mkpart primary zfs 0% 100%
parted  /dev/sdb mklabel msdos Y mkpart primary zfs 0% 100%

Device naming during bootup on Linux isn’t always static, so we want to reference the disks through a static identifier like disk ID (/dev/disk/by-id), unfortunately Grub fails to properly identify the devices from the zfs command output, so we need this hack for now to make sure that update-grub will correctly identify the disks.  Specifically, it forces creation of symlinks in /dev which map to the device names in /dev/disk/by-id.  Some additional details about the issue can be found in this grub2 launchpad bug.

echo ‘KERNEL==”sd*[!0-9]”, IMPORT{parent}==”ID_*”, SYMLINK+=”$env{ID_BUS}-$env{ID_SERIAL}”
> KERNEL==”sd*[0-9]”, IMPORT{parent}==”ID_*”, SYMLINK+=”$env{ID_BUS}-$env{ID_SERIAL}-part%n”‘ > /etc/udev/rules.d/90-zfs.rules
udevadm trigger

Create Your Zpool

Now we actually create the zpool and import it to /mnt:

zpool create -m none -o ashift=12 -O compression=lz4 rpool /dev/sda1 /dev/sdb1
zfs create -o mountpoint=/ rpool/root
zpool export rpool
zpool import -d /dev/disk/by-id -R /mnt rpool

So here we create the zpool by device name, and then re-import it by device ID while mounting at /mnt.

And then create various partitions off of the root filesystem:

zfs create -o setuid=off rpool/root/home
zfs create -o mountpoint=/root rpool/root/home/root
zfs create -o canmount=off -o setuid=off -o exec=off rpool/root/var
zfs create -o com.sun:auto-snapshot=false rpool/root/var/cache
zfs create rpool/root/var/log
zfs create rpool/root/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/root/var/tmp

We break out these various subdirectories so we have the ability to optionally enable/disable compression, support for setuid/exec, and various other options.  We can also choose to limit the maximum size of each mount independently.

Install Ubuntu

The install of Ubuntu itself is pretty straightforward:

debootstrap the OS

debootstrap xenial /mnt
zfs set devices=off rpool
grep -v cdrom /etc/apt/sources.list > /mnt/etc/apt/sources.list
cp /etc/udev/rules.d/90-zfs.rules /mnt/etc/udev/rules.d/90-zfs.rules

You’ll notice at the end here we copy our hacky udev rule to the new filesystem.  The “devices=off” option we set here disables the ability to create device nodes in the filesystem.  This works since /dev is actually a devtmpfs partition of its own.

Configure the network interface

export INTERFACE=$(ip addr list | grep ^[0-9]: | grep -v “lo” | awk {‘print $2’} | cut -d “:” -f 1)
echo test > /mnt/etc/hostname
echo 127.0.1.1  >> /mnt/etc/hosts
echo “auto $INTERFACE
iface $INTERFACE inet dhcp” > /mnt/etc/network/interfaces.d/$INTERFACE

Enter the chroot for some final setup

mount –rbind /dev /mnt/dev
mount –rbind /proc /mnt/proc
mount –rbind /sys /mnt/sys
chroot /mnt /bin/bash –login
 
locale-gen en_US.UTF-8
echo ‘LANG=”en_US.UTF-8″‘ > /etc/default/locale
apt-get update
apt-get install –yes zfsutils-linux zfs-initramfs grub-pc linux-image-generic ssh
dpkg-reconfigure tzdata
update-initramfs -c -k all

Set Up Grub’s configuration

sed -i ‘s/^\(GRUB_CMDLINE_LINUX_DEFAULT\)=.*/\1=””/g’ /etc/default/grub
sed -i ‘s|^\(GRUB_HIDDEN_TIMEOUT=.*\)|#\1|g’ /etc/default/grub
sed -i ‘s/^\(GRUB_CMDLINE_LINUX\)=”\(.*\)”/\1=”boot=zfs \2″/g’ /etc/default/grub
ln -s /proc/mounts  /etc/mtab
update-grub
rm /etc/mtab

Set a root password and exit the chroot

passwd root
exit

Install the grub bootloader

grub-probe /mnt
grub-install –root-directory=/mnt /dev/sda
grub-install –root-directory=/mnt /dev/sdb

Reboot into the environment

reboot

Plex space usage

One item to consider when setting up a Plex server for the first time is space allocation.  Plex servers can use up a lot of disk space and not necessarily where you expect, so it’s better to be educated than blind sided.

The vast majority of space you can expect to use on your server is going to be for movie storage.  The location where you store your movies is completely up to you and is configurable when you first set up your libraries.  In my case, I created a path called /home/media on the server which contains directories for movies, tv shows, music, etc.  Each of those directories results in its own Library.

The less expected space is under your /var partition, which can be a problem if you allocated a small amount of space for / or /var and put the majority of your disk space under /home.  (As an aside, consider using LVM or ZFS for your disks in order to be able to reallocate space on the fly.  I’ll have to write up more about this in the future).

The space in /var will be found in /var/lib/plexmediaserver/Library/Plex Media Server on Ubuntu.  I suspect other Linux distributions will be the same, although there could be some variance if you’re using something like FreeNAS or installing Plex via a container.

This space contains a lot of really useful things, for example your installed plugins, logs, the metadata for your server, and transcoded media files.  The last one is where the bulk of your space will go to.  On my server this space in /var takes about 20GB (compared to 2TB in /home/media).  Plex does however support preemptively optimizing video files for playback.  This feature is useful on slower hardware that may not be able to transcode on the fly, but the resulting media files will also be stored in /var.

Ripping Blu-Ray

Blu-Ray discs require a different approach for ripping than DVDs do for a few reasons.  Mainly in that you need to have Blu-Ray reader in your computer and the size of the content is going to be a lot larger.  While you can expect a DVD rip to weigh in at 5GB or so prior to recompressing, Bluray discs are typically closer to 30GB.  Additionally the encryption used to protect the content is different and there isn’t a way to decrypt it in Handbrake itself.

My software of choice for ripping Blu-Ray is MakeMKV.  MakeMKV is commercial software with some open source components.  The Windows and Mac downloads are available with a 30 day trial, but it costs $50 for a perpetual license after that.  The Linux version is still currently considered beta, and during the beta you’re allowed to continue with the trial as long as you keep installing the updated versions.  Your $50 license if you purchase one can be used on any of the platforms, including the Linux version.

MakeMKV simply copies the movie to disk and works for DVD or Blu-Ray.  Due to this, you’ll need to have enough space to hold the whole movie (again, 30+GB).  Once it is decrypted and written to disk, you can then use Handbrake or any other transcoding software to recompress it to a smaller size.  The final size will depend on the codec you choose and the resolution to encode it for, but my Blu-Ray rips often end up a few Gigabytes.  You can also just choose to put the raw rip on your server if you want to put a lot of disk space aside for the media.

 

Ripping DVDs

My tool of choice for my DVD ripping needs is Handbrake.  It is an Open Source tool which supports Windows, Mac OS X or Linux.  It is highly configurable, but also has a great set of defaults for various codecs and container formats.

On the ripping side, it natively supports ripping unencrypted DVDs and supports ripping encrypted DVDs if your system has a DVD decryption library built in.  On Mac OS X, that support is built into the operating system itself.  On Windows and Linux, you need to install a copy of libdvdcss to handle the DVD decryption for you.

For instructions on installing libdvdcss on Windows for Handbrake to you, you can find instructions with links to the library.  On Linux the instructions will vary a bit based upon the distribution.  I typically use Ubuntu but you can also find instructions for Fedora and other other common distros.

Once properly installed, ripping is fairly easy.  Simply launch the program, select “Open Source”, and select your DVD drive:

handbrake-1

handbrake-2

Handbrake 3.PNG

After a brief delay as the disc is scanned, one of the tracks on the disc will be selected by default, and a filename is generated.  Assuming the filename, Title/Track and the preset you want are all selected, simply click “Start Encode” to begin ripping.  You’ll be notified once the ripping is complete.  The time required for this to finish will depend on the size of the movie, speed of your DVD-ROM drive and the processing power required/available for the encoding part.

Some DVDs may contain multiple Titles/Tracks.  This is especially common when ripping DVDs of TV shows where you’ll see multiple episodes per disc, or for ripping multi-packs of movies which may have multiple movies on the disc.  To rip multiple tracks, just select each track individually, name the destination files, and click “Add to Queue”.  You can queue up as many rips as you need to.

Handbrake can also be used to re-encode existing video files on your drive if desired.  This is especially helpful when doing Blu-ray rips.  While Handbrake cannot rip Blu-ray discs, it can re-encode Blu-ray data files once they are ripped from the disc by another program.

Protecting Your Content

Assuming you’re ripping and decoding your own media, you’ll find that the process is slow.  How slow depends on your specific hardware.  You can expect a DVD rip and encode to take at least 10 minutes or so though.  Even longer for a Bluray rip.

This isn’t too much of a hassle when buying and then ripping discs which you have purchased, but it can be extremely painful when you’re first ripping all of your existing discs.  Due to this, you’ll want to take the right steps in order to ensure that you do not need to rip the discs more than once.

Backup

The most important thing to do in order to protect your content is to come up with a backup solution of some sort.  There’s no hard and fast rule for how you should back up your content.  Instead you’ll need find the balance between cost, hassle and effectiveness that makes sense for your needs.

The simplest backup solution would be to have an additional external hard drive of an appropriate size for you to copy your data to on some regular basis.  This approach tends to be fairly fast and not very expensive and will help protect against loss caused by mistakes made on the server or a failure of your disk.  This approach however will not help if something was to happen to your home.  A fire could easily take out both your primary server and your backup disk.  Perhaps this is an acceptable risk for you, or at least the least of your worries in the event of a house fire.

Another inexpensive method requires that you have a friend with a server set up as well.  You can choose to mirror each other’s content over the internet in order to provide both of you with a backup plan.

Another option is using an online backup solution.  Personally I use Crashplan.  Crashplan has a free option which provides for encrypted backup to other people’s computers for no cost.  Alternatively they offer backup to their own servers for a pretty reasonable monthly cost.

Finally, you can roll your own solution using one of the various cloud providers out there.  Box.net, Dropbox, Google Drive, Rsync.net and others all offer large amounts of file storage for purchase.  The costs, compatibility and performance will vary from service to service.

Storage Redundancy

While offline backup is critical for ensuring your data, recovering can be time consuming and painful, so any steps you can take in order to avoid needing to do it is useful.  The simplest step is to protect yourself against disk failure by using some form of redundancy.

The most common form of this sort of redundancy is a Redundant Array of Inexpensive Disks, or RAID.  RAID comes in different levels which provide for different amounts of protection.  They range from RAID1 which completely mirrors your content between two disks, and RAID5/RAID6 which allow you to combine a number of disks together to increase your storage capacity while putting aside one or two disks for parity.  These parity disks allow you to survive losing a disk or two in your array without losing your data.

The other solution is ZFS, which is a newer system developed by Sun Microsystems (now part of Oracle), and open sourced as part of OpenSolaris.  It was then made available for FreeBSD and now a port is available for Linux as well (although it does have some licensing issues).

ZFS works a lot like RAID, but provides a lot of advanced features too including snapshotting, transparent compression, and remote replication.  It’s a good choice for NAS devices or Linux systems if you’re an advanced Linux admin.