What are media formats?
By media format, I mean a few different things which tie together in order to contain your movie and the rest of its support media.
The container is essentially the file format. Containers can hold multiple video, audio, picture or subtitle tracks in a single file. For example your container may contain multiple video tracks for a movie (one per chapter), English audio, Spanish audio, Dolby 5.1 audio in various languages as well as subtitles or closed captioning.
The most common container formats I’ve used with Plex are MPEG-4 Part 14 (better known as MP4) and Matroska Multimedia Container, also known as MKV. You may also see AVI, MPEG-2, Ogg or Flash. You can find quite a few container formats on Wikipedia.
Within the container file you’ll file one or more video streams encoded in one or more video codecs. Your container can contain a number of different video streams in different codecs, and video streams of a given codec may be contained in different container formats as needed.
Some common video codecs are the H.26X codecs from the Motion Pictures Experts Group (MPEG). These include H.262, H.263, H.264 and H.265 among others. The MPEG group puts out very good video codec standards, but tends to require licensing fees for devices which use them.
Google has published their own set of video codecs including VP6, VP7,VP8 and VP9. Or really, the codecs were created and published as proprietary codecs by a company called On2 which has since been purchased and opened by Google.
Theora is another open video codec which was initially based upon the VP3 codec from On2. It was created by the Xiph.org Foundation, the creators of the Ogg container format.
Your container files will also contain one or more audio tracks. These could be multiple sequential audio files, or they could be the audio for the same section of video available in multiple languages.
What is right for you?
At the end of the day what is right for you depends on a number of things including available disk space, available bandwidth, hardware playback support and server processing power.
Lossless compression formats will provide you with the best quality, but at the expense of size. If you’re willing to discard some of the information in the video stream in order to decrease the amount of actual data stored, then you will be able to fit more movies in the same amount of space.
Larger files also take more bandwidth to transfer. If you’re attempting to serve the files over a low bandwidth connection, then heavier compression may work better for you. Note that this typically will mean your local LAN bandwidth rather than your internet connection, unless you’re serving video to people outside of your home.
Hardware Playback Support
Video decompression can take a lot of CPU cycles to perform. This is not a big problem for modern desktop hardware, but it could be an issue for embedded devices like phones, tablets or set top boxes like the Roku/TiVo/etc.
One of the way hardware manufacturers help make their devices more usable for video streaming is to include specialized decoding hardware in order to offload the decompression from the CPU itself. This hardware will only have support for specific container or codecs, so you may find it useful to create your video files in a manner that your hardware specifically supports.
Server Processing Power
In order to improve support for various clients, Plex servers also support transcoding. Transcoding is the process of decoding the video streams on the fly, and then re-encoding them in a format that the client can more easily play. This provides you with better client compatibility but at the expense of CPU processing time on the server side.
Depending on your server hardware, you may find that transcoding is sufficient to support your needs. If your hardware is low end however, like some of the available Network Attached Storage (NAS) hardware with built in plex servers, then you may not be able to keep up with the clients. Also, if you share your plex server with friends, then multiple video streams transcoding at once may be too much for even beefier hardware.
What I usually do
My hardware is a bit beefier and modern, so I tend to stick with the default recommended settings for Handbrake. I direct stream to my clients when possible, and allow the server to transcode as needed. So far it has worked for me.