Blu-Ray discs require a different approach for ripping than DVDs do for a few reasons. Mainly in that you need to have Blu-Ray reader in your computer and the size of the content is going to be a lot larger. While you can expect a DVD rip to weigh in at 5GB or so prior to recompressing, Bluray discs are typically closer to 30GB. Additionally the encryption used to protect the content is different and there isn’t a way to decrypt it in Handbrake itself.
My software of choice for ripping Blu-Ray is MakeMKV. MakeMKV is commercial software with some open source components. The Windows and Mac downloads are available with a 30 day trial, but it costs $50 for a perpetual license after that. The Linux version is still currently considered beta, and during the beta you’re allowed to continue with the trial as long as you keep installing the updated versions. Your $50 license if you purchase one can be used on any of the platforms, including the Linux version.
MakeMKV simply copies the movie to disk and works for DVD or Blu-Ray. Due to this, you’ll need to have enough space to hold the whole movie (again, 30+GB). Once it is decrypted and written to disk, you can then use Handbrake or any other transcoding software to recompress it to a smaller size. The final size will depend on the codec you choose and the resolution to encode it for, but my Blu-Ray rips often end up a few Gigabytes. You can also just choose to put the raw rip on your server if you want to put a lot of disk space aside for the media.