Coyote Blog

Because this is a topic that will only be interesting to some, and because it has gotten so long that it fills most of the home page, I am putting the article on how I ripped my home movies and created a video streaming system around XBMC below the fold.  For those who are not sure if they want to bother clicking through, here are some teaser photos of the media center I ended up with:

By the way, I know that in 10 5 years, this will all likely be superseded by streaming accounts.  For the time being, I have fun with this.

I have a ton of DVD's and BluRay disks.  Quite a while ago, I decided that I wasn't really using my collection because it was too tedious to seek out movies in the notebooks where I stored the disks.  So I embarked on an even more tedious task, but one with a substantial reward at the end - I ripped every singled DVD and BluRay disk to a digital file.  I had done this for my 600 disk CD collection, so I thought I was prepared, but it was still an act of will to get through them all.

In doing these rips, I applied no compression and did no file conversion (things go faster that way).  If I were to do it all over again, I might convert them all to .MKV files, but instead I simply ripped them to a mirror image of the file structure on the disks (you will see this listed sometimes as video_ts or _ts format).  I used DVDFab to do the ripping.  It removes the copy protection and makes an exact copy, either of just the movie or of the whole disk with menus and special features  (note that many streaming systems like Sage and XBMC can display Bluray movie files but cannot display bluray menus).   I have found DVDFab to be awesome, and they are constantly releasing updates so you can continue to copy your own disks for your own use, despite changing encryption technologies.  I laughed the other day -- I bought a Hunger Games Bluray about 2 days after it was released, and DVDfab had already released an update to handle a new copy protection dodge on this particular disk.  All that effort to design a new scheme and it lasted about 24 hours.  In DVDfab, I use the "copy" option either just to get the main movie or for the whole disk.  In DVDFab parlance, "copy" preserves the DVD or Bluray file structure, while "ripping" converts the DVD to a different file format, level of compression, and/or encoding.

Ripping to uncompressed files means that I have the flexibility to change file formats in the future, since any DVD ripper reads a video_ts folder on a hard drive as identical to a DVD.  So I could, say, batch convert them all to MKV some day or rip them to smaller files for my iPad.  I just have not found a reason to do so (and one of the downsides of format conversion is I tend to lose the chapter breaks in movies -- ripping to video_ts preserves the chapter breaks for me).  The choice of video file format had one definite affect on my later choices -- Windows Media Center, perhaps the obvious choice for media organization and streaming (since it is already on my windows 7 computers) will not stream video_ts files, and likely never will (this is not a technical issue but one of copyright sensitivities).  So this option was ruled out early on.

By the way, you will save yourself a LOT of time later if you follow a couple of simple rules for organizing your digital copies or rips:

1. One file folder per movie.  The file folder should be named the same as the movie, e.g. "Memento".  In that folder will be either the movie file, like "memento.mkv" or folders video_ts and audit_ts if you ripped to that format.  I can't remember the Bluray file format.
2. One file folder per TV series, with the series name.  One file folder within that per season, with names like "Sherlock - Season 1".  Within that season folder, one folder per episode with a name like "Sherlock S01Eo1" for Sherlock season 1 episode 1.  If each folder is a DVD with multiple episodes, it should be named "Sherlock So1Eo1Eo2" meaning the folder includes a disk that has both the first and second episodes of the first season of Sherlock.  You will thank me later.  No one warned me and I had to go back and change hundreds of folder names.

As the ripped files added up, I needed a place to store them with a LOT of space.  A typical DVD holds 4-8GB, and a Bluray disk can hold up to 50GB.  Ripping these uncompressed means that 150 movies or TV show disks might be a terrabyte (TB), even with just the main movie and without special features, and just 30 bluray movies might take up another terrabyte.  For this reason I took an old windows computer from work (it does not have to be very powerful) and made it my media server.  It now has 2 RAIDs in RAID 5 configuration each with 5 2-TB hard drives, giving me about 16TB of total storage (I used Sans Digital enclosures which are an older model but something like these).  You could also use an NAS or investigate an inexpensive PC-based RAID-like solution called unRaid.  Either way, you have to have redundancy.  If you have 10TB of files, you are not going to be able to do a hard drive backup.  And if you spent the brutal hours ripping enough disks to get to 10TB, you are not going to ever, ever want to do it again.

All of what follows presupposes you have a home network.  You can certainly do this without a network on a standalone PC at your TV, but all this work makes more sense if you can leverage the effort over multiple screens.  If you have a network, you can probably get away with fast wireless or very fast powerline networking to play movies ripped from DVD's and standard definition material.  But my experience, and that of many others, is that you will need a hardwired (cat 5e or 6) network to reliably stream 1080p and bluray.   You may not believe me, since the data rates you need for even 1080p/60hz streaming are lower than that published for many wireless N products.  But unless you are very unique, you are not going to get anywhere close to those specc'd data rates.

OK - we now have a nice fast network and a bunch of video files on a server.  All we have to do is get them to the TV.  There are many boxes out there that will do this.  I have tried a Western Digital box and it worked OK, for example.  GoogleTV has some useful functionality.   AppleTV will fulfill some of this role, but only if you rip to iTunes and live full-time in the iOS universe.  What I did not like about these boxes was the operating system.  Navigation through large libraries was not easy, and they did not have menus of movies along with movie posters, lists of actors, plot descriptions, etc.  And the navigation screens were ugly and generally could not be customized (I once saw a $75,000 Kaleidescape system at someone's house that had all these features and set that as my goal, just for a lot less money).

This is a screen shot from a Kaledescape.  With a lot of work, my installation is going to be better, at 1/10 the cost

Several years ago I settled on SageTV as my streaming solution.  At the time, Sage was a company that combined powerful yet inexpensive set-top boxes with an operating system that did all the things I wanted it to do.  The operating system ran on one's main movie server and then was mirrored on set top boxes that streamed the video and decoded audio formats (e.g. DTS). It also had extensive TIVO-like PVR functionality if you had a tuner card in your PC  (since I use DirecTV, and since a DirecTV PC tuner card is long-rumored but still non-existant, I never tried the PVR stuff).

Sage was less of a polished product and more of a hobby.   It was for folks who wanted (and were willing) to constantly fiddle around with the carburetor to keep it running well.  I loved it.  At least, that is, up until the day over a year ago when Google bought the company.  It went dark, and has not been heard from again, at least officially.  A dedicated community of users and hackers still kind of keeps it going, but you can see the energy falling off every month.  I needed something else.

XBMC's origins can be found in its original name - XBox Media Center.  At one time, a user group found a way to put some software on the Xbox to make it a fairly powerful video streaming and media solution.  Over time, that user group has grown and morphed  (to the point that they recently released version 11 of the software and it actually no longer supports the Xbox any more as a platform).  Currently there are Windows, Mac, Linux-Ubuntu, and most recently an Android version of the software.  If you are willing to root your AppleTV2, there is a version that runs on that little box  (it does well up to 720p but chokes on 1080p unless you rip your files to a very specific format.  The AppleTV3 should be powerful enough to run everything, but to my knowledge no one has figured out how to root it yet).   The great thing is that all these platforms use the same basic core, so that, for example, a configuration file to connect your media server properly on an Windows installation is the exact same file one would use for this purpose on an AppleTV or Linux installation.  Get one installation working, and you can drop your configuration files on all future instances, no matter what their OS, and they will likely work as well.

Over time I have successfully installed XBMC on Windows boxes, on an AppleTV 2, and most recently in Linux installations.  The point of this is not to write another tutorial.  The XBMC wiki, web site, and boards have plenty.  But there are a lot of choices as to how to pull this off, and I wanted to share some design lessons I have learned that makes this process easier.  Like Sage, this is not a polished product that works out of the box  -- though the Openelec version I will discuss in a minute comes damn close.   No single problem I faced was ever technically too hard, but the challenge was in the breadth of issues I eventually faced - from graphics drivers to video formats to OS quirks to networking issues.

My first step was to install XBMC on my computer in my home office, which runs Windows 7.   An early decision one needs to make is whether each XBMC will maintain its own library data base (list of movies, meta data, etc) or whether one will have a single database for the library that all XBMC installations will draw from.  This latter is a bit slower and more complicated to pull off, but was the approach I chose.  It is definitely worth the effort to only have one database to update.  Otherwise, every time you load a movie onto the server you would have to go to every TV and update the database there separately.

Anyway, starting on a Windows 7 computer and working with a video server running windows 7 simplified things for me at first, because I know Windows well.  It also is much easier to learn the system and get networking functioning while dealing with only one OS  (networking through multiple OS's just adds complexity, particularly in the way each does file shares).  I followed the instructions in the tutorial for setting up a shared mysql database on my media server (this article is also helpful).  I had never before messed with mysql except on my web sites, so I knew little about it, but the instructions were just like a cookbook and following them got me what I wanted.  I then created the advancedsettings.xml file, per the instructions, for the XBMC installation on my office computer that told it to use that remote database.  Then, I added all my shared video folders to my library.

This is a time consuming process.  There are lots of good guides to this.  You may have to search forums for individual answers once in a while -- in fact, if you are not comfortable searching forums for the odd solution, this is not the right project for you.  If done right, XBMC will simultaneously read in metadata for your movies (and music and TV shows) including movie posters, trailers, fan art, descriptions, cast, crew, etc (basically all the stuff on IMDB) as it adds the file to the library.   This takes some time, as well as some manual intervention depending on how well your video files are named (the file name is generally being used to perform the search, so a folder named "memento" will likely find the right metadata but one named "video_123_c" will not.  This is probably the hardest step.  You may have network issues (everything works much better, for example, if you permanently assign IP addresses to these devices rather than let them drift around).  You may have file sharing and permission issues  (make sure all computers in the same workgroup, make sure all video folders are set to share with all users, you may need to specify user names and passwords, etc.)  Use google search and the XBMC forum search function to find fixes for problems  The answers are probably out there.... somewhere.  Again, not for the feint of heart.  But don't go forward until you are successfully playing movies on one computer that are streamed from another.   The good news is once this is working, additional installations are (mostly) a snap.

Let me give you one example of the type of issues that sometimes must be overcome.  All the above worked pretty well for me except my TV series disks.  If I had, for example, four Firefly disks each with four episodes, I ripped the entire disk with menus rather than rip each episode as a separate file.  However, this is a problem for XBMC - it thinks each file is a single episode.  However, you can fix this problem, but it took me a lot of forum digging.  The answer was here, and required me to rename some files and then add in a new section to my advancedsettings.xml file (see section in article on "TV shows in DVD folders")  The Firefly example reminds me of another problem -- when the metadata was getting paired with files on Firefly, it got confused because it was using "as aired" episode order and the DVD order was different.  This is fixable, it just took some searching.  Again, though, the good news is once I had this set up right the first time, I just used the same library and same advancedsettings.xml on each new installation and they worked great.

The advantage of the shared library now comes into play - I do not have to do this import step again when adding additional XBMC boxes.  All future installations of XBMC will be set to use the same database (by using the exact same advancedsettings.xml file).  When I add new movies to the server, I do a library update once from my desk in my office, which I think of as the maintenance console for the system, and again the changes appear everywhere.

So I added a couple of small windows PC's to the system at my TV's and home theater.   I built one for my main theater that is a mini-desktop (specs:  Apex mini-itx Mi-008 case, Asrock H67M-ITX mobo, Intel core I3-2100, GT430 low profile graphics card, 40GB SSD and 8GB memory) and bought a net-top on ebay  (Acer AR3700-U3002) for a bedroom TV.  Note I have passed over a few mistaken builds here -- these were not my first two attempts but are two, with the exception possibly of wifi under Linux, that I know to be compatible with all versions of XBMC, including the Linux ones, discussed below.

One problem with this approach is that Windows boxes are expensive, if nothing else because of Windows itself.  It is frustrating that nothing really approaches the price-value of the old SageTV set top boxes.  The rooted Apple TV2 was close, but I wanted better HD support.  If you only rip DVDs and standard def stuff and like the Apple environment, this may be your choice.  The rooting keeps all the Apple TV functionality, just adds a new XBMC menu choice.  Of course, you can never again update your AppleTV OS without breaking the installation, so that might be a problem for you.

I could have lived with the cost -- after all, I bought the boxes -- but there is another problem with Windows.  No matter what you do, how many automatic functions one turns off, how much you strip down the computer, how many startup delays one adds -- Windows from time to time seems to want to minimize XBMC.  The program loses the OS's focus.   This is fine on my office PC, I have a keyboard and mouse and just maximize it again.  But for a set top box, I don't have these immediately available (just a remote).  And users may not know what to do when they encounter it -- my wife for example just throws up her hands.   And even with a windows remote, it is a hard problem to manage (there is a button on the Windows remote that is meant to serve the purpose of returning focus to media, but it is virtually impossible to have it do anything but open Windows Media Center).  All a long story here, YMMV.  Suffice it to say, Windows was not giving me the family-friendly works-every-time unattended user experience I wanted.

So after I found the hacked  Apple TV2 to be under-powered, I tried an ubuntu-linux install.  I was eventually thwarted by two problems:

1. I didn't know jack about Linux, so I spent a ton of time just learning basic stuff, like why examples of command line code used "sudo" more often than Hawaiians say "aloha."
2. Drivers are a big problem in Linux.  There is a reason why, for all its faults, Windows is a standard.  That is because every manufacturer makes damn sure that as a minimum, they have good drivers for Windows.  Linux - maybe, maybe not.

Eventually, the second problem killed me, using both the Ubuntu version as well as the XBMC-Live version, which is supposed to be plug and play but was not for me.  If I had been better at Linux, I might have overcome, but I did not.  But I learned a lot about compatibility that would help in the next phase of our story, when I come back to Linux from a different direction.   The key, I think, is to select components from the outset with good Linux compatibility and drivers.

• Use Nvidia graphics.   Either GT series cards or the ION chips in net-tops.   Period.  I love some of the new Radeon cards, but Nvidia driver support in Linux is way better.  I never got 1080p playing satisfactorily in any Linux installation, even the successful one below, on a non-Nvidia graphics chip.
• Windows remotes need to be totally on spec.  No deviations, and I will say that most manufacturers deviate.  The best I found was the AVS Gear GP-IR02BK.  Just use this one.  Trust me.
• Wifi adapters are a real crap shoot.  Even the same model might be built with two different chips, and one will work well in Linux and one will not.  I finally gave up and wired all locations.   If you must have wireless,  I did get this one to work and it appears to be as fast as any N adapter.

By the way, be careful on sound output.  Some graphics cards will not actually output audio over HDMI unless you install a special jumper.  Others work without a jumper.  On my installations, I use the optical output for 7.1 sound from my mobo to my main home theater reciever and on all the other TV's just output 2 channel over HDMI to the TV itself.

With my Linux failure, I went back to running windows and being unhappy about losing focus from time to time.  Then, I heard about another group called Openelec that was taking XBMC and packaging it with a very thin Linux core to create what was effectively an XBMC operating system.  The system boots straight to XBMC and is lightening fast -- on a net-top box I upgraded with a very small SSD, it goes from off to fully booted into XBMC in about 5 seconds.  Openelec is less well supported than XBMC, but the vast majority of problems you might have are XBMC problems and those can be solved trolling the XBMC boards.

As it turns out, Openelec is truly plug and play -- again, when you have the right components.  I installed it from USB onto  a series of boxes  (I actually took out the old hard drive with the working windows installation in case I needed to go back and installed to a new hard drive.  This is not so expensive as the drive can be really small, just 10-20GB or even less).  With the right hardware, I had none of the driver and file-sharing issues I had trying to install Ubuntu myself.  Whoever wrote this thing knows his stuff, because I could immediately see all the folders I needed on the installation as shares from my main Windows computer, right from windows explorer (via the IP address).  I copied over my tried and true advancedsettings.xml, the exact same file I am using on my windows installation, rebooted the computer, and it worked.  I have tweaked it since, getting the menus laid out the way I want, changing the skin, adding some add-ons, but it worked and streamed my movie collection with just two steps:  1) install openelec from USB 2) copy advancedsettings.xml.

I did run into one problem.  Well, a couple.  I struggled with wifi connections, but as I said, I have gone all wired so I sort of cut the Gordian knot on that one.  I also originally had a nettop with a Radeon Fusion graphics card that simply would not run 1080p in openelec.  This was the Acer Revo RL70, and it was a nice computer but I needed nVidia graphics for better Linux compatibility.  I sold it on eBay and bought the Acer Revo 3700 (listed earlier), actually an older version but one with a Nvidia ION2 card.  To date, both the net top as well as the more traditional build with the GT430 do absolutely everything right for me with openelec.  They are fast, never lose focus on the application, and just work.   By the way, the 3700 is very well thought of on the Openelec boards as one of the best, most compatible openelec platforms.  When buying a computer, search the Openelec boards to see what people are using.

As a final note, for those of you who need the PVR function, the new version of Openelec has a PVR version.  I have no idea if it works or how well, I have never tried it.

Here are some screen shots.  The first is the main menu that greets you on boot (click on any to see a larger version)

The background images rotate in the main menu (they are user selectable, as are the exact options show on the menu).  Here is what the movie list looks like when you click on the movie option above

As you scroll, you get art and a description of the movie your cursor is at.  One can also set up custom playlists that allow special movie lists, like this one for all animated movies (for when kids visit):

Clicking on any movie will bring up the play button, as well as movie information and other links, such as to cast lists and the trailer (if it is available online)

There are lots of ways to view the movie list.  I don't prefer it, but you can get a wall of movie posters to look just like Kaleidescape

Or something like the iPod coverflow:

We can also see movies in a particular genre

Or find all the movies in the collection with a particular actor

Here is how the TV show listing looks

And finally, for a given show and season, the episode lists and information

The result has been a fun system that really rewards the effort and expense.  Movies can be sorted by rating, date, or title.  We can find all movies with a certain actor or for a certain genre (e.g. animation).  For almost no cost in terms of extra time or cost, I also get all my music streaming to the same locations via XBMC.  My next thing to try is to work on the picture streaming piece to be able to do family slide shows on the main home theater.