A mass migration is normally a natural phenomenon witnessed as herds of caribou, say, or flocks of geese move to warmer climates in search of food. It’s a perilous journey, fraught with risk as the animals cross thousands of miles, but they must push through to simply survive. Most of us would never see these incredible events if not for the amazing work of the production crews that film and produce documentaries. The Media industry does a fantastic job of bringing the incredible natural world closer to all of us.

Behind the scenes in the media industry there is another type of mass migration that takes place: the movement of content from an older form of tape to a newer form of tape, complete with upgrades to tape read/write devices. This important activity protects the valuable content from being damaged or lost on degrading tape. Also there are limits to backwards compatibility and support for old read/write hardware which requires an upgrade.

Migrations have not traditionally been from tape to disk, only from tape to tape. Why? Relatively, disk has not been as cost-effective as tape for storing “archived” content. The definition of cost has traditionally been based on the price of the storage medium itself, and effective has related to the firm belief that the content is too valuable to throw away. What if we needed this again someday? After all the investment to make the content, it seems a shame to not keep it for the sake of a few dollars to pay for the tape that holds it. And so content is stored just like we would keep family heirlooms in a box in the dusty attic.

At the same time, the reality of where we store most content looks more like a busy kitchen rather than the dusty attic. Tapes live in tape libraries, with robotic arms moving tapes to the tape machines. Tape machines are continuously reading and writing content to and from production, post-production and playout operations. In many mid to large broadcast operations, 5-6 terabytes of data per day are being written in and read out of the tape library. This is the fast-paced environment where migration has to take place.

The perils of migration

Many organizations see this migration process as costly and time-consuming, while some see it as perilous. The extra pressure of migrating content while managing normal operations is a load on the total system that has to be carefully managed. Professional services engagements ensure this process is safely managed, overseeing the operational downtime required to migrate. Luckily this process has only had to happen every few years. But what does the future look like?

First, a wrinkle in the perennial tape-to-tape migration process has been created by the introduction of LTO-8 hardware and the lack of support for LTO-6 tapes. So now an operation planning to move to LTO-8 has to either migrate fully from LTO-6 to LTO-8, or they have to stand up a separate island of LTO-8 storage hardware and tapes. This forced migration earlier than expected is an issue for many.

Second, as measured in terms of volume of data, content creation is growing fast: Estimates from the industry are that tape libraries of a few petabytes will grow by 2-3 times over the next 3-5 years. Simply storing an additional 5 terabytes per day will add 1.8 petabytes per year. And this doesn’t account for producing content in 4K, which will quadruple the average file size. This is just over the next few years. When considering the next 10-20 years, the first big question is how big will the library get? The next big question is if the library gets that big, what will the migration process look like?

It’s feasible to imagine a time when a mass migration from tape format to tape format becomes so relatively long and costly that it’s not worth doing. The LTO roadmap highlights that each generation a tape’s raw capacity will almost double, while raw transfer speed will only increase by about 50 percent. The net effect is that it will take 25 percent longer to read a whole tape. So migrations become longer and costlier not because of the cost of tape, but because of the migration process itself. The only way to go faster is to increase the number of read/write devices in the robot. Then what happens? Does the content get left to die, or does it get migrated at any cost? Fortunately, an alternative solution has emerged in recent years, and it’s ideally positioned for the Media industry.

Object storage to the rescue

Object storage companies entered the market in the early 2000s to provide capabilities not supported by file storage architectures, like interfaces that can be directly programmable by the application and a namespace that can span multiple instances of physical hardware. An overarching goal was to support the exponential growth in digital media driven by consumer mobile devices and the internet. It simultaneously addressed multiple business requirements for long-term retention of unstructured data (i.e. documents, photos, videos) in a secure, protected, accessible, scalable format.

The concept of cloud storage services was created around the same time, utilizing object storage systems and delivering storage as a service. While object storage is relatively new, it is an old practice to outsource your data to a service provider and make your storage evolution their problem to manage. While the appeal of a cloud storage service to the Media industry is high because of its flexible and scalable nature, there are also challenges to be addressed because of the Media industry’s relationship with its content. Some content is high touch, it is difficult to predict when certain content will be needed, the file sizes are large, and when content is needed it’s generally needed very quickly.

Additionally, the concept of “software-defined storage” evolved, that utilized COTS (commercial off the shelf) storage hardware. Some leading storage companies produced their own proprietary object storage solutions (e.g. EMC, NetApp, HDS), while the opensource community developed the powerful object storage engine, Ceph. Today, you can choose from a proprietary object storage solution with COTS hardware in an appliance solution, through to a do-it-yourself open-source object storage solution that you deploy yourselves on your chosen hardware. The middle-ground is an open-source object storage solution with tuned management tools for maximum ease of use. Whichever path you take, performance and price can vary significantly so it’s important to look closely at the solution architecture to find the best object storage solution.

The right solution

Media organizations are therefore searching for the right solution to meet the often conflicting requirements for long-term retention, data protection, operational control, migration avoidance, fast access, and value for money. It sounds like a wish list, and indeed it is in a tape world, a file-based storage world, or a cloud service world. But in an object storage world, it’s not.

How? Object storage has a few defining characteristics that make it an ideal fit for the Media industry. Object storage scales without pre-determining the architecture, just like cloud storage is designed to do, supporting a business’s natural need for flexibility. Object storage uses COTS hardware to achieve a range of cost-performance points that support the whole spectrum of “archive” to “online” requirements. Object storage upgrades in an erasure-coded cluster mean that content is simply rebalanced across hardware while maintaining full operational capability. Object storage scales capacity and performance simultaneously, so you won’t run out of read-write performance as your library gets bigger. Object storage protects data and metadata by spreading it across the available hardware, providing a robust resilience-to-accessibility ratio. And perhaps most critically, object storage makes content always available for the “on-demand” world that Media companies are serving.

Object storage is therefore an ideal fit for Media organizations driving to the next level in revenues and efficiencies. So maybe it is now time to redefine what cost-effective really means to your business. A cost-effective solution in this “on-demand” era should help you to access new revenues and reach new levels of efficiency, not one or the other. New revenues and some efficiencies derive from better access to your content. But stopping the perennial mass migration will give time back to your organization to focus on the most important thing: winning and not just surviving.