During the last decade, volume of industrial video and audio recordings has been growing exponentially. IDC, a leading market research firm estimates that total amount of stored digital information is around 2.8 Exabytes (2.8 million Terabytes) with tenfold expected growth every five years! This huge and growing volume sets difficult tasks for the industry of media archive management (MAM). An amount of data, both current and archived, stored by a single TV production company can soon reach thousand Terabytes (a Petabyte). This information needs to be securely stored, managed, and, most important, quickly and efficiently searched through and accessed.
In this paper we will not discuss pros and cons of different storage media. Instead, we will explore problems and solutions of archive storage management. The main issues with storage and archive managements may be roughly classified into four categories.
- The unusually big and growing volume of storages;
The storage size requirements for video and audio increase constantly mainly because of two factors: increase of broadcast volume and increase in definition. There is no doubt that the advent of High Definition TV will not end the 'quality race', and picture improvement requirements and expectation will grow to match, at least, those provided by 3D OMNIMAXes.
- Heterogeneous and distributed nature of archives;
While, storage media itself become less and less expensive, the cost of total archive upgrade to new type of media is prohibitive for any media company. It is the ubiquitous situation, that archivists and engineers face problem of managing and integrating data stored in legacy formats, such as tapes, with up-to-date archives on hard disks, solid media, etc.
- Need for fast search and retrieval based on metadata;
Fast search and retrieval of information from distributed heterogeneous archives is a headache for archivists and company IT personnel. Data accumulated during decades need to be quickly and efficiently found in archive by brief descriptions, keywords, date of creation, etc. TV production engineers also benefit from short low-resolution extracts from the footage called proxies, which majority of outdated storages fail to provide.
- Reliability and impenetrability of storages.
The last, but not least problem is insuring that all data are stored safely and securely. By safety, we understand the protection against data loss due to system malfunction(s). Any archival system should be able to mitigate hardware failure and restore partially or completely lost data with minimal efforts. Data security functions protect data against an intruder striving to obtain and use data illegitimately. Since many investigation and footages can be confidential and damaging to the company or a third person, the issue of safeguarding them is of utmost importance.
There are several solutions on the market, addressing all or majority of problems discussed above. They range from fragmentary 'patching' approaches trying to reconcile legacy with current and future requirements, to claiming to be universal 'fit all in one box' approaches.
One of the possible solution to the storage and archive problems is use of proprietary file systems. Huge data storage requirements may be easily and comfortably accommodated by a file system providing either convenient way of placement an retrieval of complex data into a single virtual storage, or by providing a user mode interface presenting data stored elsewhere as a file and folders similar to “ordinary” file systems. Neither user, nor legacy third party application have to worry how these data are actually stored and retrieved. This task is left to developers implementing all required callback functions.
The same approach – use of proprietary file system – helps with heterogeneous legacy storages. There is no need to invest large sums of money to transfer all archives onto storage media once or twice a decade. A file system based on callbacks allows easy extendability without sacrifice of the past development. Upon appearance of new storage formats or attractive media opportunities, only several new functions need to be implemented, while the system as a whole continues to work with minimal, if any, service interruptions.
Needless to say, the handling of metadata by widespread popular filesystems is less than efficient. There is no easy way of searching for necessary piece of information by user defined non-standard multi-modal annotations. Efficient implementation of custom task-specific file system will permit creation and storage of any metadata associated with original footage: starting from additional manual annotation to complex proxies, extracts, etc. Any sort and retrieve operations based on these metadata will not require development and implementation of additional technologies.
Data integrity and reliable storing may be assured by using journaling function provided by custom file systems. Constant monitoring and recording of all data-related operations make failure recovery a feasible task. This journaling can be performed on-a-fly without human involvement and in resource-economical way, thus making possible continuous data protection.
Data security is assured by use of modern encryption/decryption algorithms. Many symmetric key based tools, included in as a standard option in custom file systems, may quickly and easily encode and decode files of any type during upload or download process. This will efficiently protect the data from many attack vectors, save human factor.
Temporary data integration solution designed to patch current deficiency are not expendable and may be cost-effective only in a very short run. Modern developments of file-system software significantly simplify development, implementation, and use of large archival system with legacy issues, such as media data storages. These technologies help to bridge the legacy chasm in th most efficient way.