=========================== Title: =========================== File formats for audiovisual preservation: How to choose? =========================== Abstract: =========================== Every memory institution dealing with audiovisual material must face these questions sooner or later: Which file format shall we store our collection in? Which video container? Which video codec? Lossy or lossless encoding? What can be opened in the future? Have we made the right choice? No matter whom you ask, you will get different answers. The answers might be correct, but they they might not be the right solution for your use-cases. There is no evergreen format. There is no one-size-fits-all solution. This talk addresses what to look out for when choosing a file format: Now and in the future. =========================== Paper =========================== Introduction =========================== Which properties should an archivist look out for when choosing a format for long-term preservation of audiovisual materials? When asking other professionals in that domain, they often refer to a format which is promoted and supported as industry standard. Or one is simply choosing to use what others are using for the job. Today we are living in a time wheretechnology is becoming more and more complex, more intransparent. Especially digital video is a challenge for preservation, since it combines all things to be considered for preserving images, audio and metadata. Additionally, audiovisual material requires vast amounts of storage and data bandwidth. Non-trivial, even with today's existing hardware. Different institutions, different use-cases --------------------------- In difference to digital audio audio formats, there is currently no "one size fits all" format to be used for digital video. Before we continue, it is important that the reader is aware that there are different use-cases and needs of professional institutions dealing with audiovisual material. The main ones are: * production * broadcast * preservation Production and broadcast institutions also want preserve their own collections, but their main intention is not necessarily preservation. This should be kept in mind, as it means that their focus is not on the best preservation, but rather on short retrieval or archive-to-air times. In this paper, we will focus on the use-cases and needs of professional preservation of audiovisual material. Mainly video, but the basic principles can be applied to film and possibly other media types, too. Video: An endangered species --------------------------- In comparison to many other formats that we archivists try to preserve, video is a different challenge on its own: * Very short market relevance time, compared to other media, such as audio or film for example. * Therefore: * Replayers discontinued * Lack of spare parts and service manuals * even for mainstream formats (e.g. VHS) * Video professionals mainly in broadcast * Film is very different than video * Electric tape layout complex to understand * and many more... Digital video primer --------------------------- Digital video always consists of at least 3 different entities. Let's call this the "digital video trinity": * Video codec * Audio codec * Container For proper preservation purposes, it is also important to consider a fourth entity: * Embedded metadata There are pros and cons regarding embedding more metadata into a videofile, since it adds additional complexity. This limits the number of applications that will be able to properly read or write certain metadata, which also increases dependence on certain implementations of hardware/software. For several reasons, this article will not go into details about pros/cons of embedded metadata. The video container format is often what users see as the file ending of a video file. Popular examples for video container formats are "AVI", "MOV", "MKV", "MXF" or nowadays more and more "MP4". Unfortunately, it has become so common to use the name of the container only as answer to the question "which format?". This leads to a lot of uncertainty, confusion and problems when speaking about video formats, because it's actually necessary to always consider all three components when speaking about "a video format": Videocodec, audiocodec and container. This is by the way similar in audio, where "WAV" is merely the container format - and the actual encoding/codec used is almost always PCM uncompressed, but could also be something different. Even "MP3" as codec in WAV is legal, according to the container specification. Using uncompressed PCM as audio codec is perfectly possible and well supported in audiovisual files. Therefore, this article will only focus on the remaining 2 entities which are most unclear - and also have the most impact for preservation: * Video codec * Container Defining technology-neutral properties for long-term preservation =========================== Technology will always change. This has a direct effect on availability, support and sustainability of tools to handle collection's contents. At the end of the day, one must make a choice for technology being available to them today - but one day this technology will be outdated or unavailable. Therefore, it is a good idea to define a list of property-requirements that a format for storing ones collection must have. Independent of which technology provides it. This makes it possible to apply the same set of requirements whenever a new technology, a new format must be chosen to migrate to. Since there is no evergreen format, migration to a yet-unknown format in the future will be necessary one day. By defining a proper set of perservation format requirements, obstacles for migrating to future formats can greatly be reduced. Take written text for example: The "technology" being used over ages changed dramatically: * carving in bones, stone, wood, etc. * drawing on stone, papyrus, wood, paper, leather, etc. * and many more... Yet, the required properties for sustaining the content stayed the same: * Visually perceivable contrast to store and retrieve the information * Means of understanding the visual content: symbols / language These requirements could then also be applied to digital technology: From basic text files to more complex digital document formats. Property list ----------------------------------- The properties listed in this chapter are based on a list of desired properties for video material that was put together by technicians at the "Österreichische Mediathek" [Link: mediathek.at], Austria's national audio/video archive. Their motivation and necessity for writing such a list came up in 2009 as a result of the evaluation phase when considering buying a system for bulk-digitization of their video collection [LINK: DVA Doku]. After talking to, and questioning many other memory institutions as well as broadcasting archives, it turned out that there was no commonly agreed upon method - or formats - for preserving video in digital form. In order to make it easier to apply and use this list, it is here split into 2 different categories: 1. Signifcant properties These are the properties which are considere significant in order to maintain the actual content as accurately as possible in a technical way. 2. Preservation-improving properties These are properties that increase the chances for long-term preservation. Some of these properties are there to avoid unnecessary issues that may cause additional efforts or problems when dealing with a format in the future. Such as dealing with format obsolescence or future mass migration. ### Significant properties * No digital loss (=lossless compression or uncompressed) * Resolution independent (SD, HD, and beyond...) * Aspect ratio preserving * Colors as native as possible ### Preservation-improving properties * Handleable data amount * Non-proprietary * Hardware independent * Minimalistic standard Properties in detail ----------------------------------- ### Significant properties #### No digital loss With analogue material, even high-quality copies were a degradation compared to the original. One of the major benefits of digitization is the fact that the content can be copied infinitely without any loss. In the audio domain for example, it is already common to record and edit recordings in a digitally lossless way. Non-professional, consumer devices use lossy compression for recording in order to save space. Yet, in professional environments it would be considered inacceptable to record or edit audio material using lossy formats, such as MP3 or MP4, for example. Regardless how good it may sound to the listener. For audio material archiving, it is therefore mentioned as requirement in the technical guidelines of [TC03](TODO Link TC03 quote), that only uncompressed data, or lossless compression is acceptable for proper long-term preservation. For digital video material this is currently the other way around. The reason for that is that the data size of digital video is still non-trivial to handle by current technology. Here are short examples of the filesizes of digital video: 1. PAL Standard Definition (SD): * Video: * 720 x 576 pixels resolution * 25 frames per second (fps) * 8 bits-per-component YUV * 4:2:2 chroma subsampling (=16 bits per pixel) = 720 * 576 * 25 * 30 / 8 Bytes per second = TODO: calculate this in MB/GB per second 2. PAL High Definition (Full HD): * Video: * 1920 x 1080 pixels resolution * 25 frames per second (fps) * 8 bits-per-component YUV * 4:2:2 chroma subsampling (=16 bits per pixel) = 1920 * 1080 * 25 * 30 / 8 Bytes per second = TODO: calculate this in MB/GB per second These are only illustrative data rates for the image data only. Assuming SDI standard for audio as example, with 48 kHz and 24 bits resolution, that is another XXX MB per second (TODO: calculate this!) to add for each audio channel present. One must also consider that additional space might be required for audio tracks, higher framerate (e.g. 50fps), less chroma-subsampling (e.g. 4:4:4) and higher bits-per-component (bpc) sample depth. For film at resolutions of 2k at 14bpc and more, you can imagine the impacts not only on storage, but also CPU processing and network performance requirements. #### Resolution independent Although there are are numerous standard resolutions common for certain video formats, it is desirable to have a preservation format that is able to store arbitrary resolutions. This makes it possible to store any material as natively as possible without the need to resample the image data. It also allows for using the same format for future resolutions not common at the day of choosing the format. In practice there are cases where the video format definition is not limited to certain resolutions, but the actual implementation of a hardware or software might be. #### Aspect ratio preserving anamorphic, letterbox, pillarbox, widescreen, etc. #### Colors as native as possible colorspace subsampling ### Preservation-improving properties #### Handleable data amount Given the huge amounts of data size of audiovisual material, even if it is just video and not even film, several entities of an institution's workflow must be considered. There is no sense in choosing a format that one cannot handle given the current technology- and/or budget-limitations. The main entities to be considered are: * storage size * network speed * disk speed * data bus speed (RAM, etc) When talking about digital video preservation formats, the most prominent factor mentioned are often only the storage costs. Although storage size is one of the most noticeable cost factors, it is also important to consider the impact of these data sizes on other infrastructure requirements within an institution, too. Even if a regular office-grade PC is able to playback video with Full-HD resolution smoothly in realtime over a limited bandwidth, such as an Internet connection for example, the same hardware might stutter when trying to play the same video in an uncompressed format. Therefore, one might want to go through the following checklist: * How much storage space is needed for 1 copy? * Plus: At least 1 backup? * How much data throughput (MB/s) is required for smooth real-time playback? * Which network speed do I need for this (1GB, 10GB, etc)? * Do I have enough free network bandwidth for storing daily ingest + backups? * How many concurrent users do I have, accessing the material in-house? NOTE: One very important aspect though is, that one must not forget that preservation properties and quality should not suffer. Even if this means additional costs. A very common misconception is to use high-quality lossy compression, although this is, again referring to TC03 guidelines for audio, not recommended for long-term preservation. Regardless if it might look "good enough" for now: Looking at video encodings done in the recent past of only a few years ago on today's equipment, one can immediately see the impact of the choice for smaller size back then. Even though it was the best available technical solution existing back then. See the chapter on "Eternal migration" for additional details and things to be considered, caused by generation loss. On the other hand, using lossless compression formats the gain in size due to compression might allow smaller storage (or more backups), as well as using existing network infrastructure for the same amount of playback hours and concurrent users. #### Non proprietary At the time of this writing, the default of implementations available for dealing with audiovisual material are proprietary. This means that an end-user does not get information about the inner workings of the equipment they use and require for handling their own material. If one is using proprietary technology or formats, there is a great dependence on the good-will of that manufacturer. This might often not be noticable until a company seizes to exist, or one wants to migrate to another technology - or another product from another vendor. This dependency is called "vendor lock-in". This is not a new concept, yet the nature of older technology - even electromechanical - allowed users to use, reverse-engineer and modify such equipment in ways that did not need the explicit aid or consensus of the original manufacturer. This is very different with digital technology and modern electronics: In the past skilled engineers employed in archives were able to understand and even fix problems, by applying common-sense and their electromechanical understanding of things. Nowadays, try to find out why a digital video works in one application, and fails to render correctly in another? As we can see in practice as well in other areas of preservation, archivists need to be able to maintain their equipment independent of whether a certain technology or format is still actively available on the market. Just take a look at what is necessary and current practice for keeping old equipment alive and working for reproduction of media such as audio tapes, vinyl records or film - to mention just a few prominent ones. In the past it was common for professional equipment to come with so called "Service Manuals". These included schematics and often detailed information about the machines themselves, as well as how to modify, adapt and repair them. Taking a look at all fields of long-term preservation, in order to preserve and access old material, archivists are still "hacking" together clever solutions to challenging problems on a daily basis. With great success. The same requirements and necessities are to be applied to electronic and digital equipment. For some reason, it has become increasingly common by vendors to reduce and withhold technical information from their customers. Up to a point where they become even unfriendly one asks them for specification or implementation details necessary to access or migrate collection material. Choice for technology used in archives is getting increasingly linked to the product market lifetime: An average of 3 to 5 years. I hope that the reader will agree that memory institutions define "long-term" as a much more longer period that that. It is therefore obvious, that this dependency must be kept as little as possible by archives in order to be able to fulfill their task of proper long-term preservation. Imagine one could archive not only existing equipment, such as replayers for example, but also the schematics and building components required to use, study, share and improve their equipment as it fits ones needs. For software, this is possible already today. The license definition of "Free and Open Source Software" (FOSS) states that certain conditions must be provided to the end-user at all times. This is defined by the rights to: * use * study * share * improve Every computer program is built out of its "source code". This source code is nothing else than just written text, interpreted by a computer. Having a copy of the source code of the tools used to create or open digital formats, makes it possible for developers to adapt it to any future technology. Even if not known yet. This is the equivalent to what has always been done in the past to make content stored on outdated technologies available until today. Yet, the fact that software is only written word coming to live, archivists today have the advantage that they are not limited by physics or huge machinery to rebuild the apparatus necessary to maintain their collection's formats. Therefore, using FOSS for preservation purposes, counteracts issues such as format obsolescense or vendor dependency. #### Hardware independent #### Minimalistic standard As simple as possible. As complicated as necessary. Try before you buy =========================== ### Concept * A set of tests to perform before choosing a system/technology * Only use hardware/applications *not* originating from the same manufacturer * Due to above explained reasons, use FOSS and Open Hardware wherever possible ### Testset * Can you play it back, using applications from other manufacturers? * Is propietary hardware required to properly play it back (e.g. decoder card)? * If metadata is embedded: * Which metadata can you access/extract using applications from other manufacturers? * Can you transcode the file to lossless/uncompressed encoding using ree Software / Open Source tools? * For playback with other applications, or transcoded copies: * Is audio/video synchronous throughout the playback? * Try to get the format specification and/or source code of the implementation. * Can you edit and export the file in applications from other manufacturers - without any digital loss? Eternal migration =========================== Things to consider: * Container migration * Codec migration * Metadata migration * Generation loss * Color conversions / resampling