Architecture for Embedding Audiovisual Feature Extraction Tools in Archives

Soon, it will no longer be sufficient for only archivists to annotate audiovisual material. Not only is the number of archivists limited, but the time they spend on annotating one item is insufficient to create time-based and detailed descriptions about the content to make fully optimized video search possible. Furthermore, as a result of file-based production methods, we observe an accelerated increase in newly created audiovisual material that must be described. Fortunately, high-quality feature extraction (FE) tools are increasingly being developed by research institutes. These tools examine the audiovisual essence and return particular information about the analyzed video, audio, or both streams. For example, the tools can automatically detect shot boundaries, detect and recognize faces and objects, and segment audio streams. As a result, they quickly and cheaply generate metadata that can be used for indexing and searching. In addition, they relieve archivists of the need to perform tedious, repetitive, but necessary low-added value tasks, such as identifying within an audio stream the speech and the music segments. Although most tools are not yet commercially offered, these solutions are expected to become available soon for broadcasters and media companies alike. This paper describes a solution for integrating such FE tools within the annotation workflow of a media company. This solution, in the form of an architecture and workflow, is scalable, extensible, and loosely coupled and has clear and easy-to-implement interfaces. As such, our architecture allows additional tools to be plugged in irrespective of the software and hardware used by the media company. By integrating FE tools within the workflow of the annotating audiovisual essence, more and better metadata can be created, allowing other tools to improve indexing, search, and retrieval of media material within audiovisual archives.