Toward Efficient Multicodec Streaming

One of the biggest challenges in modern-era media delivery systems is the fragmentation of the population of receiving devices in terms of codec capabilities. For example, modern Apple devices can decode and seamlessly switch between H.264/Advanced Video Coding (AVC) and high efficiency video encoding (HEVC) streams. Most new TVs or set-top boxes can also decode HEVC, but they cannot switch between HEVC and H.264/AVC streams in the same streaming session. And there are still plenty of older devices/streaming clients that can only receive and decode H.264/AVC streams. With the arrival of next-generation codecs, such as AV1 and Versatile Video Coding (VVC), the fragmentation of codec support across devices becomes even more complex. This situation prompts the question—how can we serve such a population of devices most efficiently by using codecs delivering the best performance in all cases yet producing the minimum possible number of streams such that the overall cost of media delivery is minimal? In this article, we explain how this problem can be formalized and solved at the stage of dynamic generation of encoding profiles for adaptive bitrate (ABR) streaming. The proposed solution effectively generalizes the per-title or context-aware encoding (CAE) class of techniques, considering multiple sets of renditions generated using each codec and codec usage distributions by the population of the receiving devices. We also discuss additional system-level means (proper manifest generation, HLS and Dynamic Adaptive Streaming over HTTP (DASH) quality annotations, device detection, and edge logic) needed to make the proposed solution practically deployable.