Layered media coding technologies such as Scalable Video Coding (SVC) or Multiview Video Coding (MVC) generate video bit-streams with various media layers, each of them representing another level of quality. Due to inter-layer prediction there exist hierarchies between these media layers, where media layers depend on other media layers for successful decoding. Layer-Aware Forward Error Correction (LA-FEC) exploits the knowledge of the existing dependency structures within layered media streams. LA-FEC generates FEC data so that protection of less important media layers can be used with protection data of more important media layers for joint error correction. In this way, LA-FEC enables new functionalities and increases the protection of more important media without increasing the total amount of data.
Layered media streams
Figure 1 shows an exemplary layered media stream with two media layers, base and enhancement layer, such as produced by SVC or MVC. The base layer (BL) represents the base quality, which is in both cases, SVC and MVC, a video stream with a resolution of 720p. An additional integration of the enhancement layer (EL) in the decoding process improves the video quality. With SVC, the quality increases to a resolution of 1080p. With MVC, decoding of the EL adds a second view for 3D television. As shown in Figure 1, there are dependencies between the media layers. The BL can be decoded independently from other media layers while the EL needs the BL for error-free decoding.
Forward Error Correction (FEC) for layered media codecs
Forward Error Correction (FEC) is used for protecting data against channel impairments. FEC is typically applied in transmission systems on physical layer or upper system layers such as link or application layer. FEC algorithms implemented in today’s transmission systems, such as RaptorQ or LDPC, are optimized for transmission of single layer data streams and do not exploit the knowledge of the dependency structure in layered media codecs. There are two ways for generation of FEC data for layered media streams with today’s FEC codes which are illustrated in Figure 2. One is to simply treat all media layers as one data block and generate FEC data from that FEC source block (left part of the figure). This approach preserves the optimized correction capabilities of the FEC code. However, doing so requires the reception of all media layers which makes reception of media layers separately in a multicast manner or prioritizing of more important media layers impossible. The second approach is to generate for each media layer a separate FEC code (right part of the figure). The latter approach does not have the same limitations as the first approach but is still not optimal. This becomes obvious in situations where the base layer cannot be corrected. In such situations, the enhancement layer becomes useless for the media decoding process due to its dependencies to the base layer and with it all the associated protection data, although it has been successfully received.
The idea of the Layer-Aware FEC (LA-FEC) is to produce FEC data of a certain media layer over the source data of the media layer itself and all media layers this layer depends on. Thereby, FEC data additionally protects data it depends on. The LA-FEC process is illustrated in Figure 3. Since the base layer (BL) does not have any dependencies to other media layers, its FEC generation remains untouched and is generated from the source data of BL only. The FEC data of the enhancement layer (EL) is produced over the source data of both media layers as illustrated by the extended LA-FEC source block. After transmission of both media layers, all FEC coded data can be used for a combined error correction as illustrated in the following example.
Layer-Aware FEC with a simple parity FEC code
To illustrate the principle of the LA-FEC approach, we apply a simple FEC algorithm which generates parity bits by XOR combinations of source symbols. Note that LA-FEC can be integrated in more efficient FEC algorithms as shown by the integration of LA-FEC in RaptorQ and LDPC below. Figure 4 compares the encoding and decoding process of Standard FEC on the left side and LA-FEC on the right side of the figure. The exemplary bit stream consists of two media layers, layer 0 and layer 1, where layer 1 depends on layer 0. Each layer consists of k=3 source bits and the simple FEC algorithm generates p=2 parity bits for each layer. As can be seen in the figure, LA-FEC integrates the source bits of layer 0 in the encoding process of the parity bits of layer 1. The resulting codewords have the same length of n=5 for standard FEC and LA-FEC and for both media layers. After FEC encoding, the codewords are transmitted over an erasure channel. In the outlined situation, the codeword of layer 0 is affected by three erasures which are highlighted by ‘E’, while the codeword of layer 1 remains error free. In case of standard FEC, there are not enough parity bits within layer 0 for successful FEC decoding. The source bits can therefore not be recovered. Although the layer 1 codeword is correctly received, it cannot be used due to the missing media coding dependencies on layer 0. Using LA-FEC, the parity bits of layer 1 can be used jointly with the parity bits of layer 0 for correcting layer 0. Since layer 1 is correctly received, there are in total four parity bits available for correction of the three source bits of layer 0. In the given example, both media layers can only be corrected with LA-FEC. It should be noted that if using the LA-FEC, the enhancement layer cannot be corrected independently of the base layer. Therefore, the improvement in base layer protection comes at the expense of reduced recovery probability of the enhancement layer. Nevertheless, in cases where the base layer is lost, the enhancement layer data cannot be used in the media decoding process anyway due to missing media dependencies within the media stream. Therefore, LA-FEC never performs worse than standard FEC in terms of media quality.
Prioritized Transmission of Layered Media with Layer-Aware FEC,
Mensch & Buch Verlag, Berlin, ISBN 978-3863873769, November 2013.
Cornelius Hellge, David Gomez-Barquero, Thomas Schierl, and Thomas Wiegand: Layer-Aware Forward Error Correction for Mobile Broadcast of Layered Media,Multimedia, IEEE Transactions on, Volume 13, Issue 3, pp. 551-562, 2011.
Integration in FEC algorithms
Raptor (RFC 5053):
Cornelius Hellge, Thomas Schierl, and Thomas Wiegand:
Mobile TV using scalable video voding and layer-aware forward error correction IEEE,
International Conference on Multimedia and Expo (ICME'08), Hanover, Germany, pp. 1177-1180, June 2008.
RaptorQ (RFC 6330):
ISO/IEC DIS 23008-10: Information technology -- High efficiency coding and media delivery in heterogeneous environments -- Part 10: MPEG Media Transport Forward Error Correction (FEC) codes.
César Díaz, Cornelius Hellge, Julián Cabrera, Fernando Jaureguizar, and Thomas Schierl:
Enhancement of Pro-MPEG COP3 Codes and Application to Layer-Aware FEC Protection of two-layered video transmission,
Proceedings of IEEE International Conference on Image Processing (ICIP'13), Melbourne, Australia, September 2013.
Jimin Xiao, Tillo, T., and Yao Zhao:
Real-Time Video Streaming Using Randomized Expanding Reed–Solomon Code,
Circuits and Systems for Video Technology, IEEE Transactions on , vol.23, no.11, pp.1825,1836, Nov. 2013.
Random Network Coding:
Vukobratovic, D., Khirallah, C., Stankovic, V., and Thompson, J.:
Random Network Coding for Multimedia Delivery over LTE-Advanced, Multimedia and Expo (ICME), 2012 IEEE International Conference on, pp. 200-205, 9-13 July 2012.
Huo, Yongkai, El-Hajjar, Mohammed, and Hanzo, Lajos:
Inter-Layer Turbo Coded Unequal Error Protection for Multi-Layer Video Transmission, Vehicular Technology Conference (VTC Fall), 2013 IEEE 78th, pp.1-5, 2-5 Sept. 2013.
Standardization activities related to LA-FEC
ISO/IEC FDIS 23008-1: Information technology -- High efficiency coding and media delivery in heterogeneous environments -- Part 1: MPEG Media Transport
ISO/IEC DIS 23008-10: Information technology -- High efficiency coding and media delivery in heterogeneous environments -- Part 10: MPEG Media Transport Forward Error Correction (FEC) codes