Modern layered or scalable video coding technologies generate a video bit stream with various inter layer dependencies due to references between the layers. The layer-aware FEC (L-FEC) generates repair symbols so that protection of less important (higher) dependency layers can be used with protection of higher important (lower) layers for combined error correction. Protection gains for lower layers can be achieved without increasing the total FEC code rate.
Media transmission is affected by information loss due to transmission errors on channels without Quality of Service (QoS). Due to the multifaceted dependency structure of modern video codecs like H.264/AVC or especially the scalable video coding extension of H.264/AVC (SVC), the effect of losses on the decoded video quality mainly depends on which parts of the video bit stream are affected. Forward error correction (FEC) technologies are typically used to reduce losses on multicast or broadcast channels. Many protection schemes have been proposed addressing this issue like unequal error protection (UEP) or priority encoding transmission (PET). Both examples give more protection to more important (lower) layers. Using a dependency-aware approach shows that generating protection for less important (higher) layers over all referenced layers can show an additional gain in protection for the referenced and more important layers. The presented layer-aware FEC (L-FEC) approach additionally considers the layer structure of scalable video coding and the existing dependencies of multiple dimensions. This can be very beneficial in transmission schemes where multiple layers of a scalable video stream are transmitted separately and the FEC protection must be independent of different layers as e.g. in a receiver driven layered multicast scenario.
Dependency structures in modern video codecs
In modern video codecs, various dependency structures can be used. One important dependency structure is introduced by motion compensation, where a picture reference e.g. from the past is used to predict another picture. If a reference picture or a part of the reference picture is lost, the pictures using the reference are affected, too. Therefore, the first picture in a sequence of pictures that reference pictures from the past is most important. A loss at this point typically affects all succeeding pictures in some way. Hence, it is desirable to protect as many pictures as possible in such a dependency chain.
Another set of dependency structures is introduced by the layered video coding of SVC, where a base layer is referenced by one or more enhancement layers. Such an enhancement layer can be further referenced by other enhancement layers and so on. Figure 1 depicts a simple dependency structure in a SVC media stream with successive dependency layers. Let l be the identifier of a layer out of 0 to L - 1, where l = 0 is the base layer, l = 1 identifies the first enhancement layer referencing the base layer. In SVC, the loss of any layer x typically leads to corrupt decoding results for all layers l > x referencing x. SVC allows scalability dimensions as spatial (resolution), temporal (frame rate) or fidelity (SNR quality) scalability. A combination of those results into multidimensional dependency structures within the media bit stream (see Figure 2). Other scalable dimensions like bit-depth scaling, or multi-view coding can add further dependencies.
Forward Error Correction
Today’s FEC protection schemes do not take potentially multiple dependencies in a (scalable) video bit stream into account. In such schemes, typically for a number of k source symbols, a number of p = n - k redundancy symbols are generated where n is the code word length and p the number of redundancy or parity symbols. Let us assume the usage of an ideal FEC code so that each number of erasures of symbols smaller or equal to p can be overcome. Even, when using more FEC redundancy (stronger protection) for the more important lower layers like applied in UEP, there is no guarantee that in any error condition the reconstruction of relatively stronger protected layers is also of higher probability than reconstruction of less strong protected layers. If a lower layer is lost, higher enhancement layers cannot be decoded due to missing references as depicted in Figure 1. In this example layer 1 at time t is lost due to transmission error, so that layers with l > 1 can not be decoded. Unlike using the layer-aware approach, the successfully received FEC redundancy of the enhancement layers is useless.
Layer-Aware Forward Error Correction
The idea of the layer-aware FEC (L-FEC) is to generate redundancy over multiple layers in a media stream. Resulting redundancy symbols in higher layers also protect lower layers and can be used for a combined decoding. By following existing dependency structures in the media stream there is no additional disadvantage despite a slightly increased computational complexity. All redundancy symbols in the same dependency chain can be used together for error correction. A dependency chain contains all dependent layers for decoding a particular layer in the order of importance.
Figure 3 depicts the L-FEC generation for one dimensional dependency. After partitioning of the media bit stream into L dependency layers, redundancy symbols (FEC 0) of the lowest layer l = 0 are typically generated given by the FEC coding technique T. Considering the dependency structure, redundancy symbols of the enhancement layer l = x are calculated incorporating source symbols of all lower layers l <= x. FEC 1 (layer l = 1) symbols are generated over source symbols of layer l = 0 and layer l = 1. Furthermore, FEC 2 (layer l = 2) symbols are generated over source symbols of layer l = 0, layer l = 1 and layer l = 2 and so on up to FEC L-1 which is generated over source symbols of all lower layers l = 0 to L - 1. Note, that the number of redundancy symbols remains constant and redundancy symbols of different layers but the same dependency path can be jointly used. The L-FEC generation over multiple dependencies is shown in Figure 4 for three dependency dimensions as present in SVC.
The L-FEC approach can also be extended to dependency structures in multiple dimensions of e.g. SVC. Figure 4 schematizes the generation of redundancy symbols along multiple dependency paths. There are three dependency dimensions, D_1, D_2, D_3, according to the temporal, spatial and fidelity dimensions in SVC. In each dimension, several layers l_Di are present where the arrows represent the dependencies. In this example each layer depends on all lower layers of the same dimension and partially on the layers of other dimensions. All redundancy symbols FEC l_D1 l_D2 l_D3 are generated over all depending layers. The redundancy symbols within a particular dependency path can be jointly used for correcting all source symbols of that path. The base layer, e.g., is included in all FEC symbols. Hence, there are multiple paths where redundancy symbols can be jointly used for correcting errors in the base layer. In the one dimensional case, allows for the joint use of symbols within one dimension. I.e., FEC 200, FEC 100, and FEC 000 can be jointly used for decoding source symbols in dimension D_1 up to the maximum layer used for FEC generation. In the multi dimensional case, FEC 111, FEC 110, FEC 010, FEC 100, and FEC 000 can be jointly used for correcting errors in all dimensions up to the maximum layer used for L-FEC generation. In the concrete case of using the proposed L-FEC approach with the Raptor code error correction is successful, if there are enough linear independent equations for each layer for solving the equation systems generated by received Raptor symbols.
The coding process is exemplary demonstrated for two dependency layers in Figure 5 with a very simple FEC code, i.e. parity bits are generated by simple XOR combinations of the source bits. Note that this example can be extended to multiple layers of multiple dimensions of dependency.
The source bits I_0 of the base layer, layer 0, are encoded using the standard coding process. I.e. as depicted in (1) the parity bits p_00, p_01 are a random XOR combination of the source bits i_00, i_01, i_02. The parity bits P_0 are concatenated with the source bits I_0 to the resulting code word C_0 for layer 0.
For the upper layer layer 1 the parity symbols p_10, p_11 are calculated following the L-FEC approach (2). I.e. the random XOR combinations are not only computed over the source bits I_1 of the layer 1 but also include the source bits I_0 of the lower dependency layer layer 0. The resulting parity bits are concatenated with the source bits of layer 1 to the code word C_1.
In (3) the code words C_0 and C_1 are transmitted over an error prone channel, i.e. the received code word R_0 is affected by three losses marked by a '?' whereas the received code word of layer 1 R_1 remains error free.
In (4) the decoding process for layer 0 fails due to the lost bits r_00,r_01 and r_02 in the received code word R_0. Due to the dependencies within the media stream, the upper layer 1 could be reconstructed but cannot be decoded without layer 0. I.e. with a standard FEC there cannot be decoded any layer at all. The L-FEC approach allows the usage of redundancy symbols of the upper layers for lower layers. Therefore in (5) with the additional connections to layer 0 both source bits I_0 and I_1 can be reconstructed with a combined decoding of the received bits R_0 and R_1.
As shown in the example in Figure 5 the layer-aware approach allows additional protection for lower layers without an increase in bit-rate. The L-FEC approach introduces additional dependencies for the upper layers to lower layers. But following the existing dependencies in the media stream this disadvantage is negligible.
A possible application for the L-FEC approach is a wireless receiver driven layered multicast (RLM) scenario depicted in Figure 6. In the presented cut-out there are two clients connected to a radio tower. Client 1 has a maximum throughput of 400 kbps and client 2 can receive data up to 750 kbps. The media stream is encoded using SVC with a H.264/AVC base layer (layer 0) and one SNR enhancement layer (layer 1), which depends on layer 0. Due to packet losses in the channel each layer is protected by additional redundancy packets FEC 1 and L-FEC 2. FEC 1 is based on a standard encoding process and protects only the base layer. L-FEC 2 uses the L-FEC approach and is generated across both layers. Redundancy of layer 1 protects both layers and can be used together with FEC 1 for combined decoding.
Due to the available throughput client 1 would only request the base layer which is protected by a standard FEC FEC 1. Client 2 can receive higher bandwidth and would request both layers. For client 2 the base-layer is protected by FEC 1 plus the redundancy of layer 1 L-FEC 2 which also protects layer 0 due to the L-FEC approach. The redundancy of the layers can be combined, i.e. client 2 has an increase in protection for the base-layer without any cost in terms of additional bit-rate.
The protection gain for the base layer using the L-FEC approach provides a more robust transmission for client 2 without the need of increasing the total bit-rate.
Publications in Conference Proceedings
- Cornelius Hellge, Thomas Schierl, and Thomas Wiegand:
MULTIDIMENSIONAL LAYERED FORWARD ERROR CORRECTION USING RATELESS CODES,
IEEE International Conference on Communications (ICC'08), Beijing, China, May 2008.
- Cornelius Hellge, Thomas Schierl, and Thomas Wiegand:
MOBILE TV USING SCALABLE VIDEO CODING AND LAYER-AWARE FORWARD ERROR CORRECTION,
IEEE International Conference on Multimedia and Expo (ICME'08), Hannover, Germany, June 2008.