Joint Source/Channel Coding

Goal: Improved video transmission for very low delay applications (e.g. video telephony) in error prone environments.

A video signal that is compressed by a hybrid video coder like H.264/MPEG4-AVC is extremely vulnerable to transmission errors. Transmission errors can be reduced by appropriate channel coding techniques. For channels without memory, such as the AWGN channel, channel coding techniques provide very significant reductions of transmission errors at a comparably moderate bit rate overhead. For the mobile fading channel or the Internet, however, the effective use of forward error correction is limited when assuming a small end-to-end delay. Here the use of error resilience techniques in the source codec becomes important.

In Inter mode, i.e., when motion-compensated prediction is utilized, the loss of information in one frame has a considerable impact on the quality of the following frames. As a result, spatio-temporal error propagation is a typical transmission error effect for predictive coding. Because errors remain visible for a longer period of time, the resulting artifacts are particularly annoying to end users. To some extent the impairment caused by transmission errors decays over time due to leakage in the prediction loop. However, the leakage in standardized video decoders like H.264/MPEG4-AVC is not very strong, and quick recovery can only be achieved when image regions are encoded in Intra mode, i.e., without reference to a previous frame. The Intra mode, however, is not selected very frequently during normal encoding and completely Intra coded frames are not usually inserted in real-time encoded video as is done for storage or broadcast applications. Instead, only single macroblocks, i.e., parts of the picture, are encoded in Intra mode for regions that cannot be predicted efficiently.

The work on this subject started with motion estimation and mode decision with multiple reference pictures in error-prone environments [3], [12], [16]. The idea of estimating the distortion introduced by packet losses at the encoder and utilizing this estimate in the encoder control has been driven further and been proposed and accepted by VCEG during the development of H.26L. This estimate is obtained by running multiple versions of the decoder at the encoder side with each version simulating a realization of the random decoding process [1], [5]-[7], [13]-[15].

Impact of Transmission Errors

  • If just one part or an entire frame is missing => concealment at decoder side => reference pictures at coder and decoder differ => motion compensation => spatio-temporal error propagation
  • Spatio-temporally propagated errors decay slowly => mitigate error propagation
  • Mitigating error propagation
    • Use of multiple reference pictures
    • INTRA picture and INTRA macroblock refresh

Assignment of Intra Macroblock Coding

  • Intra coding provides lower coding efficiency than Inter coding => Trade-off between error resilience and coding efficiency
  • Random transmission errors cause that decoding result becomes random variable
  • Model decoding random variable using N sample functions
  • Optimize encoding operation for average decoding result for a given packet-loss rate p

Modified Coder Control

  • Given the N decoded versions with random packet losses of probabilityp at the encoder
  • Lagrangian Mode Decision

$$D_{2}(M|Q)+\lambda_{M}\cdot R(M|Q)$$

  • With modified distortion measure

Approach adaptively increases costs for Inter coding and therefore increases Intra coding rate.


Publications in Journals


  • Thomas Stockhammer, Miska M. Hannuksela, and Thomas Wiegand:
    H.264/AVC in Wireless Environments
    IEEE Transactions on Circuits and Systems for Video Technology
    , vol. 13, no. 7, pp. 657-673, July 2003.

Publications in Conference Proceedings


  • Thomas Stockhammer, Tobias Oelbaum, Florian Obermeier, and Thomas Wiegand:
    Video Coding and Transport Layer Techniques for H.264-based Transmission over Packet-Lossy Networks,
    IEEE International Conference on Image Processing (ICIP'03), Barcelona, Spain, September 2003.


  • Thomas Stockhammer and Thomas Wiegand:
    Optimized Transmission of H.26L Video over Packet-Lossy Networks,
    IEEE International Conference on Image Processing (ICIP'02), Rochester, NY, USA, September 2002.


  • Gary J. Sullivan, Thomas Wiegand, and Thomas Stockhammer:
    Using the Draft H.26L Video Coding Standard for Mobile Applications,
    IEEE International Conference on Image Processing (ICIP'01), Thessaloniki, Greece, September 2001, Invited Paper.

Contributions to Standardization


  • Thomas Stockhammer, Tobias Oelbaum, Thomas Wiegand, and Detlev Marpe:
    Simulation Results for Common Conditions for H.323/Internet Case,
    VCEG meeting - ITU-T SG16/Q.6, Santa Barbara, USA, ITU-T/SG16/VCEG-N50, September 2001.
  • Thomas Stockhammer, Tobias Oelbaum, Thomas Wiegand, and Detlev Marpe:
    Simulation Results for Common Conditions for Video Performance Evaluation in H.324/M Error-Prone Systems,
    VCEG meeting - ITU-T SG16/Q.6, Santa Barbara, USA, ITU-T/SG16/VCEG-N39, September 2001.
  • Thomas Stockhammer, Günther Liebl, Tobias Oelbaum, Thomas Wiegand, and Detlev Marpe:
    Simulation Results for Common Test Conditions for RTP/IP over 3GPP/3GPP2,
    VCEG meeting - ITU-T SG16/Q.6, Santa Barbara, USA, ITU-T/SG16/VCEG-N38, September 2001.


  • Niko Färber, Thomas Wiegand, and Bernd Girod:
    Error Correcting RVLC,
    VCEG meeting - ITU-T SG16/Q.6, Whistler, Canada, ITU-T/SG16/Q15-E-32, July 1998.