Prediction of Transmission Distortion for Wireless Video Communication: Analysis

Transcription

1 Prediction of Transmission Distortion for Wireless Video Commnication: Analysis Zhifeng Chen and Dapeng W Department of Electrical and Compter Engineering, University of Florida, Gainesville, Florida 326 Abstract Transmitting video over wireless is a challenging problem since video may be seriosly distorted de to pacet errors cased by wireless channels. The capability of predicting transmission distortion (i.e., video distortion cased by pacet errors) can assist in designing video encoding and transmission schemes that achieve maximm video qality or minimm endto-end video distortion. This paper is aimed at deriving formlae for predicting transmission distortion. The contribtion of this paper is two-folded. First, we identify the governing law that describes how the transmission distortion process evolves over time, and analytically derive the transmission distortion formla as a closed-form fnction of video frame statistics, channel error statistics, and system parameters. Second, we identify, for the first time, two important properties of transmission distortion. The first property is that the clipping noise, prodced by non-linear clipping, cases decay of propagated error. The second property is that the correlation between motion vector concealment error and propagated error is negative, and has dominant impact on transmission distortion, compared to other correlations. De to these two properties and elegant error/distortion decomposition, or formla provides not only more accrate prediction bt also lower complexity than the existing methods. Index Terms Wireless video, transmission distortion, clipping noise, slice data partitioning (SDP), neqal error protection (UEP), time-varying channel. I. INTRODUCTION Both mltimedia technology and mobile commnications have experienced massive growth and commercial sccess in recent years. As the two technologies converge, wireless video, sch as video phone and mobile TV in 3G/4G systems, is expected to achieve nprecedented growth and worldwide sccess. However, different from the traditional video coding system, transmitting video over wireless with good qality or low end-to-end distortion is particlarly challenging since the received video is sbject to not only qantization error bt also transmission error. In a wireless video commnication system, end-to-end distortion consists of two parts: qantization distortion and transmission distortion. Qantization distortion is cased by qantization errors dring the encoding process, and has been extensively stdied in rate distortion theory [], [2]. Transmission distortion is cased by pacet errors dring the transmission of a video seqence, and it is the major part Please direct all correspondence to Prof. Dapeng W, University of Florida, Dept. of Electrical & Compter Engineering, P.O.Box 63, Gainesville, FL 326, USA. Tel. (352) Fax (352) w@ece.fl.ed. Homepage: This wor was spported in part by the US National Science Fondation nder grant ECCS Copyright (c) 2 IEEE. Personal se of this material is permitted. However, permission to se this material for any other prposes mst be obtained from the IEEE by sending a reqest to pbs-permissions@ieee.org. of the end-to-end distortion in delay-sensitive wireless video commnication nder high pacet error probability (PEP), e.g., in a wireless fading channel. The capability of predicting transmission distortion at the transmitter can assist in designing video encoding and transmission schemes that achieve maximm video qality nder resorce constraints. Specifically, transmission distortion prediction can be sed in the following three applications in video encoding and transmission: ) mode decision, which is to find the best intra/inter-prediction mode for encoding a macrobloc (MB) with the minimm rate-distortion (R-D) cost given the instantaneos PEP, 2) cross-layer encoding rate control, which is to control the instantaneosly encoded bit rate for a realtime encoder to minimize the frame-level end-to-end distortion given the instantaneos PEP, e.g., in video conferencing, 3) pacet schedling, which chooses a sbset of pacets of the pre-coded video to transmit and intentionally discards the remaining pacets to minimize the grop of pictre (GOP)- level end-to-end distortion given the average PEP and average brst length, e.g., in streaming pre-coded video over networs. All the three applications reqire a formla for predicting how transmission distortion is affected by their respective control policy, in order to choose the optimal mode or encoding rate or transmission schedle. However, predicting transmission distortion poses a great challenge de to the spatio-temporal correlation inside the inpt video seqence, the nonlinearity of both the encoder and the decoder, and varying PEP in time-varying channels. In a typical video codec, the temporal correlation among consective frames and the spatial correlation among the adjacent pixels of one frame are exploited to improve the coding efficiency. Nevertheless, sch a coding scheme brings mch difficlty in predicting transmission distortion becase a pacet error will degrade not only the video qality of the crrent frame bt also the following frames de to error propagation. In addition, as we will see in Section III, the nonlinearity of both the encoder and the decoder maes the instantaneos transmission distortion not eqal to the sm of distortions cased by individal error events. Frthermore, in a wireless fading channel, the PEP is time-varying, which maes the error process a non-stationary random process and hence, as a fnction of the error process, the distortion process is also a non-stationary random process. According to the aforementioned three applications, the Delay-sensitive wireless video commnication sally does not allow retransmission to correct pacet errors since retransmission may case long delay.

2 2 existing algorithms for estimating transmission distortion can be categorized into the following three classes: ) pixel-level or bloc-level algorithms (applied to mode decision), e.g., Recrsive Optimal Per-pixel Estimate (ROPE) algorithm [3] and Law of Large Nmber (LLN) algorithm [4], [5]; 2) frame-level or pacet-level or slice-level algorithms (applied to cross-layer encoding rate control) [6], [7], [8], [9], []; 3) GOP-level or seqence-level algorithms (applied to pacet schedling) [], [2], [3], [4], [5]. Althogh the existing distortion estimation algorithms wor at different levels, they share some common properties, which come from the inherent characteristics of wireless video commnication system, that is, spatio-temporal correlation, nonlinear codec and timevarying channel. However, none of the existing wors analyzed the effect of non-linear clipping noise on the transmission distortion, and therefore cannot provide accrate distortion estimation. In this paper, we derive the transmission distortion formlae for wireless video commnication systems. With consideration of spatio-temporal correlation, nonlinear codec and timevarying channel, or distortion prediction formlae improve the accracy of distortion estimation from existing wors. Besides that, or formlae spport, for the first time, the following capabilities: ) prediction at different levels (e.g., pixel/frame/gop level), 2) prediction for mlti-reference motion compensation, 3) prediction nder slice data partitioning (SDP) [6], 4) prediction nder arbitrary slice-level pacetization with flexible macrobloc ordering (FMO) mechanism [7], [8], 5) prediction nder time-varying channels, 6) one nified formla for both I-MB and P-MB, and 7) prediction for both low motion and high motion video seqences. In addition, this paper also identifies two important properties of transmission distortion for the first time: ) clipping noise, prodced by non-linear clipping, cases decay of propagated error; 2) the correlation between motion vector concealment error and propagated error is negative, and has dominant impact on transmission distortion, among all the correlations between any two of the for components in transmission error. De to the page limit, we move most of the experimental reslts to or seqel paper [9], which ) verify the accracy of the formlae derived in this paper and compare that to existing models, 2) discss the algorithms designed based on the formlae, 3) apply or algorithms in practical video codec design, and 4) compare the R-D performance between or algorithms and existing estimation algorithms. The rest of the paper is organized as follows. Section II presents the preliminaries of or system model nder stdy to facilitate the derivations in the later sections, and illstrates the limitations of existing transmission distortion models. In Section III, we derive the transmission distortion formla as a fnction of frame statistics, channel condition, and system parameters. Section IV concldes the paper. II. SYSTEM DESCRIPTION A. Strctre of a Wireless Video Commnication System Fig. shows the strctre of a typical wireless video commnication system. It consists of an encoder, two channels Inpt Video captre f - Motion compensation Motion estimation Encoder e ˆ + mv f mv T/Q fˆ Q - /T - eˆ + Clipping fˆ Memory Channel Residal Channel MV Channel S(r) S(m) Decoder Q - /T - Residal Error Concealment ~ f + ~ mv Motion compensation m ( v MV Error concealment eˆ ( e ~ f + e~ Clipping Memory Otpt ~ f Video display Fig.. System strctre, where T, Q, Q, and T denote transform, qantization, inverse qantization, and inverse transform, respectively. and a decoder where residal channel and motion vector (MV) channel may be either the same channel or different channels. If residal pacets or MV pacets are erroneos, the error concealment modle will be activated. In typical video standard sch as H.263/264 and MPEG-2/4, the fnctional blocs of an encoder can be divided into two classes: ) basic parts, sch as predictive coding, transform, qantization, entropy coding, motion compensation, and clipping; and 2) performance-enhancing parts, sch as interpolation filtering, deblocing filtering, B-frame, mlti-reference prediction, etc. Althogh the p-to-date video standard, e.g. the coming new video standard HEVC, incldes more and more performanceenhancing parts, the basic parts do not change. In this paper, we analyze the transmission distortion for the strctre with the basic parts in Fig.. Note that in this system, both residal channel and MV channel are application-layer channels; specifically, both channels consist of entropy coding and entropy decoding, networing layers 2, and physical layer (inclding channel encoding, modlation, wireless fading channel, demodlation, channel decoding). Althogh the residal channel and MV channel sally share the same physical-layer channel, the two application-layer channels may have different parameter settings (e.g., different channel code-rate) for different SDP pacets nder neqal error protection (UEP) consideration. Table I lists notations sed in this paper. All vectors are in bold font. Note that the encoder needs to reconstrct the compressed video for predictive coding; hence the encoder and the decoder have a similar strctre for pixel vale reconstrction. To distingish the variables in the reconstrction modle of the encoder from those in the reconstrction modle of the decoder, we add ˆ on top of the variables at the encoder and add on top of the variables at the decoder. B. Clipping Noise In this sbsection, we examine the effect of clipping noise on the reconstrcted pixel vale along each pixel trajectory over time (frames). All pixel positions in a video seqence form a three-dimensional spatio-temporal domain, i.e., two dimensions in spatial domain and one dimension in temporal 2 Here, networing layers can inclde any layers other than physical layer.

3 3 TABLE I SUMMARY OF NOTATIONS : A pixel with position in the -th frame f : Vale of the pixel e : Residal of the pixel mv : MV of the pixel : Clipping noise of the pixel ε : Residal concealment error of the pixel ξ : MV concealment error of the pixel ζ : Transmission reconstrcted error of the pixel S : Error state of the pixel P : Error probability of the pixel D : Transmission distortion of the pixel D : Transmission distortion of the -th frame V : Set of all the pixels in the -th frame V : Nmber of elements in set V (cardinality of V ) α : Propagation factor of the -th frame β : Percentage of I-MBs in the -th frame λ : Correlation ratio of the -th frame domain. Each pixel can be niqely represented by in this three-dimensional time-space, where means the -th frame in temporal domain and is a two-dimensional vector in spatial domain, i.e. position in the -th frame. The philosophy behind inter-prediction of a video seqence is to represent the video seqence by virtal motion of each pixel, i.e., each pixel recrsively moves from position v in the frame, i.e. v, to position. The difference between these two positions is a two-dimensional vector called MV of pixel, i.e., mv = v. The difference between the pixel vales of these two positions is called residal of pixel, that is, e = f Recrsively, each inter-predicted +mv. pixel in the -th frame has one and only one reference pixel trajectory bacward towards the latest I-bloc. 3 At the encoder, after transform, qantization, inverse qantization, and inverse transform for the residal, the reconstrcted pixel vale for may be ot-of-range and shold be clipped as = Γ( + ê +mv ), () where Γ( ) fnction is a clipping fnction defined by γ l, x < γ l Γ(x) = x, γ l x γ h γ h, x > γ h, where γ l and γ h are ser-specified low threshold and high threshold, respectively. Usally, γ l = and γ h = 255. The residal and MV at the decoder may be different from their conterparts at the encoder becase of channel impairments. Denote mv and ẽ the MV and residal at the decoder, respectively. Then, the reference pixel position for at the decoder is ṽ = + mv, and the reconstrcted pixel vale for at the decoder is (2) f = Γ( f + ẽ + mv ). (3) In error-free channels, the reconstrcted pixel vale at the receiver is exactly the same as the reconstrcted pixel vale at the transmitter, becase there is no transmission error and 3 We will also discss intra-predicted pixels in Section III. hence no transmission distortion. However, in error-prone channels, we now from (3) that f is a fnction of three factors: the received residal ẽ, the received MV mv, and the propagated error f The received residal ẽ + mv. depends on three factors, namely, ) the transmitted residal ê, 2) the residal pacet error state, which depends on instantaneos residal channel condition, and 3) the residal error concealment algorithm if the received residal pacet is erroneos. Similarly, the received MV mv depends on ) the transmitted mv, 2) the MV pacet error state, which depends on instantaneos MV channel condition, and 3) the MV error concealment algorithm if the received MV pacet is erroneos. The propagated error f incldes the error + mv propagated from the reference frames, and therefore depends on all samples in the previos frames indexed by i, where i < and their reception error states as well as error concealment algorithms. In this paper, we consider the temporal error concealment [2], [2] in deriving the transmission distortion formlae. The non-linear clipping fnction within the pixel trajectory maes the distortion estimation more challenging. However, it is interesting to observe that clipping actally redces transmission distortion. In Section III, we will qantify the effect of clipping on transmission distortion. C. Definition of Transmission Distortion In a video seqence, all pixel positions in the -th frame form a two-dimensional vector set V, and we denote the nmber of elements in set V by V. So, for any pixel at position in the -th frame, i.e., V, its reference pixel position is chosen from set V for single-reference motion compensation. Given the joint probability mass fnction (PMF) of and f, we define the pixel-level transmission distortion (PTD) for pixel by D E[( f ) 2 ], (4) where E[ ] represents expectation and the randomness comes from both random video inpt and random channel error state. Then, we define the frame-level transmission distortion (FTD) for the -th frame by D E[ V ( f ) 2 ]. (5) V It is easy to prove that the relationship between FTD and PTD is characterized by D = V D. (6) V In fact, (6) is a general form for distortions of all levels. If V =, (6) redces to (4). For slice/pacet-level distortion, V is the set of the pixels contained in a slice/pacet. For GOP-level distortion, V cold be replaced by the set of the pixels contained in a GOP. In this paper, we only show how to derive formlae for PTD and FTD. Or methodology is also applicable to deriving formlae for slice/pacet/gop-level distortion by sing appropriate V.

4 4 D. Limitations of the Existing Transmission Distortion Models We define the clipping noise for pixel at the encoder as ˆ ( + ê +mv ) Γ( + ê +mv ), (7) and the clipping noise for pixel at the decoder as ( f + ẽ + mv ) Γ( f + ẽ + mv ). (8) Using (), Eq. (7) becomes = and sing (3), Eq. (8) becomes f = +mv + ê ˆ, (9) f + ẽ + mv, () where ˆ only depends on the video content and encoder strctre, e.g., motion estimation, qantization, mode decision and clipping fnction; and depends on not only the video content and encoder strctre, bt also channel conditions and decoder strctre, e.g., error concealment and clipping fnction. In most existing wors [3], [7], [9], [], [5], both ˆ and are neglected, i.e., these wors assme = +ê +mv and f = +ẽ. However, this assmption is only valid f + mv for stored video or error-free commnication, where = ˆ, since ˆ = with very high probability. For error-prone commnication, decoder clipping noise has a significant impact on transmission distortion and hence shold not be neglected. Withot taing into consideration, the estimated distortion can be mch larger than tre distortion [22]. III. TRANSMISSION DISTORTION FORMULAE In this section, we derive formlae for PTD and FTD. The section is organized as follows: Section III-A presents an overview of or approach to analyzing PTD and FTD. Then, we elaborate on the derivation details in Section III-B throgh Section III-E. Specifically, Section III-B qantifies the effect of residal concealment error (RCE) on transmission distortion; Section III-C qantifies the effect of motion vector concealment error (MVCE) on transmission distortion; Section III-D qantifies the effect of propagated error and clipping noise on transmission distortion; Section III-E qantifies the effect of correlations (between any two of the error sorces) on transmission distortion. Finally, Section III-F smmarizes the ey reslts of this paper, i.e., the formlae for PTD and FTD. A. Overview of the Approach to Analyzing PTD and FTD To analyze PTD and FTD, we tae a divide-and-conqer approach. We first divide transmission reconstrcted error into for components: three random errors (RCE, MVCE and propagated error) de to their different physical cases, and clipping noise, which is a non-linear fnction of these three random errors. This error decomposition allows s to frther decompose transmission distortion into for terms, i.e., distortion cased by ) RCE, 2) MVCE, 3) propagated error pls clipping noise, and 4) correlations between any two of the error TABLE II DEFINITIONS RCE : residal concealment error MVCE: motion vector concealment error PTD : pixel-level transmission distortion FTD : frame-level transmission distortion XEP : pixel error probability PEP : pacet error probability FMO : flexible macrobloc ordering UEP : neqal error protection SDP : slice data partitioning PMF : probability mass fnction sorces, respectively. This distortion decomposition facilitates the derivation of a simple and accrate closed-form formla for each of the for distortion terms. Next, we elaborate on error decomposition and distortion decomposition. Define transmission reconstrcted error for pixel by ζ f. From (9) and (), we obtain ζ = (ê + = (ê ẽ ) + ( ˆ +mv ) (ẽ + f + mv ) +mv ) + mv + ( f ) ( ˆ + mv + mv ). () Define RCE ε by ε ê ẽ, and define MVCE ξ by ξ. Note that f = +mv + mv + mv + mv ζ, which is the transmission reconstrcted error of the + mv concealed reference pixel in the reference frame; we call propagated error. As mentioned in Section II-D, we ζ + mv assme ˆ =. Therefore, () becomes ζ = ε + ξ + ζ + mv +. (2) (2) is or proposed error decomposition. In Table II, we list the abbreviations that will be sed freqently in the following sections. Combining (4) and (2), we have D = E[( ε + ξ + ζ + + mv ) 2 ] = E[( ε ) 2 ] + E[( ξ ) 2 ] + E[( ζ + + mv ) 2 ] + 2E[ ε ξ ] + 2E[ ε ( ζ + + mv )] + 2E[ ξ ( ζ + + mv )]. (3) Denote D(r) E[( ε ) 2 ], D(m) E[( ξ ) 2 ], D(P ) + ) 2 ] and D(c) 2E[ ε ξ ] + 2E[ ε E[( ζ + mv ( ζ + mv + )]+2E[ ξ ( ζ + mv + )]. Then, (3) becomes D = D (r) + D (m) + D (P ) + D (c). (4) (4) is or proposed distortion decomposition for PTD. The reason why we combine propagated error and clipping noise into one term (called clipped propagated error) is becase clipping noise is mainly cased by propagated error and sch decomposition will simplify the formlae. There are three major reasons for or decompositions in (2) and (4). First, if we directly sbstitte the terms in (4) by (9) and (), it will prodce 5 second moments and cross-correlation terms (assming ˆ = ); since there

5 5 are 8 possible error events de to three individal random errors, there are a total of 8 (5 + ) = 2 terms for PTD, maing the analysis highly complicated. In contrast, or decompositions in (2) and (4) significantly simplify the analysis. Second, each term in (2) and (4) has a clear physical meaning, which lessens the reqirement for joint PMF of and f and leads to accrate estimation algorithms with low complexity. Third, sch decompositions allow or formlae to be easily extended for spporting advanced video codec with more performance-enhancing parts, e.g., mlti-reference prediction [22] and interpolation filtering in fractional-pel motion estimation [23]. To derive the formla for FTD, from (6) and (4), we obtain where D = D (r) + D (m) + D (P ) + D (c), (5) D (r) = D (m) = D (P ) = D (c) = V D (r), (6) V D (m), (7) V D (P ), (8) V D (c). (9) (5) is or proposed distortion decomposition for FTD. Usally, the cardinality, i.e. the nmber of elements, of set V in a video seqence is the same for all frames. That is, V = = V = for all 4. Hence, we remove the frame index and denote V for all by. Note that in a video codec, e.g. H.264/AVC [6], a reference pixel may be in a position ot of pictre bondary; however, the cardinality of set consisting of reference pixels, althogh larger than the cardinality of the inpt pixel set, is still the same for all frames. B. Analysis of Distortion Cased by RCE In this sbsection, we first derive the pixel-level residal cased distortion D (r). Then, we derive the frame-level residal cased distortion D (r). ) Pixel-level Distortion Cased by RCE: We denote S as the state indicator of whether there is transmission error for pixel after channel decoding. Note that as mentioned in Section II-A, both the residal channel and the MV channel contain channel decoding; hence in this paper, the transmission error in the residal channel or the MV channel is meant to be an ncorrectable error after channel decoding. To distingish the residal error state and the MV error state, here we se S (r) to denote the residal error state for pixel. That is, S (r) = if ê is received with error, and S (r) = if ê is received withot error. At the receiver, if there is no residal transmission error for pixel, ẽ is eqal to ê. 4 Note that althogh they have the same cardinality, different sets are very different, i.e. V V. However, if the residal pacets are received with error, we need to conceal the residal error at the receiver. Denote ě the concealed residal when S(r) =, and we have, { ẽ ě =, S(r) = ê, S(r) (2) =. Note that ě depends on ê and the residal concealment method, bt does not depend on the channel condition. From the definition of ε and (2), we have ε = (ê ě ) S (r) + (ê ê ) ( S (r)) = (ê ě ) S (r). (2) ê depends on the inpt video seqence and the encoder strctre, while S (r) depends on the random mltiplicative and additive noises in the wireless channel. Under or framewor shown in Fig., the inpt video seqence and the encoder strctre are independent of commnication system parameters. Therefore, we assme ê and S (r) are independent as the following assmption. Assmption : S (r) is independent of ê. Denote ε ê ě ; we have ε = ε S (r). Denote P (r) as the residal pixel error probability (XEP) for pixel, that is, P (r) P {S (r) = } 5. Then, given P (r), from (2) and Assmption, we have D (r) = E[( ε ) 2 ] = E[(ε ) 2 ] E[(S (r)) 2 ] = E[(ε ) 2 ] ( P (r)) = E[(ε ) 2 ] P (r). (22) Hence, or formla for the pixel-level residal cased distortion is D (r) = E[(ε ) 2 ] P (r). (23) Note that we may also generalize (23) for I-MB. For pixels in I-MB, if the pacet containing those pixels has error, ě is still available since all the erroneos pixels will be concealed in the same way. However, since there is no ê available, in order to se (23) to predict the transmission distortion, we may need to find the best reference, in terms of R-D cost, for the reconstrcted I-MB by doing a virtal motion estimation and then calclate ê for (23). The estimated mv can be sed to predict D (m) for I-MB in the next sbsection. An alternative method to calclate ê for I-MB is to se the same position of previos frame as reference, i.e. assming mv =. Note that if the pacet containing those pixels in I-MB is correctly received, D (r) =. 2) Frame-level Distortion Cased by RCE: To derive the frame-level residal cased distortion, the encoder needs to now the second moment of RCE for each pixel in that frame. In most, if not all, existing distortion models [3], [7], [9], [], [5], the residal error concealment method is to let ě = for all erroneos pixels. However, as long as ê and ě satisfy some properties, we can derive a formla for more general residal error concealment methods instead of assming ě =. We mae the following assmption for ê and ě. 5 P (r) depends on the commnication system parameters sch as delay bond, channel coding rate, transmission power, channel gain of the wireless channel.

6 6 Assmption 2: The residal ê is stationary with respect to 2D variable in the same frame. In addition, ě only depends on {ê v : v N } where N is a fixed neighborhood of. In other words, Assmption 2 assmes that ) ê is a 2D stationary stochastic process and the distribtion of ê is the same for all V, and 2) ě is also a 2D stationary stochastic process since it only depends on the neighboring ê. Hence, ê ě is also a 2D stationary stochastic process, and its second moment E[(ê ě ) 2 ] = E[(ε ) 2 ] is the same for all V. Therefore, we can drop from the notation, and let E[(ε ) 2 ] = E[(ε ) 2 ] for all V. Denote Ni (r) as the nmber of pixels contained in the i-th residal pacet of the -th frame; denote Pi (r) as PEP of the i-th residal pacet of the -th frame; denote N (r) as the total nmber of residal pacets of the -th frame. Since for all pixels in the same pacet, the residal XEP is eqal to its PEP, from (6) and (23), we have D (r) = = E[(ε ) 2 ] P (r) (24) V E[(ε ) 2 ] P (r) (25) V (a) = E[(ε ) 2 N ] (r) (Pi (r) Ni (r)) (26) i= (b) = E[(ε ) 2 ] P (r). (27) where (a) is de to P (r) = Pi (r) for pixel in the i-th residal pacet; (b) is de to P (r) N (r) (Pi (r) Ni (r)). (28) i= P (r) is a weighted average over PEP of all residal pacets in the -th frame, in which different pacets may contain different nmbers of pixels. Hence, given PEP of all residal pacets in the -th frame, or formla for the frame-level residal cased distortion is D (r) = E[(ε ) 2 ] P (r). (29) Note that with FMO mechanism, many neighboring pixels may be encoded into different slices and transmitted in different pacets. Since each pacet may experience different PEP especially in a fast fading channel, even neighboring pixels may have very different XEP. Therefore, (29) wors perfectly nder the FMO consideration. This sitation is taen into consideration throghot this paper. C. Analysis of Distortion Cased by MVCE Similar to the derivations in Section III-B, in this sbsection, we derive the formla for the pixel-level MV cased distortion D (m), and the frame-level MV cased distortion D (m). ) Pixel-level Distortion Cased by MVCE: Denote the MV error state for pixel by S(m), and denote the concealed MV by mv ˇ for general temporal error concealment methods when S(m) =. Therefore, we have { mv mv ˇ =, S(m) = mv, S(m) (3) =. +mv + ˇ Denote ξ where ξ mv, depends on the accracy of MV concealment, and the spatial correlation between reference pixel and concealed reference pixel at the encoder. A more comprehensive analysis of effect of inaccrate MV estimation on ξ can be fond in Ref. [24], which is then extended to spport mltihypothesis motion-compensated prediction [25] and to derive a rate-distortion model taing into accont the temporal prediction distance [26]. We also mae the following assmption. Assmption 3: S(m) is independent of ξ. Denote P (m) as the MV XEP for pixel, that is, P (m) P {S(m) = }. Note that it is possible that P (m) P (r) if SDP and UEP are applied. Given P (m), following the same derivation process in Section III-B, we can obtain D (m) = E[(ξ ) 2 ] P (m). (3) Also note that in H.264/AVC specification [6], there is no SDP for an instantaneos decoding refresh (IDR) frame; so S(r) = S(m) in an IDR-frame and hence P (r) = P (m). This is also tre for MB withot SDP. For P-MB with SDP in H.264/AVC, S(r) and S(m) are dependent. In other words, if the MV pacet is lost, the corresponding residal pacet cannot be decoded even if it is correctly received, since there is no slice header in the residal pacet. Therefore, the residal channel and the MV channel in Fig. are actally dependent if the encoder follows H.264/AVC specification. In this paper, we stdy transmission distortion in a more general case where S(r) and S(m) can be either independent or dependent. 6 2) Frame-level Distortion Cased by MVCE: To derive the frame-level MV cased distortion, we also mae the following assmption. Assmption 4: The second moment of ξ is the same for all V. Under Assmption 4, we can drop from the notation, and let E[(ξ ) 2 ] = E[(ξ) 2 ] for all V. Denote Ni (m) as the nmber of pixels contained in the i-th MV pacet of the -th frame; denote Pi (m) as PEP of the i-th MV pacet of the - th frame; denote N (m) as the total nmber of MV pacets of the -th frame. Then, given PEP of all MV pacets in the -th frame, following the same derivation process in Section III-B2, we obtain the frame-level MV cased distortion for the -th frame as D (m) = E[(ξ ) 2 ] P (m), (32) 6 To achieve this, we add side information to the H.264/AVC reference code JM4. by allowing residal pacets to be sed for decoder withot the corresponding MV pacets being correctly received, that is, ê can be sed to reconstrct f even if mv is not correctly received.

7 7 where P (m) N m i= (P i (m) N i (m)), a weighted average over PEP of all MV pacets in the -th frame, in which different pacets may contain different nmbers of pixels. D. Analysis of Distortion Cased by Propagated Error Pls Clipping Noise In this sbsection, we derive the distortion cased by error propagation in a non-linear decoder with clipping. We first derive the pixel-level propagation and clipping cased distortion D(P ). Then, we derive the frame-level propagation and clipping cased distortion D (P ). ) Pixel-level Distortion Cased by Propagated Error Pls Clipping Noise: First, we analyze the pixel-level propagation and clipping cased distortion D(P ) in P-MBs. From the definition, we now D(P ) depends on propagated error and clipping noise; and clipping noise is a fnction of RCE, MVCE and propagated error. Hence, D(P ) depends on RCE, MVCE and propagated error. Let r, m, p denote the event of occrrence of RCE, MVCE and propagated error respectively, and let r, m, p denote logical NOT of r, m, p respectively (indicating no error). We se a triplet to denote the joint event of three types of error; e.g., {r, m, p} denotes the event that all the three types of errors occr, and { r, m, p} denotes the pixel experiencing none of the three types of errors. When we analyze the condition that several error events may occr, the notation cold be simplified by the principle of formal logic. For example, { r, m} denotes the clipping noise nder the condition that there is neither RCE nor MVCE for pixel, while it is not certain whether the reference pixel has error. Correspondingly, denote P { r, m} as the probability of event { r, m}, that is, P { r, m} = P {S(r) = and S(m) = }. From the definition of P (r), the marginal probability P {r} = P (r) and the marginal probability P { r} = P (r). Similarly, P {m} = P (m) and P { m} = P (m). Define D(p) E[( ζ + { r, m}) 2 ]; and define +mv α D (p), which is called propagation factor for pixel D +mv. The propagation factor α defined in this paper is different from the propagation factor [], leaage [7], or attenation factor [5], which are modeled as the effect of spatial filtering or intra pdate; or propagation factor α is also different from the fading factor [8], which is modeled as the effect of sing fraction of referenced pixels in the reference frame for motion prediction. Note that D(p) is only a special case of D(P ) nder the error event of { r, m} for pixel. However, most existing models inappropriately se their propagation factor, obtained nder the error event of { r, m}, to replace D(P ) directly. To calclate E[( ζ + + mv ) 2 ] in (3), we need to analyze in for different error events for pixel : ) both residal and MV are erroneos, denoted by {r, m}; 2) residal is erroneos bt MV is correct, denoted by {r, m}; 3) residal is correct bt MV is erroneos, denoted by { r, m}; and 4) both residal and MV are correct, denoted by { r, m}. So, D(P ) = P {r, m} E[( ζ + + mv ˇ {r, m}) 2 ] + P {r, m} E[( ζ +mv + {r, m}) 2 ] + P { r, m} E[( ζ + + mv ˇ { r, m}) 2 ] + P { r, m} E[( ζ +mv + { r, m}) 2 ]. (33) Note that the concealed pixel vale shold be in the clipping fnction range, that is, Γ( f + ě + mv ) = f + ě + mv, so {r} =. Also note that if the MV channel is independent of the residal channel, we have P {r, m} = P (r) P (m). However, as mentioned in Section III-C, in H.264/AVC specification, these two channels are dependent. In other words, P { r, m} = and P { r, m} = P { r} for P-MBs with SDP in H.264/AVC 7. In sch a case, (33) is simplified to D(P ) = P {r, m} D + P + mv ˇ {r, m} D +mv + P { r} D(p). (34) Note that for P-MB withot SDP, we have P {r, m} = P { r, m} =, P {r, m} = P {r} = P {m} = P, and P { r, m} = P { r} = P { m} = P. Therefore, (34) can be frther simplified to D(P ) = P D + ( P + mv ˇ ) D(p). (35) Also note that for I-MB, there will be no transmission distortion if it is correctly received, that is, D (p) =. So (35) can be frther simplified to D(P ) = P D + mv ˇ. (36) Comparing (36) with (35), we see that I-MB is a special case of P-MB with D(p) =, that is, the propagation factor α = according to the definition. It is important to note that D(P ) > for I-MB since P. In other words, I-MB also contains the distortion cased by propagation error and it can be predicted by (36). However, existing linear time-invariant (LTI) models [7], [8] assme that there is no distortion cased by propagation error for I-MB, which nderestimates the transmission distortion. In the following part of this sbsection, we derive the propagation factor α for P-MB and prove some important properties of clipping noise. To derive α, we first give Lemma as below. Lemma : Given the PMF of the random variable ζ +mv and the vale of, D(p) can be calclated at the encoder by 7 In a more general case, where P { r, m} =, Eq. (34) can be sed as an approximation. This is becase E[( ζ + + mv ˇ { r, m}) 2 ] only happens in SDP condition, where the probability of MV pacet error is sally less than the probability of residal pacet error and the probability of the event that a residal pacet is correctly received bt the corresponding MV pacet is in error, i.e. P { r, m}, is very small. In addition, since {r} =, for the for different error events in (33), { r, m} is mch more similar to { r, m} than to {r, m} and {r, m}. Therefore, we may approximate the last two terms in (33) by P { r} E[( ζ + +mv { r, m}) 2 ], i.e. P { r} D(p).

8 8 D(p) = E[Φ 2 ( ζ +mv, )], where Φ(x, y) is called error redction fnction and defined by y γ l, y x < γ l Φ(x, y) y Γ(y x) = x, γ l y x γ h (37) y γ h, y x > γ h. Lemma is proved in Appendix A. In fact, we have fond in or experiments that in any error event, ζ +mv approximately follows Laplacian distribtion with zero mean. If we assme ζ follows Laplacian distribtion with zero +mv mean, the calclation for D(p) becomes simpler since the only nnown parameter for PMF of ζ is its variance. +mv Under this assmption, we have the following proposition. Proposition : The propagation factor α for propagated error with Laplacian distribtion of zero-mean and variance σ 2 is given by α = y γ 2 e l b ( y γ l b + ) γ 2 e h y b ( γ h y + ), b (38) where y is the reconstrcted pixel vale, and b = 2 2 σ. Proposition is proved in Appendix B. In the zero-mean Laplacian case, α will only be a fnction of and the variance of ζ +mv, which is eqal to D in this case. +mv has already been calclated dring the phase of Since D +mv predicting the ( )-th frame transmission distortion, D(p) can be calclated by D(p) = α D via the definition +mv of α. Then, we can recrsively calclate D(P ) in (34) since both D and D have been calclated previosly + mv ˇ +mv for the ( )-th frame. Next, we prove an important property of the non-linear clipping fnction in Proposition 2. To prove Proposition 2, we need to se the following lemma. Lemma 2: The error redction fnction Φ(x, y) satisfies Φ 2 (x, y) x 2 for any γ l y γ h. Lemma 2 is proved in Appendix C. From Lemma 2, we now that the fnction Φ(x, y) redces the energy of propagated error. This is the reason why we call it error redction fnction. With Lemma, it is straightforward to prove that whatever the PMF of ζ +mv is, D(p) = E[Φ 2 ( ζ, +mv )] E[( ζ ) 2 ] = D +mv +mv, (39) i.e., α. In other words, we have the following proposition. Proposition 2: Clipping redces propagated error, that is, D(p) D or α +mv,. Proposition 2 tells s that if there is no newly indced errors in the -th frame, transmission distortion decreases from the ( )-th frame to the -th frame. Fig. 2 shows the experimental reslt of transmission distortion propagation for bs seqence in cif format, where the third frame is lost at the decoder and all other frames are correctly received 8. 8 Since showing the experimental reslts for all trajectories is almost impossible in the paper, we jst show the reslt with mean sqare error (MSE) of all pixels in the same frame. MSE distortion Fig Only the third frame is lost The effect of clipping noise on distortion propagation. The experiment setp for Fig. 2, Fig. 3, Fig. 4, Fig. 5 and Fig. 6 is: JM4. [27] encoder and decoder are sed; the first frame is an I-frame, and the sbseqent frames are all P-frames withot inclding I-MB; For temporal error concealment, MV error concealment is the defalt frame copy in JM4. decoder de to its simplicity; residal pacets can be sed for decoder withot the corresponding MV pacets being correctly received as aforementioned; interpolation filter and deblocing filter are disabled. That is, the error redction is cased only by the clipping noise. In fact, if we consider the more general cases where there may be new error indced in the -th frame, we can still prove that E[( ζ + ) 2 ] E[( ζ ) 2 ] as shown in (6) + mv + mv dring the proof for the following corollary. Corollary : The correlation coefficient between ζ + mv and is non-positive. Specifically, they are negatively correlated nder the condition { r, p}, and ncorrelated nder other conditions. Corollary is proved in Appendix D. This property is very important for designing a low complexity algorithm to estimate propagation and clipping cased distortion in PTD, which is presented in the seqel paper [9]. 2) Frame-level Distortion Cased by Propagated Error Pls Clipping Noise: Define D (p) as the mean of D(p) V D (p); the formla over all V, i.e., D (p) for frame-level propagation and clipping cased distortion is given in Lemma 3. Lemma 3: The frame-level propagation and clipping cased distortion in the -th frame is D (P ) = D P (r) + D (p) ( P (r))( β ), (4) where P (r) is defined in (28); β is the percentage of I-MBs in the -th frame; D is the transmission distortion in the ( )-th frame. Lemma 3 is proved in Appendix F. Define the propagation factor for the -th frame α D (p) ; then, we have D V α D α +mv =. As explained in Appendix F, when D the nmber of pixels in the ( )-th frame is sfficiently

9 9 large, the sm of D over all the pixels in the ( )- +mv th frame will converge to D de to the randomness of Temporal correlation between residals in one trajectory mv. Therefore, we have α V α D +mv V D, which is +mv a weighted average of α with the weight being D +mv. As a reslt, D (p) D (P ) with high probability 9. However, most existing wors directly se D (P ) = D (p) in predicting transmission distortion. This is another reason why LTI models [7], [8] nderestimate transmission distortion when there is no MV error E. Analysis of Correlation Cased Distortion In this sbsection, we first derive the pixel-level correlation cased distortion D (c). Then, we derive the frame-level correlation cased distortion D (c). ) Pixel-level Correlation Cased Distortion: We analyze the correlation cased distortion D (c) at the decoder in for different cases: i) for { r, m}, both ε = and ξ =, so D (c) = ; ii) for {r, m}, ξ = and D (c) = 2E[ε ( ζ +mv + {r, m})]; iii) for { r, m}, ε = and D (c) = 2E[ξ ( ζ + ˇ + mv { r, m})]; iv) for {r, m}, + {r, m})] + D (c) = 2E[ε ξ ] + 2E[ε ( ζ + ˇ mv 2E[ξ ( ζ + + mv ˇ {r, m})]. From Section III-D, we now {r} =. So, we obtain (a) Temporal correlation between residals in one trajectory 3 4 D (c) = P {r, m} 2E[ε ζ +mv ] + P { r, m} 2E[ξ ( ζ + + mv ˇ { r, m})] + P {r, m} (2E[ε ξ] + 2E[ε ζ + mv ˇ ] + 2E[ξ ζ + mv ˇ ]). (4) In or experiments, we find that in the trajectory of pixel, ) the residal ê is almost ncorrelated with the residal in all other frames ê i v where i, i.e. their correlation coefficient is almost zero, as shown in Fig. 3 ; and 2) also the residal ê is almost ncorrelated with the MVCE of the corresponding pixel, i.e. ξ, and the MVCE in all previos frames, i.e. ξv, i where i <, as shown in Fig. 4. Based on the above observations, we frther assme that for any i <, ê is ncorrelated with ê i v and ξv i if v i is not in the trajectory of pixel, and mae the following assmption. Assmption 5: ê is ncorrelated with ξ, and is ncorrelated with both ê i v and ξv i for any i <. Since ζ and ζ are the transmission reconstrcted errors accmlated from all the frames before the +mv + mv ˇ -th frame, ε is ncorrelated with ζ and ζ de +mv + mv ˇ to Assmption 5. Ths, (4) becomes D (c) = 2P {m} E[ξ ζ + mv ˇ ] + 2P { r, m} E[ξ { r, m}]. (42) 9 When the nmber of reference pixels in the ( )-th frame is small, V α D may be larger than D in case the reference pixels +mv with high distortion are sed more often than the reference pixels with low distortion. Fig. 3, Fig. 4, Fig. 5 and Fig. 6 are plotted for low motion seqence, e.g. foreman, and high motion seqence, e.g. stefan, in cif format. All other seqences show the similar statistics. Fig (b) (a) foreman-cif, (b) stefan-cif. 2 However, we observe that in the trajectory of pixel, ) ê is correlated with ξ i v when i > and especially when i = + there are peas, as seen in Fig. 4; and 2) ξ is highly correlated with ξ i v as shown in Fig. 5. These interesting statistical relationships cold be exploited by an error concealment algorithm, e.g. finding a concealed MV for pixel v i with proper ξ i v given ê or ξ, and is sbject to or ftre stdy. As mentioned in Section III-D, for P-MBs with SDP in H.264/AVC, P { r, m} =. So, (42) becomes D (c) = 2P {m} E[ξ ( + mv ˇ 3 4 f + mv ˇ (43) )]. Note that in the more general case that P { r, m} =, Eq. (43) can be sed as an approximation since in (42), E[ξ { r, m}] is mch smaller than E[ξ ζ and + mv ˇ ] P { r, m} is mch smaller than P {m}. For MBs withot SDP, since P {r, m} = P { r, m} = and P {r, m} = P {r} = P {m} = P as mentioned in Section III-D, (4) can be simplified to D (c) = 2P (2E[ε ξ ] + 2E[ε Under Assmption 5, (44) redces to (43). ζ ] + 2E[ξ + mv ˇ ζ + mv ˇ ]). (44)

10 Temporal correlation between residal and concealment error in one trajectory Temporal correlation between concealment errors in one trajectory Residal frame index MV frame index (a) Temporal correlation between residal and concealment error in one trajectory (a) Temporal correlation between concealment errors in one trajectory Residal frame index (b) 2 MV frame index 3 2 (b) 2 3 Fig. 4. (a) foreman-cif, (b) stefan-cif. Fig. 5. (a) foreman-cif, (b) stefan-cif. Define λ E[ξ f + mv ˇ ] E[ξ ; + mv ˇ ] λ is a correlation ratio, that is, the ratio of the correlation between MVCE and concealed reference pixel vale at the decoder, to the correlation between MVCE and concealed reference pixel vale at the encoder. λ qantifies the effect of the correlation between the MVCE and propagated error on transmission distortion. Note that althogh we do not now the exact vale of λ at the encoder, its range is characterized by XEP of all pixels in the trajectory T, which passes throgh the pixel, as i= P i T(i) { r, m} λ, (45) where T(i) is the reference pixel position in the i-th frame for the trajectory T. For example, T( ) = + mv and T( 2) = T( ) + mv T( ). The left ineqality in (45) holds in the extreme case that any error in the trajectory will case ξ and f to be ncorrelated, which is sally tre + mv ˇ for high motion video. The right ineqality in (45) holds in another extreme case that all errors in the trajectory do not affect the correlation between ξ and f that is E[ξ + mv ˇ, f ] E[ξ + mv ˇ, which is sally tre for low + mv ˇ ] motion video. The details on how to estimate λ is presented in the seqel paper [9]. Using the definition of λ, we have the following lemma. Lemma 4: D (c) = (λ ) D (m). (46) Lemma 4 is proved in Appendix G. If we assme E[ξ] =, we may frther derive the correlation coefficient between ξ and Denote ρ as + mv ˇ. their correlation coefficient, from (7), we have ρ = E[ξ ] E[ξ + mv ˇ ] E[ + mv ˇ ] σ ξ σ = E[(ξ ) 2 ] = σξ. 2 σ ξ σ 2 σ (47) Similarly, it is easy to prove that the correlation coefficient between ξ σ ξ and is +mv 2 σ. This agrees well with the experimental reslts shown in Fig. 6. Via the same derivation process, one can obtain the correlation coefficient between ê and and between ê +mv, and. One possible application of these correlation properties is error concealment with partial information available.

11 Measred ρ between ξ and +mv Estimated ρ between ξ and +mv Measred ρ between ξ Estimated ρ between ξ and + mv ˇ and + mv ˇ (a) Measred ρ between ξ and +mv Estimated ρ between ξ and +mv Measred ρ between ξ Estimated ρ between ξ and + mv ˇ and + mv ˇ (b) Fig. 6. Comparison between measred and estimated correlation coefficients: (a) foreman-cif, (b) stefan-cif. 2) Frame-Level Correlation Cased Distortion: Denote Vi {m} the set of pixels in the i-th MV pacet of the -th frame. From (9), (7) and Assmption 4, we obtain D (c) = E[(ξ ) 2 ] = E[(ξ ) 2 ] V (λ ) P (m) N (m) i= {P i (m) V i {m} (λ )}. (48) Define λ V λ ; N V i (m) i {m} λ will converge to λ for any pacet that contains a sfficiently large nmber of pixels. By rearranging (48), we obtain D (c) = E[(ξ ) 2 ] N (m) i= = (λ ) E[(ξ ) 2 ] P (m). {P i (m) N i (m) (λ )} (49) From (32), we now that E[(ξ ) 2 ] P (m) is exactly eqal to D (m). Therefore, (49) is frther simplified to D (c) = (λ ) D (m). (5) F. Smmary In Section III-A, we decomposed transmission distortion into for terms; we derived a formla for each term in Sections III-B throgh III-E. In this section, we combine the formlae for the for terms into a single formla. ) Pixel-Level Transmission Distortion: Theorem : Under single-reference motion compensation, the PTD of pixel is D = D(r) + λ D(m) + P {r, m} D + mv ˇ + P {r, m} D + P +mv { r} α D (5) +mv. Proof: (5) can be obtained by plgging (23), (3), (34), and (7) into (4). Corollary 2: Under single-reference motion compensation and no SDP, (5) is simplified to D = P (E[(ε ) 2 ] + λ E[(ξ) 2 ] + D + mv ˇ ) + ( P ) α D +mv. (52) 2) Frame-Level Transmission Distortion: Theorem 2: Under single-reference motion compensation, the FTD of the -th frame is D = D (r) + λ D (m) + P (r) D + ( P (r)) D (p) ( β ). (53) Proof: (53) can be obtained by plgging (29), (32), (4) and (5) into (5). Corollary 3: Under single-reference motion compensation and no SDP, the FTD of the -th frame is simplified to D = P (E[(ε ) 2 ] + λ E[(ξ ) 2 ] + D ) + ( P ) α D ( β ) (54) Following the same deriving process, it is not difficlt to obtain the distortion prediction formlae nder mlti-reference case. De to the space limit, in this paper we jst present the formlae for distortion estimation nder single-reference case. Interested reader may refer to Ref. [22] for the analysis of mlti-reference case. In Ref. [22], we also identify the relationship between or reslt and existing models, and specify the conditions, nder which those models are accrate. IV. CONCLUSION In this paper, we derived the transmission distortion formlae for wireless video commnication systems. With consideration of spatio-temporal correlation, nonlinear codec and timevarying channel, or distortion prediction formlae improve the accracy of distortion estimation from existing wors. Besides that, or formlae spport, for the first time, the following capabilities: ) prediction at different levels (e.g., pixel/frame/gop level), 2) prediction for mlti-reference motion compensation, 3) prediction nder SDP, 4) prediction nder arbitrary slice-level pacetization with FMO mechanism, 5) prediction nder time-varying channels, 6) one nified formla for both I-MB and P-MB, and 7) prediction for both low motion and high motion video seqences. In addition, this paper also identified two important properties of transmission distortion for the first time: ) clipping noise, prodced by