The analysis and representation of random signals

Transcription

1 The analysis and reresentation of random signals Bruno TOÉSNI B. Torrésani LTP Université de Provence.1/30

2 Outline 1. andom signals Introduction The Karhunen-Loève Basis roximate and adative KL bases. Linear vs Non-linear aroximations simle examle Non-linear aroximations: the rice of nonlinearity Back to redundancy B. Torrésani LTP Université de Provence./30

3 # andom signals 1. GENELITIES Let be a robability sace and denote by the sace of random variables on : or. lmost sure equality of random variables defines an equivalence relation and set. For define when the integral exists. Set also and "! is equied with a ilbert sace structure thanks to the ermitean roduct #! Notice that if then. B. Torrésani LTP Université de Provence.3/30

4 $% $ $ &. - $ & & /0 /0 & /0 8 andom signals () DEFINITION. Let 1. random signal on. 3.. labelled by is a second order random signal if for all is uniformly second order +) is a maing for all +) &. for some $( - *).. DEFINITION. Let 1. its mean be a second order random signal. Introduce +). its correlation and covariance ) & 3. its covariance oerator 1 0 defined by 43 +) &. ; : ; ; : 9 for all : 9 such that the above exression exists. Notice that (formally) 9 & - 0! & 9 B. Torrésani LTP Université de Provence.4/30

5 $!!?<!!!! &? andom signals (3) LEMM. If &"< &>= is second order both and? = 1 0 and - 0 are ositive-semidefinite: for all! G & D - DFE CB < The sectral theory of will lay a maor role in the following. B. Torrésani LTP Université de Provence.5/30

6 $!!?<!!!! &? $ $ I &J &J & &J I /0 I /0 $ $ & andom signals (3) LEMM. If &"< &>= is second order both and? = 1 0 and - 0 are ositive-semidefinite: for all! G & D - DFE CB < The sectral theory of will lay a maor role in the following. Continuity and stationarity DEFINITION. J I is wide sense (or second order) stationary if for all I & such that and 1 0 J I $ 0! & DEFINITION. If is a continuous subset of is second order continuous if for all! P3 *) ) O KLNM 3 B. Torrésani LTP Université de Provence.5/30

7 andom signals (5) GUSSIN WITE NOISE ND STTIONY GUSSIN NOISE Frequency Frequency B. Torrésani LTP Université de Provence.6/30

8 andom signals (6) GUSSIN WITE NOISE ND NON-STTIONY GUSSIN NOISE Frequency Frequency B. Torrésani LTP Université de Provence.7/30

9 andom signals (7). TE KUNEN-LOÈVE BSIS Let be second order continuous and centered. Under suitable assumtions the sectral theory of rovides a natural orthonormal basis for reresenting the random signal. For examle B. Torrésani LTP Université de Provence.8/30

10 $!!! Q G Q = $ - $ Z $ W < E = andom signals (7). TE KUNEN-LOÈVE BSIS Let be second order continuous and centered. Under suitable assumtions the sectral theory of rovides a natural orthonormal basis for reresenting the random signal. For examle MECE S TEOEM. ssume is an interval in. 1. is trace class and has a countable sectrum of non-negative eigenvalues with finite multilicity Q+< G. Parseval s formula holds: 3. The corresonding family of eigenvectors orthonormal basis of. In addition 4. For all & W < E V = U TS X =. such that for all. X = X = QY= X = forms an X = & X = Q[= & - 0 where the series converges uniformly. B. Torrésani LTP Université de Provence.8/30

11 $ $% Z 8 & _ $ & W < E = andom signals (8) TE KUNEN-LOÈVE TEOEM. Let a second order random signal labelled by the comact subset second order continuous. For all introduce the random variable & *) ]!!! &! +) X = ; X = ^ = Then the ^4` ^ = ^ = are second order random variables and are orthogonal in and for all one has (in convergence) a ` = Q ` : &! X = ^ = 5) B. Torrésani LTP Université de Provence.9/30

12 $ $% Z 8 & _ & $ W < E < E V ). c b W d E b andom signals (8) TE KUNEN-LOÈVE TEOEM. Let a second order random signal labelled by the comact subset second order continuous. For all introduce the random variable & *) ]!!! &! +) X = ; X = ^ = Then the ^4` ^ = ^ = are second order random variables and are orthogonal in and for all one has (in convergence) a ` = Q ` : &! X = ^ = 5) PPOXIMTION. For set & X = ^ =. Then if is trace-class Q[= and is the best linear -dimensional aroximation of. B. Torrésani LTP Université de Provence.9/30

13 f $ X 7 ) k & X o k & andom signals (9) Examle: stationary signals. ssume that and that is a eriodic wide sense stationary second order continuous random signal. Then the covariance oerator is diagonal in the Fourier basis: h g X i 0 where lnm and ) & lnm - 0 < i 0 is the (real-valued) ower sectrum of. B. Torrésani LTP Université de Provence.10/30

14 f $ X 7 ) k & X o k & c < q & o E V ) d <. s r andom signals (9) Examle: stationary signals. ssume that and that is a eriodic wide sense stationary second order continuous random signal. Then the covariance oerator is diagonal in the Fourier basis: h g X i 0 where lnm and ) & lnm - 0 < i 0 is the (real-valued) ower sectrum of. roximation. Set again X ^ =. Then if i 0! i 0 r B. Torrésani LTP Université de Provence.10/30

15 andom signals (10) emark: The choice of local trigonometric bases in some coders (for examle JPEG or MP1) was motivated by the fact that the corresonding signals when studied within sufficiently small subdomains (squares or time frames) look stationary... hence Fourier-tye methods. B. Torrésani LTP Université de Provence.11/30

16 $ andom signals (10) emark: The choice of local trigonometric bases in some coders (for examle JPEG or MP1) was motivated by the fact that the corresonding signals when studied within sufficiently small subdomains (squares or time frames) look stationary... hence Fourier-tye methods. emark: When is not a bounded domainthe sectral theory of the covariance oerator is more difficult as may not be comact. Nevertheless the results are at least qualitatively comarable. B. Torrésani LTP Université de Provence.11/30

17 $ c $ andom signals (10) emark: The choice of local trigonometric bases in some coders (for examle JPEG or MP1) was motivated by the fact that the corresonding signals when studied within sufficiently small subdomains (squares or time frames) look stationary... hence Fourier-tye methods. emark: When is not a bounded domainthe sectral theory of the covariance oerator is more difficult as may not be comact. Nevertheless the results are at least qualitatively comarable. For examle in the case of second order continuous and stationary random signals labelled by as well as for second order stationary discrete random signals labelled by is a (continuous or discrete) convolution oerator; these signals also ossess a sectral reresentation based uon the (continuous) Fourier transform: the Cramèr reresentation. B. Torrésani LTP Université de Provence.11/30

18 andom signals (10) 3. PPOXIMTE KUNEN-LOÈVE BSES: In many situations the true Karhunen-Loève basis is not available. It has to be: Estimated from signal samles: first estimate a (samled) covariance matrix then comute eigenvalues and eigenvectors (rincial comonent analysis). The resulting eigenvectors do not necessarily yield tractable systems. B. Torrésani LTP Université de Provence.1/30

19 andom signals (10) 3. PPOXIMTE KUNEN-LOÈVE BSES: In many situations the true Karhunen-Loève basis is not available. It has to be: Estimated from signal samles: first estimate a (samled) covariance matrix then comute eigenvalues and eigenvectors (rincial comonent analysis). The resulting eigenvectors do not necessarily yield tractable systems. roximated using rior knowledge on the signal; examle are rovided by subband decomositions. B. Torrésani LTP Université de Provence.1/30

20 andom signals (10) 3. PPOXIMTE KUNEN-LOÈVE BSES: In many situations the true Karhunen-Loève basis is not available. It has to be: Estimated from signal samles: first estimate a (samled) covariance matrix then comute eigenvalues and eigenvectors (rincial comonent analysis). The resulting eigenvectors do not necessarily yield tractable systems. roximated using rior knowledge on the signal; examle are rovided by subband decomositions. Both aroximated and estimated. B. Torrésani LTP Université de Provence.1/30

21 andom signals (10) 3. PPOXIMTE KUNEN-LOÈVE BSES: In many situations the true Karhunen-Loève basis is not available. It has to be: Estimated from signal samles: first estimate a (samled) covariance matrix then comute eigenvalues and eigenvectors (rincial comonent analysis). The resulting eigenvectors do not necessarily yield tractable systems. roximated using rior knowledge on the signal; examle are rovided by subband decomositions. Both aroximated and estimated. In some situations time-frequency and time-scale bases rovide accurate aroximate Karhunen-Loève bases. This is the case for (adative) local cosine bases and locally stationary signals: signals which are close to stationary when studied within small enough time frames. B. Torrésani LTP Université de Provence.1/30

22 u - v ] - 0 w 0 u u z 9 } 3o 9 - k z ] ] 0 andom signals (11) Locally stationary signals: Consider a centered version of the covariance: &J t y w ] w ] J x v - 0 v & & - 0 Then &J t &J t ) lnm i 0 & & 9 B. Torrésani LTP Université de Provence.13/30

23 u - v ] - 0 w 0 u u z 9 } 3o 9 - k z ] ] 0 k I ~ - 0 z andom signals (11) Locally stationary signals: Consider a centered version of the covariance: &J t y w ] w ] J x v - 0 v & & - 0 Then &J t &J t ) lnm i 0 & & 9 defined by i 0 DEFINITION. The function I lnm o k ƒ ƒ. I lnm o ~ i 0 o. is the Wigner sectrum of B. Torrésani LTP Université de Provence.13/30

24 h] & q & q & h] q~ q~ f~ ] ~ andom signals (1) The heuristics: ssume that for all there exists an interval within that interval the signal looks like stationary: f& &J ] such that - 0 & & for all & J ~ ] ssume further that 0 has (reasonably) fast decay as a function of its second argument. B. Torrésani LTP Université de Provence.14/30

25 h] & q & q & h] q~ q~ f~ ] ~ W - & } o ) k & B andom signals (1) The heuristics: ssume that for all there exists an interval within that interval the signal looks like stationary: f& &J ] such that - 0 & & for all & J ~ ] ssume further that Consider now a function : lnm 0 has (reasonably) fast decay as a function of its second argument. :)ˆ :. almost localized within that interval and let B. Torrésani LTP Université de Provence.14/30

26 h] & q & q & h] q~ q~ f~ ] ~ W - & } o ) k & B u &J t o ) - B k ] 0 } o ) o k ~ - 0 & z B andom signals (1) The heuristics: ssume that for all there exists an interval within that interval the signal looks like stationary: f& &J ] such that - 0 & & for all & J ~ ] ssume further that Consider now a function almost localized within that interval and let. Taking into account the time-frequency localization roerties of one gets : lnm 0 has (reasonably) fast decay as a function of its second argument. :)ˆ : :) & lm } : - 0 & : lm k } 3o ) lnm : & : ~ i 0 so that the covariance oerator is almost diagonalized by such functions. Question: how to turn this heuristics into rigorous arguments? B. Torrésani LTP Université de Provence.14/30

27 q q M / q q Œ M ; v 8 andom signals (13) Mallat Paanicolaou and Zhang: Consider a local cosine basis with the same notations as before set LN and denote by covariance. 7 v the corresonding matrix elements of the B. Torrésani LTP Université de Provence.15/30

28 q q M / q q Œ M ; v 8 v / Z J < J z m andom signals (13) Mallat Paanicolaou and Zhang: Consider a local cosine basis with the same notations as before set LN and denote by the corresonding matrix elements of the covariance. DEFINITION. The signal is locally stationary if there exists a local cosine basis such that there exist two constants and such that 7 v +) and if for all there exists a constant = such that = = J = where Dš B. Torrésani LTP Université de Provence.15/30

29 . c z ; v 8 andom signals (14) roximation by a band matrix: Let ; consider defined by its matrix elements v œž Ÿ if otherwise and B. Torrésani LTP Université de Provence.16/30

30 . c z ; v 8 7! ] G Z = J andom signals (14) roximation by a band matrix: Let ; consider defined by its matrix elements v œž Ÿ if otherwise and TEOEM. If is the covariance oerator of a locally stationary signal In addition for all integer there exists a constant = such that for all = Therefore may be well aroximated by a band matrix. B. Torrésani LTP Université de Provence.16/30

31 . c z ; v 8 7! ] G Z = J andom signals (14) roximation by a band matrix: Let ; consider defined by its matrix elements v œž Ÿ if otherwise and TEOEM. If is the covariance oerator of a locally stationary signal In addition for all integer there exists a constant = such that for all = Therefore may be well aroximated by a band matrix. B. Torrésani LTP Université de Provence.16/30

32 k z o ) ] ) andom signals (15) denote the Kohn-Nirenberg symbol of the Covariance oerator: < i 0 Examle: Let z z )n lnm & < i 0 & & - < 0 & G satisfies for all If the Kohn-Nirenberg symbol of & q & «< ª ) and? (with - q for some function & q LN defines a locally stationary rocess. then B. Torrésani LTP Université de Provence.17/30

33 @ < b ; v 8 ; 8 andom signals (16) Estimation: ssume we are given Consider the coefficients realizations 7 v of the random signal. and introduce the estimator!!! ;! v v b < E B. Torrésani LTP Université de Provence.18/30

34 @ < b!! ; v v ; 8 ² andom signals (16) of the random signal. and introduce the estimator! realizations 7 v Estimation: ssume we are given Consider the coefficients ;! v 8 b < E POPOSITION:. is an unbiased estimator: 1. is a Gaussian signal then *). If J ± b Var B. Torrésani LTP Université de Provence.18/30

35 @ < b!! ; v v ; 8 ² G z andom signals (16) of the random signal. and introduce the estimator! realizations 7 v Estimation: ssume we are given Consider the coefficients ;! v 8 b < E POPOSITION:. is an unbiased estimator: 1. is a Gaussian signal then *). If J ± b Var and emark : Notice that even though the coefficients decay raidly as grow we still have Var Therefore it is not worth trying to estimate the full covariance matrix. z B. Torrésani LTP Université de Provence.18/30

36 z ; v 8 Ÿ e ³ d d e ³ andom signals (17) of the matrix ³ : Consider a truncated version and if œž z v otherwise and ³ Then J B. Torrésani LTP Université de Provence.19/30

37 ³ z ; v 8 ³ e ³ d e ³ d µ ³ M ³ ³ ³ andom signals (17) Consider a truncated version of the matrix : v œž Ÿ if otherwise and z Then and J Estimation in a library of local cosine bases: Let a library of local cosine bases be given. We want to choose the otimal one. natural otimality criterion is to minimize the absolute deviation for a given bandwidth :. The latter quantity being difficult to handle it is more realistic to use the quadratic deviation instead. ssuming (i.e. ) consider - 0 LN v TS U! U TS U TS B. Torrésani LTP Université de Provence.19/30

38 U S 7 7 µ ³ andom signals (18) elace the minimization of For each basis use the estimators norm as with the maximization of. as above and estimate the ilbert-schmidt U S B. Torrésani LTP Université de Provence.0/30

39 U S 7 7 µ ³ t e J 0 z andom signals (18) elace the minimization of For each basis use the estimators norm as with the maximization of. as above and estimate the ilbert-schmidt U S If the signal is Gaussian CB b J U ¹S u 7 b U ¹S ³ d the latter sum involving only those ence the bias may be controlled. and such that and. B. Torrésani LTP Université de Provence.0/30

40 U S 7 7 µ ³ t e J 0 z U S U S ³ ³ andom signals (18) elace the minimization of For each basis use the estimators norm as with the maximization of. as above and estimate the ilbert-schmidt U S If the signal is Gaussian CB b J U ¹S u 7 b U ¹S ³ d the latter sum involving only those ence the bias may be controlled. and such that and. Practical covariance estimation: maximize ³ :»º arg ¼½¾ B. Torrésani LTP Université de Provence.0/30

41 andom signals (19) This otimization may be erformed in the framework of a hierarchical search algorithm within a structured family of local cosine bases: search the basis for which the matrix is closest to diagonal. w 00 w w w w w w B. Torrésani LTP Université de Provence.1/30

42 b b b b ÀÃ Â ÂÂB ÀÁ < E = Linear vs Non-Linear The revious discussion was essentially based on linear aroximation ideas: Karhunen-Loève basis aroximation by a subset of the basis. Given an orthonormal basis tyes of -dimensional aroximations of!!! Z X = in a ilbert sace : one can consider different The Linear aroximation: start from a fixed subset of basis functions (for examle the first in the Karhunen-Loève basis case) and roect onto the closed subsace of sanned by those. X = the Non-linear aroximation: for all minimize the best -term aroximation: look for the basis functions that Xl5Ä = LN B. Torrésani LTP Université de Provence./30

43 f ] & ] Linear vs Non-Linear () 1. SIMPLE EXMPLE (Meyer Cohen & d les) Consider the robability sace equied with the Lebesgue measure and the random signal defined as follows: for all set for and for. & *) & J & +) 0 ω 1 t B. Torrésani LTP Université de Provence.3/30

44 f ] & ] ¼ ÆÅ t & ¼ < E Linear vs Non-Linear () 1. SIMPLE EXMPLE (Meyer Cohen & d les) Consider the robability sace equied with the Lebesgue measure and the random signal defined as follows: for all set for and for. & *) & J & +) 0 ω 1 t This signal is clearly a second order random signal. In addition it is also second order continuous and stationary. The Karhunen-Loève theorem alies and yields the following exansion of the signal onto its otimal basis! LN W u J ] +) B. Torrésani LTP Université de Provence.3/30

45 Ç È o ƒ < o b b Linear vs Non-Linear (3) The exansion of that signal on the KL basis converges slowly: terms are needed in order to achieve an norm error smaller than. More generally the aroximation error goes like being the number of terms. B. Torrésani LTP Université de Provence.4/30

46 Ç È o ƒ < o b b o ] b Linear vs Non-Linear (3) The exansion of that signal on the KL basis converges slowly: terms are needed in order to achieve an norm error smaller than. More generally the aroximation error goes like being the number of terms. 0 ω 1 t On the oosite the aroximation error goes like if one uses an exansion rovided the wavelet has enough vanishing moments. terms wavelet B. Torrésani LTP Université de Provence.4/30

47 Linear vs Non-Linear (4) The main difference between these two aroaches is that the wavelet aroximation is non-linear; that articular signal is an examle of those for which non-linear aroximation outerforms linear aroximation (in terms of seed of convergence). This toy examle has served as starting oint for building more realistic (cartoon) models for images (Cohen & d les Meyer...) B. Torrésani LTP Université de Provence.5/30

48 Linear vs Non-Linear (4) The main difference between these two aroaches is that the wavelet aroximation is non-linear; that articular signal is an examle of those for which non-linear aroximation outerforms linear aroximation (in terms of seed of convergence). This toy examle has served as starting oint for building more realistic (cartoon) models for images (Cohen & d les Meyer...) B. Torrésani LTP Université de Provence.5/30

49 f ] ¼ ÆÅ b. c Z = É Z b É b Linear vs Non-Linear (5) nother examle : Let & +) g be a random signal defined by b& 5) where is a discrete random variable for examle geometrically distributed Ê gain is second order wide sense zero mean second order continuous and stationary. Its Karhunen-Loève basis is the Fourier basis. owever is is easy to see that best term exansion beats the linear aroximation with Karhunen-Loève basis. B. Torrésani LTP Université de Provence.6/30

50 Linear vs Non-Linear (6). TE PICE OF NONLINEITY By definition non-linear aroximation is always at least as good as linear aroximation. For some signal rocessing tasks (signal analysis denoising...)it rovides a suitable answer. owever there are alications such as signal coding for which the differences in erformances are not so clear... B. Torrésani LTP Université de Provence.7/30

51 Linear vs Non-Linear (6). TE PICE OF NONLINEITY By definition non-linear aroximation is always at least as good as linear aroximation. For some signal rocessing tasks (signal analysis denoising...)it rovides a suitable answer. owever there are alications such as signal coding for which the differences in erformances are not so clear... Why? the addresses of significant coefficients have to be encoded as well; in a number of ractical situations this may become costly. Coefficient encoding has to come with side information: the significance ma: assign a bit to each coefficient (retained or not). B. Torrésani LTP Université de Provence.7/30

52 Linear vs Non-Linear (7) un length coding : Consider a significance ma (50 bits) and encode the lengths of regions of zeros and ones: ÌË Í] Î] ÍË Î] Ï Since these are all smaller than 8 they may be encoded using 3 bits only so 36 bits all together. B. Torrésani LTP Université de Provence.8/30

53 Ï Î] ÍË Î] Í] ÌË W Z É Ð < E = Linear vs Non-Linear (7) un length coding : Consider a significance ma (50 bits) and encode the lengths of regions of zeros and ones: Since these are all smaller than 8 they may be encoded using 3 bits only so 36 bits all together. Entroy coded run length coding : If entroy coding is used instead of constant length coding this cost may be decreased further to (aroximately) the Shannon entroy of the distribution of lengths: Z! É Ñ K Æ B. Torrésani LTP Université de Provence.8/30

54 Linear vs Non-Linear (8) Structured coefficient sets : In general situations (for examle for uniformly distributed significant coefficients) not much is gained via entroy coded run length coding. Otherwise if the coefficients satisfy some ersistence roerty i.e. regions are large enough entroy coded run length coding (or similar techniques) is worthy. B. Torrésani LTP Université de Provence.9/30

55 Ð Linear vs Non-Linear (8) Structured coefficient sets : In general situations (for examle for uniformly distributed significant coefficients) not much is gained via entroy coded run length coding. Otherwise if the coefficients satisfy some ersistence roerty i.e. regions are large enough entroy coded run length coding (or similar techniques) is worthy. Examle: if the significance ma is modeled as a Markov chain 1 π π 0 1 π 1 π The Shannon entroy may be comuted exlicitely! Ê Ñ K Æ Ê K Æ Ñ Ê Ñ K Æ Ê K Æ Ñ The larger and the most efficient the entroy coded run length coding. B. Torrésani LTP Université de Provence.9/30

56 c < E = Linear vs Non-Linear (9) 3. BCK TO EDUNDNCY The main idea of Best -term aroximation: Given an orthonormal basis the aroximation Z X = find Xl Ä = with fastest convergence. B. Torrésani LTP Université de Provence.30/30

57 c < E = Linear vs Non-Linear (9) 3. BCK TO EDUNDNCY The main idea of Best -term aroximation: Given an orthonormal basis the aroximation Z X = find Xl Ä = with fastest convergence. In such a way: the fact that the family of waveforms used in the exansion is an orthonormal basis does not aear strictly necessary. In some situations frames or dictionaries may work as well if the functions used in the exansion may be encoded efficiently. B. Torrésani LTP Université de Provence.30/30