Degrees of Freedom: Uncertainty Principle and Sampling

Transcription

1 Degrees of Freedom: Uncertainty Principle and Sampling Sergio Barbarossa Ack: M. Tsitsvero, P. Di Lorenzo, S. SardelliI 30/06/6 Sapienza Università di Roma July, 206

2 Overall summary. The WhiLaker- Nyquist- Kotelnikov- Shannon sampling theorem 2. Degrees of freedom 3. Sparse representatons 4. Uncertainty principles 5. Signals on graph 6. Conclusion 29/06/6 Sapienza Università di Roma July, 206 2

3 Sampling theorem The sampling theorem (as proved by Shannon): If a function s(t) contains no frequencies higher than B cps, it is completely determined by giving its ordinates at a series of points spaced /2B seconds apart Proof: By definition where S(f) with coefficients admits the Fourier series expansion: S n = 2B s(t) = S(f) = Z B B Z B B X S n e S(f)e j2 ft df j2 nf 2B S(f) e j2 nf/2b df = n 2B s 2B 27/06/6 Sapienza Università di Roma July, 206 3

4 Sampling theorem Substituting back, we get s(t) = X n= s n 2B sinc 2 B t n 2B As Shannon says, This is a fact which is common knowledge in the communication art. Theorem has been given previously in other forms by mathematcians but in spite of its evident importance seems not to have appeared explicitly in the literature of communicaton theory. Nyquist, however, and more recently Gabor, have pointed out that approximately 2BT numbers are sufficient, basing their arguments on a Fourier series expansion of the functon over the Tme interval T. 27/06/6 Sapienza Università di Roma July, 206 4

5 Sampling theorem A theorem with many inventors: Whittaker (95), Nyquist (928), Kotelnikov (933), Whittaker (935), Raabe (938), Gabor (946), Shannon (948), Isao Someya (949) E. T. WhiLaker, On the functons which are represented by the expansions of the interpolaton theory, 95 C(x) = X n= f(a + rw)sinc( (x a rw)/w) it is possible to interpolate given sampling values at intervals of w such that the Fourier transform of this interpolaton functon does not contain any terms with periods less than 2w V. A. Kotelnikov, On the transmission capacity of ether and wire in electrical communicatons, 933 Any func)on F(t) which consists of frequencies from 0 to f periods may be represented by the following series: X n F (t) = F sinc(2 f (t k/2f )) 2f k= 29/06/6 Sapienza Università di Roma July, 206 5

6 Sampling theorem Alternative perspective: A signal s(t) is perfectly band-limited between B and B if it remains unaltered when passing through an ideal lowpass filter with cut-off frequency B, i.e. s(t) But, since - B H(f) Substituting back, we get: B sinc(2 B(t )) = s(t) = f X n= X n= s(t) s sinc s(t) = 2 B t Z n 2B n sinc 2 B t 2B s( )sinc(2 B(t sinc n 2B n 2 B 2B )) d Note: This formulation does not require the existence of the Fourier Transform of s(t)! 29/06/6 Sapienza Università di Roma July, 206 6

7 Degrees of freedom Extensions Again, from Shannon If the function is limited to the time interval T and the samples are spaced /2B seconds apart, there will be a total of 2BT samples in the interval. All samples outside will be substantially zero. The numbers used to specify the function need not be the equally spaced samples used above One can further show that the value of the function and its derivative at every other sample point are sufficient. The value and first and second derivatives at every third sample point give a still different set of parameters which uniquely determine the function. Generally speaking, any set of 2BT independent numbers associated with the function can be used to describe it. Essentially, we have replaced a complex entity (say, a television signal) in a simple environment by a simple entity (a point) in a complex environment ( 2BT dimensional space ). 30/06/6 Sapienza Università di Roma July, 206 7

8 Degrees of freedom Formal approach (Slepian, Landau-Pollack) s(t) A band-limited signal is approximately time-limited in (-T/2, T/2), at level " T, if R t applet/2 s(t) 2 dt R s(t) 2 dt = " 2 T Let us denote by E(" T ) the set of band-limited functions approximately time-limited, at level " T E(" T ) i(t),i=0,...,n, The set is approximately N-dimensional if there exist N linearly independent functions such that Z NX min s(t) a k k (t) 2 dt apple N 2 {a k } k=0 A proper estimation of the dimension N requires the set min max min { i} s2e(" T ) {a k } Z s(t) { i (t)} NX k=0 to be chosen so that it achieves a k k (t) 2 dt 28/06/6 Sapienza Università di Roma July, 206 8

9 Degrees of freedom Def.: Time-limited operator Ds ( s(t) if t apple T/2 0 if t >T/2 Def.: Band-limited operator The optimal set is given by the eigenfunctions of the operator, i.e. and it provides the bound Bs Z B B S(f)e j2 ft df { i (t)} BDB BDB i = i i Z s(t) [2BT] X k=0 a k k (t) 2 dt apple C" 2 T where C is independent of s(t), " T, and 2BT 29/06/6 Sapienza Università di Roma July, 206 9

10 Degrees of freedom Landau attributes the following theorem to Shannon (unfortunately with no reference): > 0 C 3 = C 3 ( ) C 4 = C 4 ( ) s 2 E(" T ) Given any, there exist constants and, so that, for inf a k ks(t) N(BT) X k=0 a k k (t)k 2 apple ( + )" 2 T where N(BT) =[2BT]+C 3 log + (2BT)+C 4 In other words, for any given band-limited signal, lim T! N(BT) T =2B This result gives back the common understanding that the essential number of dimensions of a bandlimited signal, approximately concentrated in a time interval of duration T, scales as 2BT, as the time-bandwidth product increases Remark: The theorem assumes the use of the optimal waveforms (prolate spheroidal wave functions) 29/06/6 Sapienza Università di Roma July, 206 0

11 Degrees of freedom Extensions The prolate spheroidal wave functions are eigenfunctions of the linear time-varying system x(t) rect T (t) - B H(f) B f s(t) x(t) h(t, ) s(t) More generally, a class of signals can be defined as the set of possible outputs of a linear system In general, not much is known about the eigenfunctions of (non-normal) linear operators However, in case of underspread systems, good, although approximate, models are available 29/06/6 Sapienza Università di Roma July, 206

12 Degrees of freedom Extensions Suppose h(t, ) 2 L 2, then h(t, ) = X i u i (t) v i ( ) where and are the left and right singular functions associated to i=0 u i (t) v i (t) i If the system is underspread, left singular functions can be well approximated as u i (t) = X m A i,m (t) e j' i,m(t) where, and H(t, f i,m (t) 2 = i 2 f i,m (t) = 2 A i,m (t) = H(t,f) d' i,m (t) dt f=f i,m (t) 29/06/6 Sapienza Università di Roma July, 206 2

13 Degrees of freedom Example of eigenfunctions of underspread LTV systems 2 i e j' i,m(t) ' i,m (t) instantaneous frequency f i,m (t) 29/06/6 Sapienza Università di Roma July, 206 3

14 Degrees of freedom Are all level possible? No, the only admissible levels are such that the instantaneous frequencies respect an area rule f S i n i 2 t A natural discretization is inherent to the system 0/07/6 Sapienza Università di Roma July, 206 4

15 Degrees of freedom Example: Multipath channel affected by delay and Doppler shifts Contour level plots of H(t, f) 2 PSWVD of function associated to level Frequency Frequency Time PSWVD of function associated to level Time PSWVD of function associated to level Frequency Frequency Time PSWVD = Pseudo-Smoothed Wigner-Ville Distribution with Reassignment Time 29/06/6 Sapienza Università di Roma July, 206 5

16 Degrees of freedom Example: PSWD of prolate spheroidal wave functions 29/06/6 Sapienza Università di Roma July, 206 6

17 Degrees of freedom Studying the eigenfunctions of LTV systems opens the door to an infinite variety of tiling of the time-frequency domain Gabor (STFT) wavelets eigenfunctions frequency frequency frequency time time time 29/06/6 Sapienza Università di Roma July, 206 7

18 Sparse representations Given an underdetermined linear observation model y = Ax A 2 R m n x y m<n with, under what conditions can be recovered from when? y = A = observation matrix x measurement matrix s Recovery may be possible only if (or ) is sparse and satisfies some properties x x s sparsitying basis might not be directly sparse, but there should exist a sparsifying transformation leading to a sparse A s 30/06/6 Sapienza Università di Roma July, 206 8

19 Sparse representations Conditions for unique recovery x Denote by k the sparsity of and by Sk n the set of vectors of size n having at most k nonzero entries The right place to look for the conditions of unique recovery of a sparse vector is the null space of A N (A) ={z : Az = 0} x, x 0 2 S n k A(x x 0 ) 6= 0 N (A) Given any two vectors, the need to ensure requires that does not contain any vector in S2k n Spark (Kruskal rank) of a matrix: The spark of a matrix is the smallest number of columns of that are linearly dependent A A Uniqueness of sparse recovery: For any vector x 2 S n k y 2 R n, there exists at most one k-sparse y = Ax signal such that if and only if spark(a) > 2k To guarantee uniqueness, we also need m 2k 29/06/6 Sapienza Università di Roma July, 206 9

20 Sparse representations Alternative conditions for unique recovery Restricted isometry property (RIP) A k 2 (0, ) A matrix satisfies the RIP of order k if there exists a such that ( k)kxk 2 2 applekaxk 2 2 apple ( + k )kxk 2 2 for all x 2 S n k In words, if a matrix satisfies the RIP of order 2k, that matrix preserves the distance between any pair of k-sparse vectors Problem: Computing RIP constants is typically NP-hard! 29/06/6 Sapienza Università di Roma July,

21 Sparse representations Recovery algorithms Original problem (nonconvex): Basis pursuit (convex) : min kxk 0 subject to Ax = y min kxk subject to Ax = y Variants (QCBP) min kxk subject to kax yk 2 apple Problem: Under what conditions, do the relaxed problems yield same solution as the original problem? 29/06/6 Sapienza Università di Roma July, 206 2

22 Sparse representations The power of randomness Alternative algorithms require different conditions on the RIP to ensure unique recovery However, computing the RIP is NP-hard Interestingly, if A is drawn at random, unique recovery is possible with high probability provided that m n Ck ln k 29/06/6 Sapienza Università di Roma July,

23 Uncertainty principles ConTnuous- Tme signals R 2 Time spread: T = (t t 0) 2 x(t) 2 dt R, x(t) 2 dt t 0 = R t x(t) 2 dt R x(t) 2 dt R 2 Frequency spread: F = (f f 0) 2 X(f) 2 df R, X(f) 2 df f 0 = R f X(f) 2 df R X(f) 2 df, Heisenberg s principle: T F 4 A signal that is perfectly localized in Tme cannot be perfectly localized in frequency and viceversa 29/06/6 Sapienza Università di Roma July,

24 Uncertainty principles AlternaTve approach (Slepian- Landau- Pollack) Define Tme and frequency spread as the dimensions T and W of the intervals in Tme and frequency such that a given percentage of energy falls within them: Z t0 +T/2 t 0 T/2 Z x(t) 2 dt x(t) 2 dt = 2. Z f0 +W/2 f 0 W/2 Z X(f) 2 df X(f) 2 df = 2. Not all pairs (, ) are admissible Aim of the uncertainty principle is to find out the region of all admissible pairs 29/06/6 Sapienza Università di Roma July,

25 Uncertainty principles Uncertainty principle for sparse vectors Given two orthonormal bases and, any vector can be expanded uniquely in terms of each one of these bases as Setting and, uncertainty relations show that A and B cannot be small simultaneously x = A = kak 0 B = kbk 0 { `, apple ` apple n} { `, apple ` apple n} x nx a` ` = nx `= `= b` ` Mutual coherence: Thm: µ(, ) = max `,r ` r 2 (A + B) p AB µ(, ) 29/06/6 Sapienza Università di Roma July,

26 Uncertainty principles Uncertainty principle for sparse vectors If and are chosen as the cardinal (identity) basis and the Fourier bases, then µ(, )= p n In such a case, 2 (A + B) p AB p n A =[, ] Thm: Let be the concatenation of two orthonormal bases and where both matrices are unitary and of size m x m. If then, for each measurement vector such that y = Ax k< µ(, ) = µ(a) y 2 R m there exists at most one k-sparse vector x 30/06/6 Sapienza Università di Roma July,

27 Uncertainty principles Is there a good common basis for spikes and sinusoids? Numerical example 30/06/6 Sapienza Università di Roma July,

28 Signals on graph Given a graph G(V, E), a graph signal is defined as a mapping from the set of vertces to the real space of dimension equal to the cardinality of : V MoTvaTng examples f : V! R Wireless sensor networks (temperature, stress, ) Biological networks (molecule concentraton, gene actvaton, ) Social networks (politcal orientaton, ranking, ) Basic notaton: adjacency matrix: Laplacian matrix: A L = diag{a} A 29/06/6 Sapienza Università di Roma July,

29 Signals on graph Graph Fourier Transform (GFT) - Laplacian decompositon [Pes08, ZhuRab2, ShuNarFroOrtVan3, ] NX Given L = U U T = iu i u T i, GFT: i= ˆx = U T x Inverse GFT: x = U ˆx - Adjacency decompositon [SanMou3, CheVarSanKov5, ] Given A = VJV GFT: ˆx = V x Inverse GFT: x = V ˆx The key point is to find a basis that admits a compact representaton: x = Us 30/06/6 Sapienza Università di Roma July,

30 Signals on graph Graph Fourier Transform (GFT) Note: If graph is circulant the eigenvectors of the Laplacian concide with the Fourier basis and the GFT boils down to conventonal DFT Discrete- Time Signal Processing can be seen as a par8cular example of Graph Signal Processing 30/06/6 Sapienza Università di Roma July,

31 Signals on graph Examples of GFT Temperature field GFT Latitude (deg) Longitude (deg) /06/6 Sapienza Università di Roma July, 206 3

32 Signals on graph Examples of GFT Temperature field Fiedler vector Latitude (deg) Latitude (deg) Longitude (deg) Longitude (deg) 30/06/6 Sapienza Università di Roma July,

33 Signals on graph Examples of GFT Noisy temp. field ReconstrucTon with 2 eigenvectors Latitude (deg) Longitude (deg) /06/6 Sapienza Università di Roma July,

34 Signals on graph Examples of GFT Noisy temp. field ReconstrucTon with 24 eigenvectors Latitude (deg) Longitude (deg) /06/6 Sapienza Università di Roma July,

35 Signals on graph Given a graph G(V, E) and a subset of vertces S V, we define - vertex- limitng operator (projector over set ) : D = diag(d,...,d NN ) where d ii =, if i 2 S; d ii =0, if i/2 S, - band- limitng operator (projector over set ): where and U B = U U T, is a diagonal matrix selectng the frequency indices, i.e. ii =, if i 2 F; ii =0, if i/2 F is the matrix whose columns are the basis of signal expansion L F A (e.g., eigenvectors of, or eigenvectors of, ) S 30/06/6 Sapienza Università di Roma July,

36 Signals on graph Perfect localizaton conditons Def.: A vector is perfectly localized over the subset if and it is perfectly band- limited over Theorem: A vector is perfectly localized over both vertex set and frequency set if and only if In such a case, x x Dx = x F if Bx = x x S F max(bdb) = is the eigenvector associated to the the unit eigenvalue of S BDB Equivalently kbdk = ; kdbk = 30/06/6 Sapienza Università di Roma July,

37 Signals on graph Uncertainty principle Define concentraton measures: kdxk 2 kxk 2 = 2 ; kbxk 2 kxk 2 = 2 Theorem: The only admissible concentra)on pairs (, ) specified by the following inequali)es belong to the region cos + cos cos max (BD) cos p 2 + cos cos max BD cos + cos p 2 cos max BD cos p 2 + cos p 2 cos max BD B where D and represent projector on complement sets and S F 30/06/6 Sapienza Università di Roma July,

38 Signals on graph Uncertainty principle Example of admissible region 2 2 max B D 2 max (BD) cos + cos cos max (BD) Maximum sum concentraton is achieved with = 2 max B D 2 max BD 2 2 = 2 ( + max) Note: A corner curve shrinks to the corner point in case of perfect localizaton over the corresponding domain 30/06/6 Sapienza Università di Roma July,

39 Signals on graph Sampling Let us denote by r = Ds the sampled signal Theorem: Given any band- limited signal, i.e. satsfying it is possible to recover from its sampled version if and only if In words, a band- limited signal can be perfectly reconstructed iff there exists no band- limited signal that is perfectly localized on the same frequency set the complement vertex set S Bs = s s kdbk < r F and on 30/06/6 Sapienza Università di Roma July,

40 Signals on graph Sampling and uncertainty principle The upper let corner of the uncertainty region specifies the conditons for signal recovery from its samples: 2 Given 2 =, we need to stay away from the point 2 =0 2 max B D 2 max (BD) More generally, if 2 < we need to guarantee 2 max(bd) < 2 2 max B D 2 max BD 2 30/06/6 Sapienza Università di Roma July,

41 Signals on graph ReconstrucTon Algorithms: A.: A.2 (AlternaTng ProjecTon) ŝ = I DB r r (0) = r r () = r + DB r (0) r (k) = r + DB r (k ) A.3 where i are soluton of ŝ = F X i= BDB i = 2 hdr, i i i i i i 30/06/6 Sapienza Università di Roma July, 206 4

42 Signals on graph Not all samples are equal 30/06/6 Sapienza Università di Roma July,

43 Signals on graph Not all samples are equal NMSE, db Random MaxFro MinUniSet MaxSigMin MaxVol MinPinv Exhaustive NMSE, db Random MaxFro MinUniSet MaxSigMin MaxVol MinPinv Number of samples Number of samples and going random can be extremely bad! 30/06/6 Sapienza Università di Roma July,

44 Signals on graph Ques8on: Is perfect recovery possible in the presence of (very large) spiky noise? Given the observaton model r = s + Dn Apply ` - norm minimizaton ŝ = arg min s2b kr sk 30/06/6 Sapienza Università di Roma July,

45 Signals on graph Ques8on: Is perfect recovery possible in the presence of (very large) spiky noise? Numerical results 0 20 MSE, (db) F =5 F = 0 F = Number of noisy samples, S Answer: Yes, provided that # samples is sufficiently larger than # of noise samples 30/06/6 Sapienza Università di Roma July,

46 Conclusion The real number of degrees of freedom is the number of independent disciplines that Shannon mastered and brought to synthesis: communicaton theory, switching circuits, biology, Having had to invent InformaTon Theory almost from scratch looks like a necessary occurrence of an incredibly brilliant mind Sampling theory is stll well alive, its lille kids, Compressive Sensing, Finite Rate of InnovaTon Sampling, Union of Subspace Sampling, are growing well 30/06/6 Sapienza Università di Roma July,

47 Conclusion Developments Information Theory and Biology What is it that makes living systems to strive for decreasing (local) entropy? Information Theory and Topology Algebraic topology should provide the tools to classify qualitatively different information sources Topological Signal Processing and Machine Learning Graph-based methods are just the simplest way to capture two-way relations among variables, multiway relations are the general setting Information Theory and Physics Quantum gravity may receive a significant contribution from IT approaches Semantic Information Theory 0/07/6 Sapienza Università di Roma July,