Wavelets and Filter Banks on Graphs

Transcription

1 Wavelets and Filter Banks on Graphs Pierre Vandergheynst Signal Processing Lab, EPFL Joint work with David Shuman Duke Workshop on Sensing and Analysis of High-Dimensional Data Duke University, July 2011

2 2 Processing Signals on Graphs Social Network Electrical Network Neuronal Network Transportation Network

3 2 Processing Signals on Graphs Social Network Electrical Network Neuronal Network Transportation Network

4 3 Short outline Summary of one wavelet construction on graphs - multiscale, filtering Pyramidal algorithms - polyphase components and downsampling - the Laplacian Pyramid - 2-channels, critically sampled filter banks?

5 4 Spectral Graph Wavelets G=(E,V) a weighted undirected graph, with Laplacian L = D A

6 4 Spectral Graph Wavelets G=(E,V) a weighted undirected graph, with Laplacian L = D A Dilation operates through operator: T t g = g(tl)

7 4 Spectral Graph Wavelets G=(E,V) a weighted undirected graph, with Laplacian L = D A Dilation operates through operator: T t g = g(tl) Translation (localization): Define ψ t,j = T t gδ j ψ t,j (i) = response to a delta at vertex j N 1 =0 g(tλ )φ (j)φ (i) ψ t,a (u) = Lφ (j) =λ φ (j) R dω ˆψ(tω)e jωa e jωu

8 4 Spectral Graph Wavelets G=(E,V) a weighted undirected graph, with Laplacian L = D A Dilation operates through operator: T t g = g(tl) Translation (localization): Define ψ t,j = T t gδ j ψ t,j (i) = response to a delta at vertex j N 1 =0 g(tλ )φ (j)φ (i) ψ t,a (u) = Lφ (j) =λ φ (j) And so formally define the graph wavelet coefficients of f: W f (t, j) =ψ t,j,f W f (t, j) =T t gf(j) = N 1 =0 R dω ˆψ(tω)e jωa e jωu g(tλ ) ˆf()φ (j)

9 5 Frames A, B > O, h : R + R + (i.e. scaling function) 0 <A h 2 (u) + s g(t su) 2 B < scaling function wavelets φ n = T h δ n = h(l)δ n A 2 B 1 A simple way to get a tight frame: 0 0 λ 1 γ(λ )= 1 1/2 dt t g2 (tλ ) g(λ )= γ(λ ) γ(2λ ) for any admissible kernel g

10 6 Scaling & Localization ψ t,i (j)

11 6 Scaling & Localization ψ t,i (j) decreasing scale

12 7 Example

13 7 Example

14 7 Example

15 7 Example

16 7 Example

17 7 Example

18 8 Sparsity and Smoothness on Graphs scaling functions coeffs

19 9 Remark on Implementation Not necessary to compute spectral decomposition for filtering Polynomial approximation : g(tω) K 1 a k (t)p k (ω) 1 k=0 ex: Chebyshev, minimax It is to implement any Fourier multiplier 0 0 λ 40 Then wavelet operator expressed with powers of Laplacian: T t g K 1 k=0 a k (t)l k And use sparsity of Laplacian in an iterative way

20 10 Remark on Implementation W f (t, j) = p(l)f # j W f (t, j) W f (t, j) Bf W f (t n,j)= 1 2 c n,0f # + M n k=1 sup norm control (minimax or Chebyshef) c n,k T k (L)f # j T k (L)f = 2 a 1 (L a 2 I) T k 1 (L)f T k 2 (L)f Computational cost dominated by matrix-vector multiply with (sparse) Laplacian J matrix. In particular O( M n E ) n=1 Note: same algorithm for adjoint!

21 11 Graph wavelets Redundancy breaks sparsity - can we remove some or all of it? Faster algorithms - traditional wavelets have fast filter banks implementation - whatever scale, you use the same filters - here: large scales -> more computations Goal: solve both problems at one

22 12 Basic Ingredients Euclidean multiresolution is based on two main operations Filtering (typically low-pass and high-pass) Down and Up sampling

23 12 Basic Ingredients Euclidean multiresolution is based on two main operations Filtering (typically low-pass and high-pass) Down and Up sampling

24 12 Basic Ingredients Euclidean multiresolution is based on two main operations Filtering (typically low-pass and high-pass) Down and Up sampling Filtering is fine but how do we downsample on graphs???

25 13 Basic Ingredients Subsampling is equivallent to splitting in two cosets (even, odd)

26 13 Basic Ingredients Subsampling is equivallent to splitting in two cosets (even, odd) Questions: How do we partition a graph into meaningful cosets? Are there efficient algorithms for these partitions? Are there theoretical guarantees? How do we define a new graph from the cosets?

27 14 Cosets - A spectral view Subsampling is equivallent to splitting in two cosets (even, odd) Classically, selecting a coset can be interpreted easily in Fourier: f sub (i) = 1 2 f(i) 1 + cos(πi) eigenvector of largest eigenvalue

28 15 Cosets and Nodal Domains Nodal domain: maximally connected subgraph s.t. all vertices have same sign w.r.t a reference function We would like to find a very large number of nodal domains, ideally V! Nodal domains of Laplacian eigenvectors are special (and well studied)

29 15 Cosets and Nodal Domains Nodal domain: maximally connected subgraph s.t. all vertices have same sign w.r.t a reference function We would like to find a very large number of nodal domains, ideally V! Nodal domains of Laplacian eigenvectors are special (and well studied) Theorem: the number of nodal domains associated to the largest laplacian eigenvector of a connected graph is maximal, ν(φ max )=ν(g) = V IFF G is bipartite In general: ν(g) = V χ(g)+2 (extreme cases: bipartite and complete graphs)

30 16 Cosets and Nodal Domains Nodal domain: maximally connected subgraph s.t. all vertices have same sign w.r.t a reference function We would like to find a very large number of nodal domains, ideally V! Nodal domains of Laplacian eigenvectors are special (and well studied) For any connected graph we will thus naturally define cosets and their associated selection functions V + = {i V s.t. φ N 1 (i) 0} M + (i) = sgn(φn 1 (i)) V = {i V s.t. φ N 1 (i) < 0} M (i) = sgn(φn 1 (i))

31 17 Examples of cosets Simple line graph φ k (u) =sin(πku/n + π/2n) λ k =2 2 cos(πk/n) 1 k n

32 17 Examples of cosets Simple line graph φ k (u) =sin(πku/n + π/2n) λ k =2 2 cos(πk/n) 1 k n

33 18 Examples of cosets Simple line graph Simple ring graph φ 1 k(u) =sin(2πku/n) φ 2 k(u) = cos(2πku/n) 1 k n/2 λ k =2 2 cos(2πk/n)

34 18 Examples of cosets Simple line graph Simple ring graph φ 1 k(u) =sin(2πku/n) φ 2 k(u) = cos(2πku/n) 1 k n/2 λ k =2 2 cos(2πk/n)

35 19 Examples of cosets Simple line graph Simple ring graph Lattice

36 19 Examples of cosets Simple line graph Simple ring graph Lattice quincunx

37 20 The Agonizing Limits of Intuition Multiplicity of λ max - how do we choose the control vector in that subspace? - even a prescription can be numerically ill-defined - graphs with flat spectrum in close to their spectral radius Laplacian eigenvectors do not always behave like global oscillations - seems to be true for random perturbations of simple graphs - true even for a class of trees [Saito2011]

38 21 The Laplacian Pyramid Analysis operator x - G y 1 U H D y low

39 22 The Laplacian Pyramid Analysis operator x - y 1 G H D U y 0

40 23 The Laplacian Pyramid Analysis operator x - y 1 G H M y 0

41 23 The Laplacian Pyramid Analysis operator x - y 1 G H M y 0 y 0 = H m x = MHx y 1 = x Gy 0 = x GH m x

42 23 The Laplacian Pyramid Analysis operator x - y 1 G H M y 0

43 23 The Laplacian Pyramid Analysis operator x - y 1 G H M y 0 y0 y 1 y = H m x, I GH m T a

44 24 The Laplacian Pyramid Analysis operator y0 y 1 y = H m x, I GH m T a

45 24 The Laplacian Pyramid Analysis operator y0 y 1 y = H m I GH m T a x, Simple (traditional) left inverse ˆx =(G I ) T s y0 y 1 y T s T a = I with no conditions on H or G

46 25 The Laplacian Pyramid Pseudo Inverse? T a = T a T T a 1Ta T Let s try to use only filters

47 25 The Laplacian Pyramid Pseudo Inverse? T a = T a T T a 1Ta T Let s try to use only filters Define iteratively, through descent on LS: arg min x T a x y 2 2 ˆx k+1 =ˆx k + τt a T (y T aˆx k ) T a T =(H m T I H m T G T )

48 26 The Laplacian Pyramid we can easily implement T a T T a with filters and masks: With the real symmetric matrix x N = τ N 1 Q = T a T T a (I τq) j b j=0 Use Chebyshev approximation of: L(ω) =τ and N 1 b = T a T y (1 τω) j j=0

49 27 Kron Reduction In order to iterate the construction, we need to construct a graph on the reduced vertex set. A r = A[α, α] A[α, α)a(α, α) 1 A(α, α] A[α, α] A[α, α) A = A(α, α] A(α, α)

50 Kron Reduction In order to iterate the construction, we need to construct a graph on the reduced vertex set. A r = A[α, α] A[α, α)a(α, α) 1 A(α, α] A[α, α] A[α, α) A = A(α, α] A(α, α) 1.0 Kron reduction 1/3 1/ /3 [Dorfler et al, 2011]

51 28 Kron Reduction In order to iterate the construction, we need to construct a graph on the reduced vertex set. A r = A[α, α] A[α, α)a(α, α) 1 A(α, α] A[α, α] A[α, α) A = A(α, α] A(α, α) Properties: maps a weighted undirected laplacian to a weighted undirected laplacian spectral interlacing (spectrum does not degenerate) λ k (A) λ k (A r ) λ k+n α (A) disconnected vertices linked in reduced graph IFF there is a path that runs only through eliminated nodes

52 29 Example Note: For a k-regular bipartite graph kin A L = A T ki n Kron-reduced Laplacian: L r = k 2 I n AA T

53 29 Example Note: For a k-regular bipartite graph kin A L = A T ki n Kron-reduced Laplacian: L r = k 2 I n AA T ˆf r (i) = ˆf(i)+ ˆf(N i) i =1,...,N/2

54 30

55 30

56 30

57 30

58 30

59 30

60 31 Filter Banks 2 critically sampled channels Filter H Coset 1 Downsample f 0 Filter G Downsample Coset 2

61 31 Filter Banks 2 critically sampled channels Coset 1 Filter H Downsample f 0 Filter G Downsample Coset 2 Theorem: For a k-rbg, the filter bank is perfect-reconstruction IFF H(i) 2 + G(i) 2 =2 H(i)G(N i)+h(n i)g(i) =0

62 32 Conclusions Structured, data dependent dictionary of wavelets - sparsity and smoothness on graph are merged in simple and elegant fashion - fast algo, clean problem formulation - graph structure can be totally hidden in wavelets Filter banks based on nodal domains or coloring - Universal algo based on filtering and Kron reduction - Efficient IFF some structure in the graph - Unfortunately no closed form theory in general

63 33

64 34 Wavelet Ingredients Wavelet transform based on two operations: Dilation (or scaling) and Translation (or localization) ψ s,a (x) = 1 x a s ψ s

65 34 Wavelet Ingredients Wavelet transform based on two operations: Dilation (or scaling) and Translation (or localization) ψ s,a (x) = 1 x a s ψ s (T s f)(a) = 1 s ψ x a f(x)dx s (T s f)(a) =ψ (s,a),f

66 34 Wavelet Ingredients Wavelet transform based on two operations: Dilation (or scaling) and Translation (or localization) ψ s,a (x) = 1 x a s ψ s (T s f)(a) = Equivalently: 1 s ψ x a f(x)dx (T s f)(a) =ψ s (s,a),f (T s δ a )(x) = 1 s ψ x a s e iωx ˆψ (sω) ˆf(ω)dω (T s f)(x) = 1 2π

67 35 Graph Laplacian and Spectral Theory G =(V,E,w) weighted, undirected graph Non-normalized Laplacian: L = D A Real, symmetric (Lf)(i) = i j w i,j (f(i) f(j)) Why Laplacian?

68 35 Graph Laplacian and Spectral Theory G =(V,E,w) weighted, undirected graph Non-normalized Laplacian: L = D A Real, symmetric (Lf)(i) = i j w i,j (f(i) f(j)) Why Laplacian? Z 2 with usual stencil (Lf) i,j =4f i,j f i+1,j f i 1,j f i,j+1 f i,j 1 In general, graph laplacian from nicely sampled manifold converges to Laplace-Beltrami operator

69 36 Graph Laplacian and Spectral Theory d 2 dx 2 e iωx f(x) = 1 2π ˆf(ω)e iωx dω

70 36 Graph Laplacian and Spectral Theory d 2 dx 2 e iωx f(x) = 1 2π ˆf(ω)e iωx dω Eigen decomposition of Laplacian: Lφ l = λ l φ l

71 36 Graph Laplacian and Spectral Theory d 2 dx 2 e iωx f(x) = 1 2π ˆf(ω)e iωx dω Eigen decomposition of Laplacian: For simplicity assume connected graph and Lφ l = λ l φ l 0=λ 0 < λ 1 λ 2... λ N 1 For any function on the vertex set (vector) we have: N ˆf() =φ,f = φ (i)f(i) Graph Fourier Transform f(i) = N 1 =0 i=1 ˆf()φ (i)

72 37 Spectral Graph Wavelets Remember good old Euclidean case: (T s f)(x) = 1 e iωx 2π ˆψ (sω) ˆf(ω)dω We will adopt this operator view

73 37 Spectral Graph Wavelets Remember good old Euclidean case: (T s f)(x) = 1 e iωx 2π ˆψ (sω) ˆf(ω)dω We will adopt this operator view Operator-valued function via continuous Borel functional calculus g : R + R + T g = g(l) Operator-valued function Action of operator is induced by its Fourier symbol T g f() =g(λ ) ˆf() N 1 (T g f)(i) = g(λ ) ˆf()φ (i) =0

74 38 Non-local Wavelet Frame Non-local Wavelets are Graph Wavelets on Non-Local Graph ψ t, (i) increasing scale Interest: good adaptive sparsity basis

75 39 Distributed Computation Scenario: Network of N nodes, each knows - local data f(n) - local neighbors - M Chebyshev coefficients of wavelet kernel - A global upper bound on largest eigenvalue of graph laplacian

76 39 Distributed Computation Scenario: Network of N nodes, each knows - local data f(n) - local neighbors - M Chebyshev coefficients of wavelet kernel - A global upper bound on largest eigenvalue of graph laplacian To compute: Φf (j 1)N+n = 1 2 c j,0f + M k=1 c j,k T k (L)f n

77 39 Distributed Computation Scenario: Network of N nodes, each knows - local data f(n) - local neighbors - M Chebyshev coefficients of wavelet kernel - A global upper bound on largest eigenvalue of graph laplacian To compute: T 1 (L)f n Φf (j 1)N+n = 1 2 c j,0f + = 2 α (L αi)f n M k=1 c j,k T k (L)f n sensor only needs f(n) from its neighbors

78 39 Distributed Computation Scenario: Network of N nodes, each knows - local data f(n) - local neighbors - M Chebyshev coefficients of wavelet kernel - A global upper bound on largest eigenvalue of graph laplacian To compute: T 1 (L)f n Φf (j 1)N+n = 1 2 c j,0f + = 2 α (L αi)f n M k=1 c j,k T k (L)f n sensor only needs f(n) from its neighbors T k (L)f = 2 α (L αi) T k 1 (L)f T k 2 (L)f Computed by exchanging last computed values

79 Distributed Computation Communication cost: 2M E messages of length 1 per node Example: distributed denoising, or distributed regression, with Lasso a (k) Total communication cost: 1 arg min a 2 y Φ a a 1,µ i = S µi,τ a k 1 + τφ(y Φ a k 1 ) i The iterative soft thresholding algorithm converges 2 minimizer Cost of (20), E N if τ < W [25]. The fina 2 estimate of the signal is then given by W a. We now turn to the issue of how to implement th Denoised Signal 2M E messages gorithm of length in a distributed 1 fashion by sending messag neighbors in the network. OneCost option would E be 4M E messages distributed of length lasso algorithm J+1of [26], which is a spec the Alternating Direction Method of Multipliers [2 In every iteration of that algorithm, each node tr Sensing and Analysis of current High-D estimate Data of all the wavelet coefficients t neighbors. With a transform the size of the spec wavelet transform, this requires 2 E total messag Distributed Lasso [Mateos, Bazerque, Gianakis] Chebyshev Φy ΦΦ a argmin a 1 2 y W a a 1,µ, where a 1,µ := N(J+1) i=1 µ i a i. The optimizatio in (20) can be solved for example by iterative soft th [24]. The initial estimate of the wavelet coeffi is arbitrary, and at each iteration of the soft th algorithm, the update of the estimated wavelet coe given by (( [ a (k) i = S µi τ a (k 1) + τw y W a (k 1)]) i k =1, 2, where τ is the step size and S µi τ is the shrinka thresholding operator { 0, if z µi τ S µi τ (z) := z sgn(z)µ i τ, o.w. 40

80 41 Wavelets on Graphs? Existing constructions - wavelets on meshes (computer graphics, numerical analysis), often via lifting - diffusion wavelets [Maggioni, Coifman & others] - recently several other constructions based on organizing graph in a multiscale way [Gavish-Coifman] Goal - process signals on graphs - retain simplicity and signal processing flavor - algorithm to handle fairly large graphs

81 42 Scaling & Localization Effect of operator dilation?

82 42 Scaling & Localization Effect of operator dilation?

83 42 Scaling & Localization Effect of operator dilation?

84 42 Scaling & Localization Effect of operator dilation? Theorem: d G (i, j) >K and g has K vanishing derivatives at 0 ψ t,j (i) ψ t,j Dt for any t smaller than a critical scale function of d G (i, j) Reason? At small scale, wavelet operator behaves like power of Laplacian