Sparse linear models

Transcription

1 Sparse linear models Optimization-Based Data Analysis Carlos Fernandez-Granda 2/22/2016

2 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

3 Linear model A linear model for a signal x R n is a representation of the form x = m c i φ i i=1 {φ 1,..., φ m } is a family of atoms in R n c R m is an alternative representation or transform of x

4 Sparse linear model A sparse linear model contains a small number of coefficients x = i S c i φ i S m How do we choose a transform that sparsifies a class of signals? Intuition / Domain knowledge (this lecture) Learning it from the data (later on)

5 Why? Signals of interest (speech, natural images, biomedical activity, etc.) are often highly structured Sparse linear models are able to exploit this structure to enhance data analysis and processing Applications: Compression Denoising Inverse problems

6 Sparse representation in an orthonormal basis If the atoms {φ 1,..., φ n } form an orthonormal basis, then U := [ ] φ 1 φ 2 φ n UU T = I Coefficients are obtained by computing inner products with the atoms x = UU T x = m φ i, x φ i i=1

7 Sparse representation in a basis If the atoms {φ 1,..., φ n } form a basis, then B := [ φ 1 φ 2 φ n ] BB 1 = I Coefficients are obtained by computing inner products with dual atoms x = BB 1 x = m θ i, x φ i i=1 where θ T 1 B 1 θ = T 2 θ T n

8 Overcomplete dictionaries If the atoms {φ 1,..., φ m } are linearly independent and m > n D := [ φ 1 φ 2 φ m ] Two alternative sparse models 1. Synthesis sparse model x = Dc Problem: Given x find a sparse c 2. Analysis sparse model: where c is sparse D T x is sparse Both are equivalent if D is an orthonormal basis

9 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

10 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

11 Fourier series Fourier series coefficients of f : [0, 1] C Fourier series S n (t) := c k := n k= n 1 0 f (t) e i2πkt dt c k e i2πkt = n k= n φ k, f φ k Sinusoidal atoms φ k (t) = e i2πkt = cos (2πkt) + i sin (2πkt) Orthonormal basis of L 2 under the usual inner product lim n f (t) S n (t) = 0 for all f L 2

12 Discrete Fourier transform (DFT) Discretized frequency representation for vectors in C n Sinusoidal atoms are a basis for C n φ k = 1 n 1 e i2πk n e i2πk2 n e i2πk(n 1) n DFT {x} k := (Fx) k = φ k, x F := [ φ 0 φ 1 φ n 1 ] The fast Fourier transform (FFT) computes the DFT in O (n log n) The discrete cosine transform (DCT) is a related transformation designed for real vectors

13 Discrete cosine transform Signal DCT coefficients

14 Electrocardiogram

15 Electrocardiogram (spectrum)

16 Electrocardiogram (spectrum)

17 2D DFT Discretized frequency representation for 2D arrays in C n n Sinusoidal atoms are a basis for C n n φ 2D k 1,k 2 = 1 n 1 e i2πk 2 n e i2πk 2 (n 1) n e i2πk 1 n e i2π(k 1 +k 2) n e i2π(k 1 +k 2 (n 1)) n e i2πk 1 (n 1) n e i2π(k 1 (n 1)+k 2) n e i2π(k 1 (n 1)+k 2 (n 1)) n ( ) T = φ 1D k 1 φ 1D k 2 DFT 2D {X } := FXF = n 1 n 1 k 1 =0 k 2 =0 φ 2D k 1,k 2, X φ 2D k 1,k 2 Generalizes to C m n, m n, and to higher dimensions

18 Compression via frequency representation The 2D frequency representation of images tends to be sparse Thresholding the coefficients yields a compressed representation The JPEG compression standard is based on the 2D DCT High-frequency coefficients are discarded according to a perceptual model

19 Compression via frequency representation Original

20 Compression via frequency representation 10 % largest DCT coeffs

21 Compression via frequency representation 2 % largest DCT coeffs

22 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

23 Motivation Spectrum of speech, music, etc. varies over time Idea: Compute frequency representation of time segments of the signal We must use a window to avoid introducing spurious high frequencies

24 The need for windowing Signal Window = Spectrum =

25 The need for windowing Signal Window = Spectrum =

26 Short-time Fourier transform Let w : [0, 1] C be a window function localized in time and frequency STFT {f } (k, τ) := 1 0 f (t) w (t τ)e i2πkt dt = φ k,τ, f Each atom φ k,τ (t) := w (t τ) e i2πkt corresponds to w shifted by τ in time and by k in frequency In discrete time, pointwise multiplication by a shifted window followed by a DFT, equivalent to D T x where D C n m, m > n The STFT of speech tends to be sparse (analysis sparse model) Including dilations of w (in addition to time and frequency translations) yields a dictionary of Gabor atoms

27 Atom τ = 0, k = 0 Real part Imaginary part Spectrum

28 Atom τ = 1/32, k = 0 Real part Imaginary part Spectrum

29 Atom τ = 0, k = 64 Real part Imaginary part Spectrum

30 Atom τ = 1/32, k = 64 Real part Imaginary part Spectrum

31 Speech signal

32 Spectrum

33 Spectrogram (log magnitude of STFT coefficients) Frequency Time

34 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

35 Wavelets Aim: Approximate signals at different scales A wavelet ψ is a unit-norm, zero-mean function in L 2 Wavelet transform W {f } (s, τ) := 1 1 ( t τ f (t) ψ s s 0 ) dt = φ s,τ, f Atoms are dilations and translations of the mother wavelet φ s,τ (t) = 1 ( ) t τ ψ s s We can build orthonormal basis for L 2 using wavelets

36 Multiresolution approximation Sequence {V j, j Z} of closed subspaces of L 2 (R) such that P Vj (f ) is an approximation of f at scale 2 j Conditions: Dilating functions in V j by 2 yields functions in V j+1 f (t) V j f ( t V j+1 2) Approximations at a scale 2 j are always better than at 2 j+1 V j+1 V j

37 Multiresolution approximation V j is invariant to translations at the scale 2 j f (t) V j f ( t 2 j k ) V j for all k Z As j the approximation loses all information lim V j = {0} j As j the approximation is perfect lim j V j = L 2 There exists a scaling function ζ V 0 such that { ζ0,k (t) := ζ (t k), k Z } is an orthonormal basis for V 0

38 { ψ2 j,k, k Z } is an orthonormal basis for V j V j { ζ0,k (t), ψ 2 1,k, ψ 2 2,k,..., ψ 2 j,k, k Z } is an orthonormal basis for V j Wavelet basis Mallat and Meyer prove that there exists a wavelet ψ such that P Vj (f ) = P Vj+1 (f ) + k Z ψ2 j,k, f ψ 2 j,k. Many different wavelet bases: Meyer, Daubechies, Battle-Lemarie,... Discrete wavelet transform can be computed in O (n) Signal processing interpretation: Wavelets act as band-pass filters, scaling functions act as low-pass filters

39 Haar wavelet Scaling function Mother wavelet

40 Electrocardiogram Signal Haar transform

41 Scale 2 9 Contribution Approximation

42 Scale 2 8 Contribution Approximation

43 Scale 2 7 Contribution Approximation

44 Scale 2 6 Contribution Approximation

45 Scale 2 5 Contribution Approximation

46 Scale 2 4 Contribution Approximation

47 Scale 2 3 Contribution Approximation

48 Scale 2 2 Contribution Approximation

49 Scale 2 1 Contribution Approximation

50 Scale 2 0 Contribution Approximation

51 2D Wavelets Extension to 2D by using outer products of 1D atoms φ 2D s 1,s 2,k 1,k 2 := φ 1D s 1,k 1 ( φ 1D s 2,k 2 ) T Yields sparse representation of natural images The JPEG 2000 compression standard is based on 2D wavelets Many extensions: Steerable pyramid, ridgelets, curvelets, bandlets,...

52 2D wavelet transform

53 2D wavelet transform

54 Sorted coefficients

55 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

56 Overcomplete dictionaries Atoms {φ 1,..., φ m } are linearly independent and m > n D := [ φ 1 φ 2 φ m ] Synthesis sparse model: x = Dc where c is sparse Problem: There are infinite choices of c such that x = Dc Some may not be sparse at all! Example: Dictionary of sinusoids

57 Dictionary of sinusoids x = Dc c

58 First idea Apply pseudoinverse ĉ := D x = D T ( DD T ) 1 x Interpretations: Projection of c onto row space of D Solution to minimize c 2 subject to x = D c,

59 Dictionary of sinusoids: Minimum l 2 -norm coefficients

60 Computing sparse representations We would like to solve Computationally intractable minimize c 0 subject to x = D c Two possibilities: Greedy methods: Select atoms one by one l 1 -norm minimization: min c R m c 1 such that x = Dc

61 Matching pursuit (MP) Iteratively choose atoms that are most correlated with the signal Initialization: Iterations: k = 1, 2,... φ (k) = arg max j ˆx (k) = ˆx (k 1) + r (0) = x ˆx (0) = 0 r (k 1), φ k r (k 1), φ (k) φ (k) r (k) = r (k 1) r (k 1), φ (k) φ (k)

62 Dictionary of sinusoids: Coefficients Original MP

63 Dictionary of sinusoids: Approximation Original MP

64 Orthogonal matching pursuit (OMP) Makes sure approximation is orthogonal to residual at Initialization: Iterations: k = 1, 2,... φ (k) = arg max j r (0) = x r (k 1), φ k A (k) = [ φ (1) φ (2) φ (k)] ĉ (k) = A (k) x = ˆx (k) = A (k) ĉ (k) r (k) = x ˆx (k) ( A (k) A (k)t ) 1 A (k)t x

65 Dictionary of sinusoids: Coefficients Original OMP

66 Dictionary of sinusoids: Approximation Original OMP

67 l 1 -norm minimization Estimate coefficients by solving minimize c 1 subject to x = D c Computationally tractable (convex program) Known as basis pursuit in the literature

68 Geometric intuition c 2 Min. l 1 -norm solution Min. l 2 -norm solution c 1 Dc = x

69 Dictionary of sinusoids: Coefficients Original l 1 -norm min.

70 Dictionary of sinusoids: Approximation Original l 1 -norm min.

71 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

72 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

73 Denoising Aim: Extracting information (signal) from data in the presence of uninformative perturbations (noise) Additive noise model data = signal + noise y = x + z Prior knowledge about structure of signal vs structure of noise is required

74 Electrocardiogram Spectrum

75 Electrocardiogram: High-frequency noise (power line hum) Original spectrum Low-pass filtered spectrum

76 Electrocardiogram: High-frequency noise (power line hum) Original spectrum Low-pass filtered spectrum

77 Electrocardiogram: High-frequency noise (power line hum) Original Low-pass filtered

78 Electrocardiogram: High-frequency noise (power line hum) Original Low-pass filtered

79 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

80 Thresholding Prior knowledge: Signal is a sparse superposition of dictionary atoms Noise is not (incoherence between atoms and noise) Hard-thresholding operator H η (x) i := { x i if x i > η, 0 otherwise

81 Denoising via thresholding Data Signal

82 Denoising via thresholding Estimate Signal

83 Sparsity in a basis Assumption: x = Bc, where c is sparse Threshold B 1 y ĉ = H η ( B 1 y ) = H η ( c + B 1 z ) ŷ = Bĉ Noise and sparsifying atoms should be incoherent, i.e. B 1 z is not sparse Example: Orthogonal sparsifying basis and Gaussian noise

84 Denoising via thresholding in DCT basis DCT coefficients Data Signal Data

85 Denoising via thresholding in DCT basis DCT coefficients Estimate Signal Estimate

86 Denoising via thresholding in a wavelet basis

87 Denoising via thresholding in a wavelet basis

88 2D wavelet coefficients

89 Original coefficients

90 Thresholded coefficients

91 Estimate

92 Estimate

93 Denoising via thresholding in a wavelet basis Original Noisy Estimate

94 Analysis model Assumption: D T x is sparse Threshold, then use left inverse of D T L ) ĉ = H η (D T y ) = H η (D T x + D T z ŷ = Lĉ Example: Thresholding STFT coefficients for speech denoising

95 Block thresholding Assumption: Coefficients are group sparse, nonzero coefficients cluster together Block thresholding: Partition coefficients into blocks I 1, I 2,..., I k and threshold whole blocks { x i if i I j such that x Ij 2 > η, B η (x) i := 0 otherwise

96 Haar transform

97 2D wavelet transform

98 Speech denoising

99 Time thresholding

100 Spectrum

101 Frequency thresholding

102 Frequency thresholding Data DFT thresholding

103 Spectrogram (STFT) Frequency Time

104 STFT thresholding Frequency Time

105 STFT thresholding Data STFT thresholding

106 STFT block thresholding Frequency Time

107 STFT block thresholding Data STFT block thresh.

108 Wavelets

109 Sorted wavelet coefficients

110 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

111 Denoising via l 1 -norm regularized least squares Synthesis sparse model x = Dc where c is sparse We would like to solve minimize c 0 subject to y D c Computationally intractable if D is overcomplete Basis-pursuit denoising ĉ = arg min x R m y D c λ c 1 ˆx = Dĉ

112 Sines and spikes x = Dc c DCT subdictionary Spike subdictionary

113 Denoising via l 1 -norm regularized least squares

114 Denoising via l 1 -norm regularized least squares Signal Estimate

115 Denoising via l 1 -norm regularized least squares Signal Estimate

116 Introduction Linear transforms Frequency representation Short-time Fourier transform (STFT) Wavelets Overcomplete sparse models Denoising The denoising problem Thresholding Synthesis model Analysis model

117 Denoising via l 1 -norm regularized least squares Analysis sparse model We would like to solve D T x minimize subject to is sparse D T x y x 0 Computationally intractable if D is overcomplete Instead, we solve ˆx = arg min y x R x 2 m 2 + λ AT x Significantly more challenging to solve than synthesis formulation 1

118 Total variation Images and some time series tend to be piecewise constant Equivalently: Sparse gradient (or derivative) The total variation of an image I is the l 1 -norm of the horizontal and vertical components of the gradient TV (I ) := x I 1 + y I 1 Equivalent to l 1 -norm regularization with an overcomplete analysis operator Denoising via TV regularization Y Î = arg min Ĩ 2 ) (Ĩ + λ TV Ĩ R n n F

119 Denoising via TV regularization Signal Data

120 Denoising via TV regularization Signal TV reg. (small λ)

121 Denoising via TV regularization Signal TV reg. (medium λ)

122 Denoising via TV regularization Signal TV reg. (large λ)

123 Denoising via TV regularization

124 Denoising via TV regularization

125 Small λ

126 Small λ

127 Medium λ

128 Medium λ

129 Large λ

130 Large λ

131 Denoising via TV regularization Original Noisy Estimate