Machine Learning for Signal Processing Sparse and Overcomplete Representations

Size: px

Start display at page:

Download "Machine Learning for Signal Processing Sparse and Overcomplete Representations"

Clarence Cox
6 years ago
Views:

1 Machine Learning for Signal Processing Sparse and Overcomplete Representations Abelino Jimenez (slides from Bhiksha Raj and Sourish Chaudhuri) Oct 1, 217 1

2 So far Weights Data Basis Data Independent ICA PCA NNMF # basis <= dim X 2

3 So far Can we use the same structure to identify basic units that compose the signal? 3

4 Key Topics in this Lecture Basics Component-based representations Overcomplete and Sparse Representations, Dictionaries Pursuit Algorithms How to learn a dictionary Why is an overcomplete representation powerful? 4

5 Representing Data Dictionary (codebook) 5

6 Representing Data Dictionary Atoms 6

7 Representing Data Dictionary Atoms Each atom is a basic unit that can be used to compose larger units. 7

8 Representing Data Atoms 8

9 Representing Data Atoms 9

10 Representing Data 1

11 Representing Data 11

12 Representing Data 12

13 Representing Data

14 Representing Data w 1 w 5 w 2 w 6 w 3 w 7 w 4. 14

15 Representing Data Linear combination of elements in the Dictionary = 15

16 Representing Data Linear combination of elements in the Dictionary = = 16

17 Representing Data Linear combination of elements in the Dictionary = = 17

18 Quick Linear Algebra Refresher Remember, #(Basis Vectors)= #unknowns Basis Vectors (from Dictionary) Weights Input data 18

19 Overcomplete Representations What is the dimensionality of the input image? (say 64x64 image) 496 What is the dimensionality of the dictionary? (each image = 64x64 pixels) 496 x N 19

20 Overcomplete Representations What is the dimensionality of the input image? (say 64x64 image) 496 What is the dimensionality of the dictionary? (each image = 64x64 pixels) 496 x N??? 2

21 Overcomplete Representations What is the dimensionality of the input image? (say 64x64 image) 496 What is the dimensionality of the dictionary? (each image = 64x64 pixels) 496 x N VERY LARGE!!! 21

22 Overcomplete Representations What is the dimensionality of the input image? (say 64x64 image) If N > 496 (as it likely is) we have 496 an overcomplete representation What is the dimensionality of the dictionary? (each image = 64x64 pixels) 496 x N VERY LARGE!!! 22

23 Overcomplete Representations What is the dimensionality of the input image? (say 64x64 image) More generally: 496 If #(basis vectors) > dimensions of input What is the dimensionality of the dictionary? we have an overcomplete representation (each image = 64x64 pixels) 496 x N VERY LARGE!!! 23

24 Quick Linear Algebra Refresher Remember, #(Basis Vectors)= #unknowns Basis Vectors (from Dictionary) Weights Input data 24

25 Dictionary based Representations Overcomplete dictionary -based representations are composition-based representations with more bases than the dimensionality of the data D α X = Bases matrix is wide (more bases than dimensions) Input

26 Why Dictionary-based Representations? Dictionary based representations are semantically more meaningful Enable content-based description Bases can capture entire structures in data E.g. notes in music E.g. image structures (such as faces) in images Enable content-based processing Reconstructing, separating, denoising, manipulating speech/music signals Coding, compression, etc. Statistical reasons: We will get to that shortly.. 26

27 Problems How to obtain the dictionary Which will give us meaningful representations How to compute the weights? D α X = Input

28 Problems How to obtain the dictionary Which will give us meaningful representations How to compute the weights? D α X = Input

29 Quick Linear Algebra Refresher Remember, #(Basis Vectors)= #unknowns Basis Vectors Weights Input data When can we solve for α? 29

30 Quick Linear Algebra Refresher = Unique solution D: full rank = We may have no exact solution = Infinite Solutions 3

31 Quick Linear Algebra Refresher = Unique solution D: full rank = = We may have no exact solution Our Case Infinite Solutions 31

32 Using Pseudo-Inverse? 32

33 Overcomplete Representation Unknown D α X = #(Basis Vectors) > dimensions of the input 33

34 Representing Data Using bases that we know But no more than k=4 bases 34

35 Overcompleteness and Sparsity To solve an overcomplete system of the type: Make assumptions about the data. Suppose, we say that X is composed of no more than a fixed number (k) of bases from D (k dim(x)) The term bases is an abuse of terminology.. Now, we can find the set of k bases that best fit the data point, X. 35

36 Overcompleteness and Sparsity Atoms But no more than k=4 bases are active 36

37 Overcompleteness and Sparsity Atoms But no more than k=4 bases 37

38 No more than 4 bases α D = X 38

39 No more than 4 bases ONLY THE a COMPONENTS CORRESPONDING TO THE 4 KEY D DICTIONARY ENTRIES ARE NON-ZERO α = X 39

40 No more than 4 bases ONLY THE a COMPONENTS CORRESPONDING TO THE 4 KEY D DICTIONARY ENTRIES ARE NON-ZERO α = X MOST OF a IS ZERO!! a IS SPARSE 4

41 Sparsity- Definition Sparse representations are representations that account for most or all information of a signal with a linear combination of a small number of atoms. (from: 41

42 The Sparsity Problem We don t really know k You are given a signal X Assuming X was generated using the dictionary, can we find α that generated it? 42

43 The Sparsity Problem We want to use as few basis vectors as possible to do this. Min a a s.t. X Da 43

44 The Sparsity Problem We want to use as few basis vectors as possible to do this. Min a a s.t. X Da Counts the number of nonzero elements in α 44

45 The Sparsity Problem We want to use as few basis vectors as possible to do this Ockham s razor: Choose the simplest explanation invoking the fewest variables Min a a s.t. X Da 45

46 The Sparsity Problem We want to use as few basis vectors as possible to do this. Min a a s.t. X Da How can we solve the above? 46

47 Obtaining Sparse Solutions We will look at 2 algorithms: Matching Pursuit (MP) Basis Pursuit (BP) 47

48 Matching Pursuit (MP) Greedy algorithm Finds an atom in the dictionary that best matches the input signal Remove the weighted value of this atom from the signal Again, find an atom in the dictionary that best matches the remaining signal. Continue till a defined stop condition is satisfied. 48

49 Matching Pursuit Find the dictionary atom that best matches the given signal. Weight = w 1 49

50 Matching Pursuit Remove weighted image to obtain updated signal Find best match for this signal from the dictionary 5

51 Matching Pursuit Find best match for updated signal 51

52 Matching Pursuit Find best match for updated signal Iterate till you reach a stopping condition, norm(residualinputsignal) < threshold 52

53 Matching Pursuit From 53

54 Matching Pursuit Problems??? 54

55 Matching Pursuit Main Problem Computational complexity The entire dictionary has to be searched at every iteration 55

56 Comparing MP and BP Matching Pursuit Hard thresholding Basis Pursuit (remember the equations) Greedy optimization at each step Weights obtained using greedy rules 56

57 Basis Pursuit (BP) Remember, Min a a s.t. X Da 57

58 Basis Pursuit Remember, Min a a s.t. X Da In the general case, this is intractable 58

59 Basis Pursuit Remember, Min a a s.t. X Da In the general case, this is intractable Requires combinatorial optimization 59

60 Basis Pursuit Replace the intractable expression by an expression that is solvable Min a a 1 s.t. X Da 6

61 Basis Pursuit Replace the intractable expression by an expression that is solvable Min a a 1 s.t. X Da This will provide identical solutions when D obeys the Restricted Isometry Property. 61

62 Basis Pursuit Replace the intractable expression by an expression that is solvable Min a a 1 s.t. X Da Objective Constraint 62

63 Basis Pursuit We can formulate the optimization term as: Min a { X Da 2 a 1 } Constraint Objective 63

64 Basis Pursuit We can formulate the optimization term as: Min a { X Da 2 a 1 } λ is a penalty term on the non-zero elements and promotes sparsity 64

65 Basis Pursuit Equivalent to LASSO; for more details, see this paper We can by Tibshirani formulate the optimization term as: Min a { X Da 2 a 1 } λ is a penalty term on the non-zero elements and promotes sparsity 65

66 Basis Pursuit a 1 is not differentiable at a j = Gradient of a 1 for gradient descent update At optimum, following conditions hold 66 { 1 } 2 a a a D X Min at 1 [-1,1]at at 1 1 j j j j a a a a a if, if, ) ( 2 2 j j j j j X sign X a a a a a D D

67 Basis Pursuit There are efficient ways to solve the LASSO formulation. Simplest solution: Coordinate descent algorithms On webpage.. 67

68 L 1 vs L Min a a s. t. X Da X Overcomplete set of 6 bases L minimization Two-sparse solution ANY pair of bases can explain X with error 68

69 L 1 vs L Min a a 1 s. t. X Da X Overcomplete set of 6 bases L 1 minimization Two-sparse solution All else being equal, the two closest bases are chosen 69

70 Comparing MP and BP Matching Pursuit Hard thresholding Basis Pursuit Soft thresholding (remember the equations) Greedy optimization at each step Weights obtained using greedy rules Global optimization Can force N-sparsity with appropriately chosen weights 7

71 General Formalisms L minimization L constrained optimization L 1 minimization Min a Da L 1 constrained optimization a s. t. X Min a a 1 s. t. X Da Min a s. t. a Min a s. t. a X X 1 Da C Da C

72 Many Other Methods.. Iterative Hard Thresholding (IHT) CoSAMP OMP 72

73 Problems How to obtain the dictionary Which will give us meaningful representations How to compute the weights? D α X = Input

74 Trivial Solution D = Training data Impractical and useless 74

75 Dictionaries: Compressive Sensing Just random vectors! D 75

76 More Structured ways of Constructing Dictionaries Dictionary entries must be structurally meaningful Represent true compositional units of data Have already encountered two ways of building dictionaries NMF for non-negative data K-means.. 76

77 K-Means for Composing Dictionaries Train the codebook from training data using K-means Every vector is approximated by the centroid of the cluster it falls into Cluster means are codebook entries Dictionary entries Also compositional units the compose the data 77

78 K-Means for Dictionaries Each column is a codeword (centroid) from the codebook D a must be 1 sparse Only a entry must 1 = 1 1 = 1 α 1 X 78

79 K-Means Learn Codewords to minimize the total squared length of the training vectors from the closest codeword 79

80 Length-unconstrained K-Means for Dictionaries Each column is a codeword (centroid) from the codebook D a must be 1 sparse No restriction on a value = 1 α 3 X 8

81 SVD K-Means Learn Codewords to minimize the total squared projection error of the training vectors from the closest codeword 81

82 1. Initialize a set of centroids randomly 2. For each data point x, find the projection from the centroid for each cluster p cluster SVD K means cluster 3. Put data point in the cluster of the closest centroid Cluster for which p cluster is maximum 4. When all data points are clustered, recompute centroids x T m m cluster Principal Eigenvector({ x x cluster}) 5. If not converged, go back to 2 82

83 Problem Only represents Radial patterns 83

84 What about this pattern? Dictionary entries that represent radial patterns will not capture this structure 1-sparse representations will not do 84

85 What about this pattern? We need AFFINE patterns 85

86 What about this pattern? We need AFFINE patterns Each vector is modeled by a linear combination of K (here 2) bases 86

87 What about this pattern? Every line is a (constrined) combination of two bases 2-sparse Constraint: Line = a.b 1 + (1-a)b 2 We need AFFINE patterns Each vector is modeled by a linear combination of K (here 2) bases 87

88 Codebooks for K sparsity? Each column is a codeword (centroid) from the codebook D a must be k sparse No restriction on a value = k α X 88

89 Formalizing Given training data We want to find a dictionary D, such that With sparse 89

90 Formalizing Two objectives: - Approximation - Sparsity in coefficients NON-Convex!!! 9

91 An iterative method Given D, estimate We can use any method to get sparse solution Given, estimate D Difficult! 91

92 K SVD Initialize Codebook D = 1. For every vector, compute K-sparse alphas Using any pursuit algorithm a =

93 K-SVD D j j!= 1 2. For each codeword (k): For each vector x Subtract the contribution of all other codewords to obtain e k (x) Codeword-specific residual Compute the principal Eigen vector of {e k (x)} 3. Return to step 1 D = a = a j (x) j!= e k x = x-σ j k j D j 93

94 K-SVD D Termination of each iteration: Updated dictionary Conclusion: A dictionary where any data vector can be composed of at most K dictionary entries More generally, sparse composition 94

95 Problems How to obtain the dictionary Which will give us meaningful representations How to compute the weights? D α X = Input

96 Applications of Sparse Representations Many many applications Signal representation Statistical modelling.. We ve seen one: Compressive sensing Another popular use Denoising 97

97 Denoising As the name suggests, remove noise! 98

98 A toy example 99

99 A toy example Identity matrix Translation of a Gaussian pulse 1

100 Image Denoising Here s what we want 11

101 Image Denoising Here s what we want 12

102 Image Denoising Here s what we want 13

103 The Image Denoising Problem Given an image Remove Gaussian additive noise from it 14

104 Image Denoising Noisy Input Orig. Image Y = X + ε ε = Ν(, σ) Gaussian Noise 15

105 Image Denoising Remove the noise from Y, to obtain X as best as possible. 16

106 Image Denoising Remove the noise from Y, to obtain X as best as possible Using sparse representations over learned dictionaries 17

107 Image Denoising Remove the noise from Y, to obtain X as best as possible Using sparse representations over learned dictionaries We will learn the dictionaries 18

108 Image Denoising Remove the noise from Y, to obtain X as best as possible Using sparse representations over learned dictionaries We will learn the dictionaries What data will we use? The corrupted image itself! 19

109 Image Denoising We use the data to be denoised to learn the dictionary. Training and denoising become an iterated process. We use image patches of size n x n pixels (i.e. if the image is 64x64, patches are 8x8) 11

110 Image Denoising The data dictionary D Size = n x k (k > n) This is known and fixed, to start with Every image patch can be sparsely represented using D 111

111 Image Denoising Recall our equations from before. We want to find α so as to minimize the value of the equation below: Min a { X Da 2 a } Min{ X Da a } 1 a 2 112

112 Image Denoising Min a { X Da 2 a 1 } In the above, X is a patch. 113

113 Image Denoising Min a { X Da 2 a 1 } In the above, X is a patch. If the larger image is fully expressed by the every patch in it, how can we go from patches to the image? 114

114 Image Denoising Min { X Y 2 a ij,x 2 ij R ij X Da ij 2 2 } ij a ij ij 115

115 Image Denoising Min { X Y 2 a ij,x 2 ij R ij X Da ij 2 2 } ij a ij ij (X Y) is the error between the input and denoised image. μ is a penalty on the error. 116

116 Image Denoising Min { X Y 2 a ij,x 2 ij R ij X Da ij 2 2 } ij a ij ij Error bounding in each patch -R ij selects the (ij) th patch -Terms in summation = no. of patches 117

117 Image Denoising Min { X Y 2 a ij,x 2 ij R ij X Da ij 2 2 } ij a ij ij λ forces sparsity 118

118 Image Denoising But, we don t know our dictionary D. We want to estimate D as well. 119

119 Image Denoising But, we don t know our dictionary D. We want to estimate D as well. Min { X Y 2 D,a ij,x 2 } ij a ij ij ij R ij X Da ij 2 We can use the previous equation itself!!! 2 12

120 Image Denoising Min { X Y 2 D,a ij,x 2 ij R ij X Da ij 2 2 } ij a ij ij How do we estimate all 3 at once? 121

121 Image Denoising Min { X Y 2 D,a ij,x 2 ij R ij X Da ij 2 2 } ij a ij ij How do we estimate all 3 at once? We cannot estimate them at the same time! 122

122 Image Denoising Min { X Y 2 D,a ij,x 2 ij R ij X Da ij 2 2 } ij a ij ij How do we estimate all 3 at once? Fix 2, and find the optimal 3 rd. 123

123 Image Denoising Min { X Y 2 D,a ij,x 2 ij R ij X Da ij 2 2 } ij a ij ij Initialize X = Y 124

124 Image Denoising 2 Min { X Y a 2 ij } ij a ij ij ij R ij X Da ij 2 2 Initialize X = Y, initialize D You know how to solve the remaining portion for α MP, BP! 125

125 Image Denoising Now, update the dictionary D. Update D one column at a time, following the K-SVD algorithm K-SVD maintains the sparsity structure 126

126 Image Denoising Now, update the dictionary D. Update D one column at a time, following the K-SVD algorithm K-SVD maintains the sparsity structure Iteratively update α and D 127

127 Image Denoising Learned Dictionary for Face Image denoising From: M. Elad and M. Aharon, Image denoising via learned dictionaries and sparse representation, CVPR,

128 Image Denoising Min X { X Y 2 2 ij R ij X Da ij 2 2 } ij a ij ij Const. wrt X We know D and α The quadratic term above has a closedform solution 129

129 Image Denoising Min X { X Y 2 2 ij R ij X Da ij 2 2 } ij a ij ij Const. wrt X We know D and α X ( I R T R ) 1 ( Y ij ij ij ij R T ij D a ij ) 13

130 Image Denoising Summarizing We wanted to obtain 3 things 131

131 Image Denoising Summarizing We wanted to obtain 3 things Weights α Dictionary D Denoised Image X 132

132 Image Denoising Summarizing We wanted to obtain 3 things Weights α Your favorite pursuit algorithm Dictionary D Using K-SVD Denoised Image X 133

133 Image Denoising Summarizing We wanted to obtain 3 things Weights α Your favorite pursuit algorithm Dictionary D Using K-SVD Denoised Image X Iterating 134

134 Image Denoising Summarizing We wanted to obtain 3 things Weights α Dictionary D Denoised Image X- Closed form solution 135

135 Image Denoising Here s what we want 136

136 Image Denoising Here s what we want 137

137 Comparing to Other Techniques Non-Gaussian data PCA of ICA Which is which? Images from Lewicki and Sejnowski, Learning Overcomplete Representations,

138 Comparing to Other Techniques Non-Gaussian data PCA ICA Images from Lewicki and Sejnowski, Learning Overcomplete Representations,

139 Comparing to Other Techniques Predicts data here Non-Gaussian data Doesn t predict data here Does pretty well PCA ICA Images from Lewicki and Sejnowski, Learning Overcomplete Representations, 2. 14

140 Comparing to Other Techniques Data still in 2-D space ICA Overcomplete Doesn t capture the underlying representation, which Overcomplete representations can do 141

141 Summary Overcomplete representations can be more powerful than component analysis techniques. Dictionary can be learned from data. Relative advantages and disadvantages of the pursuit algorithms. 142

Machine Learning for Signal Processing Sparse and Overcomplete Representations. Bhiksha Raj (slides from Sourish Chaudhuri) Oct 22, 2013

Machine Learning for Signal Processing Sparse and Overcomplete Representations Bhiksha Raj (slides from Sourish Chaudhuri) Oct 22, 2013 1 Key Topics in this Lecture Basics Component-based representations