Introduction to Sparsity. Xudong Cao, Jake Dreamtree & Jerry 04/05/2012

Size: px

Start display at page:

Download "Introduction to Sparsity. Xudong Cao, Jake Dreamtree & Jerry 04/05/2012"

Ami Gilmore
5 years ago
Views:

1 Introduction to Sparsity Xudong Cao, Jake Dreamtree & Jerry 04/05/2012

2 Outline Understanding Sparsity Total variation Compressed sensing(definition) Exact recovery with sparse prior(l 0 ) l 1 relaxation Uncertainty principle & exact recovery Numerical Solution Least angle regression (LAR) Coordinate descent K-SVD for dictionary learning

A Encouraging and Mystical observation (a) The Logan-Shepp phantom test image. (b) Sampling domain in the frequency plane; Fourier coefficients are sampled along 22 approximately radial lines.

3 A Encouraging and Mystical observation (a) The Logan-Shepp phantom test image. (b) Sampling domain in the frequency plane; Fourier coefficients are sampled along 22 approximately radial lines. (c) Minimum energy reconstruction obtained by setting unobserved Fourier coefficients to zero (d) Reconstruction obtained by minimizing the total variation Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information

4 Exact recovery by total variation Min I TV The image to recover s. t. I(ω) = f(ω), ω Ω the observation in frequency domain I TV = x1 I x 1, x x2 I x 1, x 2 2 x 1 x 2 Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information

5 Outline Understanding Sparsity Total variation Compressed sensing(definition) Exact recovery with sparse prior(l 0 ) l 1 relaxation Uncertainty principle & exact recovery Numerical Solution Orthogonal matching pursuit(omp) Least angle regression (LAR) Coordinate descent K-SVD for dictionary learning

6 Linear equations A x = y m n matrix Generally speaking (not strictly hold) m=n unique solution m>n no solution (least square solution) m<n infinite solutions A A A

7 Compressed Sensing y: the observation x: the signal to recover A: sensing matrix (DFT, RP etc.) A Complete sensing A x = y A Over-complete sensing A Compressed sensing Exactly recover x by compressed sensing? No -- generally speaking Yes -- if x is sparse and

8 Outline Understanding Sparsity Total variation Compressed sensing(definition) Exact recovery with sparse prior(l 0 ) l 1 relaxation Uncertainty principle & exact recovery Numerical Solution Least angle regression (LAR) Coordinate descent K-SVD for dictionary learning

9 Exact recovery Min x 0 s. t. Ax = y

10 Conditions for exact recovery(1) Spark Definition: Given a matrix A, =Spark{A} is the smallest number of columns that are linearly dependent. Example: Donoho & Elad (02) A= Spark(A) = 3 Sparse & Redundant Signal Representation and its Role in Image Processing

11 Conditions for exact recovery(2) Uniqueness Rule P: Min x 0 s. t. Ax = y Optimization problem P can exactly recover x, as long as Spark(A) > 2 x 0 Sparse & Redundant Signal Representation and its Role in Image Processing (Yonina C. Eldar) Introduction to compressed sensing (Yonina C. Eldar)

12 Conditions for exact recovery(3) Uniqueness Rule Equivalent representation: Spark(A) > 2k at most one x satifies Ax = y and x 0 = k Introduction to compressed sensing (Yonina C. Eldar) P18

13 Example Condition Spark(A) > 2k can t be relaxed to Spark(A) 2k Example A = , y = T 1 1 Spark(A)= There are two different x satisfy Ax = y x 1 = T x 2 = T Contributed by Jerry

14 Conditions for exact recovery(4) Proof Target: Spark(A) > 2k at most one x satifies Ax = y and x 0 = k Proof: 1. Suppose the right side is not hold, i.e. Ax 1 = y & x 1 0 = k Ax 2 = y & x 2 0 = k 2. The difference of the above two equations Ah = 0 & h 0 2k, where h = x 1 x 2 3. According to the definition of matrix spark Spark A 2k Introduction to compressed sensing (Yonina C. Eldar) P18

15 Conditions for exact recovery(5) Proof Target: Spark(A) > 2k at most one x satifies Ax = y and x 0 = k Proof: 1. Suppose the left side is not hold, i.e. Spark A 2k 2. According to the definition of matrix spark Ah = 0 & h 0 2k 3. It is easy to construct x 1 and x 2 satisfy the following constraints h = x 1 x 2, x 1 0 = k, x 2 0 = k, x 1 x 2 4. Hence there are two different satisfy Ax = y and x 0 = k, i.e. Ax 1 = Ax 2 = y, x 1 0 = k, x 2 0 = k Introduction to compressed sensing (Yonina C. Eldar) P18

16 Outline Understanding Sparsity Total variation Compressed sensing(definition) Exact recovery with sparse prior(l 0 ) l 1 relaxation Uncertainty principle & exact recovery Numerical Solution Least angle regression (LAR) Coordinate descent K-SVD for dictionary learning

17 l 1 relaxation P: Min x 0 s. t. Ax = y Optimization problem P is NP hard Two way to address this issue Approximation (OMP) Equivalent relaxation (l 1 ) P : Min x 1 s. t. Ax = y Convex!

18 Heuristic Explanation(1) Unit Ball of p-norm

19 Heuristic Explanation(2) Geometric View of the optimization P : Min x p s. t. Ax = y Find the minimum p-norm ball tangent to plane Ax = y The tangent point is the optimal solution Example: Ax = y

20 Heuristic Explanation(3) insights The optimal point is sparse when p 1

21 Restricted isometry property(rip) : Explanation: The l 2 norm should be approximately preserved after the transform (isometry) Hence any k columns of matrix A should be approximately orthogonal The restriction become stronger as δ k approaching to 0 With strong RIP restriction, l 1 relaxation exactly recover x Introduction to compressed sensing (Yonina C. Eldar) P20 & P31

22 Different formulations Min x 1 s. t. Ax y 2 ε Min Ax y 2 s. t. x 1 m Min Ax y 2 + λ x 1 Lasso

23 Outline Understanding Sparsity Total variation Compressed sensing(definition) Exact recovery with sparse prior(l 0 ) l 1 relaxation Uncertainty principle & exact recovery Numerical Solution Least angle regression (LAR) Coordinate descent K-SVD for dictionary learning

Uncertainty Principles Heisenberg uncertainty principle: A stronger uncertainty principle by Candes and Tao with tighter bound: http://terrytao.

24 Uncertainty Principles Heisenberg uncertainty principle: A stronger uncertainty principle by Candes and Tao with tighter bound: 不确定原理的前世今生 ( 完 )by 木遥

25 Exact Recovery & Uncertainty Principle 不确定原理的前世今生 ( 完 )by 木遥

26 An intuitive interpretation of uncertainty principle Contributed by jake

27 Any function can be viewed as a vector Consider a function f(x), whose domain is {0,1} Then, f(x) can be expressed as (f(0),f(1)) which is a vector in two dimensional space, and the functional values are the coordinates (following figure). So any combinations of f(0) and f(1) is a vector f(1) f(0)

28 Integral Transformation Through this point of view, any integral transformation between two functional spaces, say f(x)= g(w)k(x,w)dw can be viewed as a coordinate transformation because the transformation can be written as the following form (integration should be viewed as summation): Then, the two functions f(x) and g(w) are connected by the coordinate transformation The kernel function k(x,w) is actually a matrix K to represent the transformation ) ( ), ( ) ( w g w x k x f

29 Orthogonal Transform and Rotation If the transformation (kernel function) k(x,w) is orthogonal, that is: KK T * =I, or k( x, w) k ( w, x') dw ( x x') (T is conjugate transpose, * is the conjugate operator) Then K is a rotational transform For example, Fourier transform is a rotational transform, where k(x,w)=exp(-ixw) For any rotational transform, we know all the vectors keep their modules under the transformation, so we have parseval theorem 2 f ( x) g( w) 2

30 In two dimensional space Suppose a functional transform U f(x)=(f(0),f(1)) and g(w)=(g(0),g(1)) satisfy the following equation: g(0) u g(1) u f (0) u f (0) u f (1) U f (1) Accordingly, Parsevall theorem just states g(0) 2 +g(1) 2 =f(0) 2 +f(1) f f (0) (1) f(1) g(1) g(0) f(0)

31 Uncertainty Principle Suppose U is a rotational transform, then the evenness of one function is connected with the other function Consider the two extreme cases In g representation, the distribution is (1,0) which is of a maximum certainty. However, under f representation, it may be (Sqrt(2)/2,Sqrt(2)/2) which is the most uncertain distribution g(1) f(1) g(0) f(0)

32 Uncertainty Principle Suppose U is a rotational transform, then the evenness of one function is connected with the other function Consider the two extreme cases In g representation, the distribution is (sqrt(2)/2,sqrt(2)/2) which is of a maximum uncertainty. However, under f representation, it may be (0,1) which is the most uncertain distribution g(1) f(1) g(0) f(0)

33 Uncertainty Principle Generally, we can define the uncertainty of f(x) as f x 2 f ( x) dx xf ( x) dx 2 Then, generally, the uncertainty principle is fg h(u) Where, h(u) is a function of the transform of u For Fourier transformation fg 1

34 Uncertainty Principle in Quantum Mechanics In quantum mechanics, f(x) is the probability amplitude of position distribution. That is, f(x)f*(x) is the probability of measuring a particle at x. g(p) is the probability amplitude of momentum distribution. Heisenberg supposes that f(x) and g(p) can be connected by Fourier transformation, therefore, the uncertainty principle is hold for these two measurements

35 Discussion Total variation revisiting

36 Outline Understanding Sparsity Total variation Compressed sensing(definition) Exact recovery with sparse prior(l 0 ) Uncertainty principle & exact recovery l 1 relaxation Numerical Solution Least angle regression (LAR) Coordinate descent K-SVD for dictionary learning

37 Different view point - Regression The angle of previous discussion Signal processing Compression & Recovery y = A x y the variable we want to predict (temperature tomorrow) A represents the observed variables(temperature today, historical temperature, etc.) x is the linear coefficients describing the dependence between x and y Each row is a training sample

38 Subset Selection for Regression What is subset selection? Selecting the best subset of observed variables What is the specific meaning of best? Why? Accuracy High correlation between observed variables Irrelevant observed variables Interpretation New framework for scientific research Data driven discovering interesting phenomenon Theoretical reasoning The elements of statistical learning P57

39 Forward stepwise subset selection Selecting the best subset NP hard Forward stepwise selection Greedy algorithm(computational efficiency) Add one variable into the active set each iteration Criterions for adding variable Correlation with the residual L2 loss after adding this variable F statistics

40 Least Angle Regression Linear regression Forward stepwise selection Criterions for adding variable Approximation to Lasso Min Ax y 2 + λ x 1

41 Algorithm(1) Preprocessing Zero mean and unit norm for each column of A Step 1 Selecting the variable which has the largest inner product with y The elements of statistical learning P74

42 Algorithm(2) Step k There are k variables in the active set x k is the coefficients of the k variables The coefficients of k-1 variables are non-zero The coefficient of the new comer is zero Let X k = 1 α x k + α A k T A k 1 Ak T y Y T R 0, α 1 The inner product between the prediction and residual keep decreasing when α is approaching 1 Y = A k X k R = y A k X k

43 Algorithm(3) Step k Criterions for adding variable Increasing α to α when a variable outside of active set satisfy Y T R = x j R X k = 1 α x k + α A T k A k 1 T Ak y Step k+1 x k = [X k ; 0] Add x j into the active set Do the same thing in step k

44 Interpretation of LAR from geometric view Contributed by Wang Fan

45 x 2 y x 1 Assume that we have two basis {x 1, x 2 } and the observation is y in the space of L(x 1, x 2 ).

46 x 2 y y w 1 x 1 (u 1 ) We begin by select the most correlated basis, i.e. x 1 in this case. denote u 1 as the direction to approach, which in this case is the same direction of x 1. And w 1 represent the coefficient of u 1, and y represent the residual.

47 x 2 (x 2 ) y θ θ y w 1 x 1 (u 1 ) As w 1 increase, the y changes at mean time. And w 1 stops when y has the same correlation with u 1 and one of the other basis, in this case x 2, which means that y bisect the angle between u 1 and x 2.

48 x 2 (x 2 ) y u 2 θ θ y* w 1 x 1 (u 1 ) As w 1 increase, the y changes at mean time. And w 1 stops when y = y* has the same correlation with u 1 and one of the other basis, in this case x 2, which means that y* bisect the angle between u 1 and x 2. And the next direction u2 is made the direction of y*.

49 x 2 (x 2 ) y u 2 θ θ w 2 w 1 x 1 (u 1 ) And use the same strategy for the coefficient of u 2, which is w 2

50 (x 3 ) x 2 (x 2 ) u 3 x 3 w 3 u 2 w 1 w 2 x 1 (u 1 ) Assume that y is in a higher dimensional space, and all the sequences could be roughly explained in this feature.

51 Connection with lasso(1) The elements of statistical learning P75 Similar, slightly different

52 Connection with lasso(2) LAR is a greedy way for solving lasso Efficiently compute the whole solution path If the coefficients do not change sign, the solution is accurate With slight revision it will be an accurate algorithm for lasso The elements of statistical learning P 74-76

53 Outline Understanding Sparsity Total variation Compressed sensing(definition) Exact recovery with sparse prior(l 0 ) Uncertainty principle & exact recovery l 1 relaxation Numerical Solution Least angle regression (LAR) Coordinate descent K-SVD for dictionary learning

54 Coordinate descent Coordinate descent optimize each parameter separately, holding all the others fixed

55 Coordinate descent for lasso Preprocessing zero mean and unit norm for each column of A For each iteration Let r j = y A jc x jc A jc is the matrix A without j th column x jc is the coefficient x without j th element Solving r j = A j x j x j = A j T r j A j T A j = A j T r j (unit norm) Soft thresholding x j = sign(x j ) Max(0, x j λ) Tricks for fast implementation Fast Regularization Paths via Coordinate Descent P (ppt)

56 Comparison with LAR

57 Outline Understanding Sparsity Total variation Compressed sensing(definition) Exact recovery with sparse prior(l 0 ) Uncertainty principle & exact recovery l 1 relaxation Numerical Solution Least angle regression (LAR) Coordinate descent K-SVD for dictionary learning

58 Linear representation y=ax DFT, PCA A is the dictionary

59 Dictionary learning Min Ax i y i 2 + λ x i 1 i Both A and x i are variables to be optimized Non-convex Alternative optimization is adopted compute x i while A fixed (sparse coding stage) compute A while x i fixed (dictionary update stage)

60 K SVD for Dictionary(1) Original representation Min Ax i y i 2 + λ x i 1 i compact representation Min AX Y 2 + λ X 1 The objective function Min AX Y 2 X is fixed In the dictionary update stage Only dictionary A is the variable

61 K SVD for Dictionary(2) Similar to Coordinate descent (not the same) One column of the dictionary A j is updated Corresponding coefficient X j is updated A j X j (Y A jc X jc ) 2 A j X j E 2 E = Y A jc X jc A jc is the matrix A without j th column X jc is the matrix X with j th row A j and X j are simultaneously solved by SVD The complexity for rank one matrix SVD decomposition is O (n 2 ) instead of O (n 3 ) Eigs in matlab K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation

62 K SVD for Dictionary(3) One issue: SVD does NOT keep the sparsity of X j Solution: SVD without the columns corresponding to zero coefficients A j X j E 2

63 Thanks

Introduction to Compressed Sensing

Introduction to Compressed Sensing Alejandro Parada, Gonzalo Arce University of Delaware August 25, 2016 Motivation: Classical Sampling 1 Motivation: Classical Sampling Issues Some applications Radar Spectral