User s Guide to Compressive Sensing

Transcription

1 A Walter B. Richardson, Jr. University of Texas at San Antonio Engineering November 18, 2011

2 Abstract During the past decade, Donoho, Candes, and others have developed a framework for representing signals/images that are known to be sparse in some basis. In their approach, data compression takes place during acquisition rather than as an afterthought. In this talk, I will give a general introduction to compressive sensing with applications to problems in computerized tomography. The intended audience will be graduate students and undergraduates having a strong background in linear algebra. Extending the definition of p-norm to the range 0 < p < 1, we show how standard variational problems from linear algebra, such as least squares minimum L 2 norm solutions to Ax = b, transform, in say the L 1 -norm, in such as way as to favor sparse solutions.

3 Outline 1 Variational Formulation of Linear Algebra Problems 2 1-Norm and Quasi-Norms (0 < p < 1) favor sparsity. 3 Underlying assumption of Compressive Sensing: signal/image sparse in some basis. 4 Information Recovery and Computational Complexity 5 Least Squares Minimum Norm Solution of Ax = b. 6 Sample Results in 1 D and 2 D.

4 Variational Problems in Engineering In many areas of engineering and mathematics, various physical and geometrical quantities are given by quotients of the form Q below. u 2 Ω Q = min ( ) u H0 1(Ω) 2/p u p Ω If p = 2, then Q is the principal Dirichlet eigenvalue If p = 1 and n = 2, then S = 4/Q is the torsional rigidity of a cylindrical beam of cross section Ω If p = 2n n 2, then Q is the best Sobolev constant S n.

5 Constrained Optimization as Regularization Perhaps surprisingly, many classical problems in linear algebra can be recast in a variational formulation, allowing calculus to be used for their solution. Consider a simple problem: given a function f (x, y) find where the min/max values of f occur subject to a constraint defined by a second function g(x, y) taking on the value K. min / max f (x, y) subject to g(x, y) = K Without constraint, the problem is straightforward; take the first derivative and set to zero to find the critical or stationary points. Theory of Lagrange multipliers says for constrained problem, f (x) = λ g(x) at critical point.

6 Constrained Optimization Using L 2 norm. Often a vector norm enters either in objective functional f or constraint g. Consider linear functional f (x, y) = 2x + 1y; solve optimization problem f = max { f (v) : v = 1}.

7 Inner Products The Euclidean norm is special because it arises from an inner product. This is true only in case the norm satifies the parallelogram law: x + y 2 + x y 2 = 2 x y 2. An inner product on C n is a function, satisfying: 1 x, x 0, with equality only when x is the null vector 2 x + y, z = x, z + y, z 3 αx, y = α x, y 4 y, x = x, y Besides the standard inner product on IR m or its complex analogue, the following inner products on the space of real valued functions on a domain Ω are useful. f (x)g(x) dx f (x)g(x) + f (x) g(x) dx Ω Ω

8 The Projection Theorem Theorem If S is a closed, convex subset of a Hilbert space {H,, } and b is a point of H, then there is a unique point w of S which is closest to b. Theorem If L is a (closed) subspace of an inner product space {H,, } and b H, then each two of the following statements are equivalent for a point w in L: (i) for every x in L, b w b x (ii) for every x in L, b w, x = 0 (iii) if {q j } j is an orthonormal basis of L, then w = j q j, b q j Note that subspaces ( are necessarily convex and that Part (iii) can ) be rewritten as w = j q jq T j b

9 Inner Products and Orthogonal Projections

10 Application of Euclidean Norms: Least Squares Given an m n rectangular matrix A, what do we mean by: solve Ax = b? May be no solution in the classical sense, but we can minimize the norm of the residual r = Ax b. Define objective functional f via f (x) = 1 2 Ax b 2 2 f (x) = A T (Ax b) Clearly, f is bounded below, setting gradient equal to zero, a minimizer satisfies the normal equations, A T Ax = A T b. Now there is always a solution: unique if A has linearly independent columns; if not, we can always choose the one of minimum Euclidean norm.

11 Least Squares Pros & Cons 1 Advantages: 1 generality & convenience 2 symmetric form even for non-self adjoint PDE s 3 less sensitive to changes in PDE type (transonic flow) 4 ease of error evaluation 5 mixed least squares finite elements without restrictive inf-sup condition of Galerkin mixed methods 2 Disadvantages: 1 ill-conditioning - e.g. normal equations for Au = b 2 degradation of iterative convergence 3 subtle scaling issues 4 can converge to a wrong solution

12 Vector Norms A vector norm on IR n is a function satisfying: 1 x 0, with equality only when x is the null vector 2 αx = α x 3 x + y x + y (triangle inequality) 1/p n For x IR n, define x p = x j p. j=1 If p 1 this is a norm: special cases are p = 2 (Euclidean), p = 1 (l 1 norm), and sup or norm, limiting case as p. For 0 < p < 1, this is a quasi-norm; triangle inequality is replaced by x + y C( x + y ). Note 0 < p 1 are important for new applications with sparsity.

13 Constrained Optimization Using p-norms

14 Minimax Characterization of Eigenvalues Theorem Suppose A is a self-adjoint compact linear operator. If the positive eigenvalues λ + k are ordered in decreasing order with multiplicities repeated, then λ + k = min max Ax, x / x 2 x π k 1 π k 1 Where π k 1 denotes a linear subspace of H of codimension k 1. Alternatively, λ + k = max min Ax, x / x 2 L k x L k Where now L k denotes a linear subspace of H of dimension k.

15 Eigenvalues - Extremal Values

16 Induced Matrix Norms m n matrix A is transformation from IR n into IR m ; choose a vector norm in the domain and in the range space. Maximize Ax over unit sphere: A (m,n) = sup Ax IR m subject to x IR n = 1.

17 Singular Value Decomposition Geometry - image of unit sphere under any m n (rectangular) matrix A is hyperellipse. Optimization problem: max f (x) = Ax 2 = Ax, Ax = A H Ax, x s.t. x 2 = 1 Use Spectral Theorem & A H A is Hermitian to prove: Theorem Every matrix A in C m n can be factored as A = U Σ V H (unitary)(diagonal)(unitary). Columns of U (m m) are eigenvectors of AA H ; columns of V (n n) are eigenvectors of A H A. If r = rank(a), then r singular values on the diagonal of Σ (m n) are the square roots of the eigenvalues of both AA H and A H A.

18 Exercise: Generalized Inverse via SVD Consider the problem of finding the minimum norm least squares (MNLS) solution to Ax = b for the 3 4 matrix A and right hand side vector b σ A = 0 σ 2 0 0, b = b 1 b 2 b 3 A = 1/σ /σ where σ 1, σ 2 are positive. (Note that both the rows and the columns of A are linearly dependent.) For a matrix in the above form it is clear how to find the projection of b onto the range of A and then among all solutions of Ax = Pb find the one of minimum norm x. Show the latter can be obtained by applying A above to the vector b. Show how the SVD can be used to achieve a MNLS solution for a general matrix A and define A, the generalized (Moore-Pembrose) inverse of A.

19 Compression Using SVD The SVD factorization can be be used to compress an image. Recall that a matrix product can be rewritten as sum of outer r products, i.e. A = U Σ V = σ j u j v j where r = rank(a). j=1 Simply discard terms with σ j < ɛ.

20 u + v Models in Image Processing Consider the following variational problem. Let H be a Hilbert space and V : H IR {+ } a lower semicontinuous, strictly convex functional. A global minimizer exists and is unique. Yosida introduced the following regularization V λ (x), λ > 0 of V (x): V λ (x) = inf{v (y) + λ x y 2 : y H} These V λ (x) in essence smooth the functional V (x). Special case is Osher-Rudin model in image processing. Given image f (x) L 2 (IR 2 ) minimize inf{j(u) : f = u + v} J(u) = u BV + λ v 2 2 where infimum is computed over all possible decompositions of f into the sum of a function u in BV (IR 2 ) and a function v in L 2 (IR 2 ).

21 Variational Image Decomposition Consider splitting (Meyer) an image f up into components f = u + v, where u is a smoothed version of f and v represents texture+noise. A variational formulation for the image model is J(u, λ) = 1 p u p + λ 1 q v q p = 2, q = 2 Heat equation p = 1, q = 2 Rudin-Osher-Fatemi TV L 2 model p = 1, q = 1 Chan-Vese TV L 1 preserves contrast & geometry LSFEM with Sobolev gradients would suggest negative, even fractional, norms useful. Connection to real interpolation spaces via K-functional ( K(t, u) = inf u v 2 Y + t 2 v 2 ) 1 2 X v X

22 Binary Decomposition Decompose f = u + v, where u H 1 and v L 2. Minimize min u H 1 J(u) = u 2 H 1 + λ f u 2 H 0 with zero Neumann boundary conditions as most reasonable. J (u)h = 2(u, h) + 2( u, h) + 2λ(f u, h), Euler-Lagrange equation is Helmholtz u (1 + λ)u = λf. Below is the f = u + v decomposition.

23 Varying the Scale Parameter λ Weighting parameter λ controls which details are to be removed from the smoothed image. Below are u, v decompositions for λ = 0.1, 0.4, 0.7.

24 Negative Norm Ternary Decomposition Shen, Osher et. al. decompose f = u + v + w, where u H 1,v H 1, w L 2. Minimize min u H 1 J(u, v) = u 2 H 1 + λ 1 v 2 H 1 + λ 2 f (u + v) 2 H 0 Below original image at left, followed by u, v, w decomposition.

25 Negative Sobolev Norms and Sobolev Gradients Mumford and Meyer: model images by distributions rather than just functions. Meyer proposed the Besov space B 1, for modeling textures. (Osher, Sole, Vese, Shen, Chan, Bertozzi, others) Use of the H 1 norm for finite element least-squares methods to obtain better error estimates and coercivity constands. (Carey, Pehlivanov, Bramble, Pasiak, Bochev, Gunzberger, McCormick, Manteuffel) Sobolev Gradients for general pdes (transonic flow, Ginzburg-Landau, minimal surface) and inverse problems Gradient descent in H 1 rather than in H 0 L 2. Precondition with the inverse Laplacian. (Neuberger, Richardson, Renka, Knowles, Mahavier)

26 Wavelet Shrinkage and u + v Modells David Donoho and Ian Johnstone, Wavelet Shrinkage: Asymptompia?, J. R. Statist. Soc., 1995, 57, No. 2, pp Yves Meyer, Oscillating Patterns in Image Processing and Nonlinear Evolution Equations. Sort wavelet coefficients - the first N coefficients provide a denoised image and also optimal nonlinear approximation. Denoising algorithms deal with u + v models: u(x) is an unknown function in BV satisfying u BV C. Observed image f (x) = u(x) + v(x) is corrupted by noise. Statistics of noise v are known (often assumed Gaussian white noise, i.e. the sampled noise sequence g k (ω), k Z 2, is i.i.d. N(0, σ). E[ ] denotes mathematical expectation with respect to these statistics.

27 Wavelet Shrinkage - Minimax Formulation Find a good substitute or estimator û of u: construct a linear or nonlinear mapping Φ : f û which Preserves a priori knowledge on u, i.e. û BV C Minimize expected error between true u and estimated û: E[ û u ]. Expected Risk depends on some norm and is R(Φ, u, σ) = E[ û u ] or E[ û u 2 ] when Hilbert space norm is used. Worst risk is ρ(φ, σ) = sup{r(φ, u, σ) : u B} where it is assumed that u belongs to some ball B = { u C}. Want the worst risk to be as small as possible by a wise choice of Φ. This leads to the minimax formulation: ρ(σ) = inf{ρ(φ, σ) : Φ M} = inf Φ sup{e[ û u 2 ] : u B}

28 Robust Statistics and Compressive Sensing 1 David Donoho, Compressed Sensing, IEEE Transactions on Information Theory, Vol. 52, No. 4, April 2006, pp Candes E., Romberg J., and Tao T. 2006b Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inf. Theory 52 pp

29 Transform Compression; Assumption of Sparsity x IR m is a signal or image with m samples or pixels and an orthonormal basis {ψ i : i = 1,..., m} of IR m (e.g. wavelet basis, Fourier basis or a local Fourier basis). Then x has transform coefficients θ i = x, ψ i which are assumed sparse: for some 0 < p < 2 and R > 0, ( ) 1/p θ, θ i p R i

30 Example: Bounded Variation Model for Images Image brightness is represented by function f (x, y) on unit square 0 x, y 1 which satisfies f (x, y) dxdy R Wavelet viewpoint: data are seen as superposition of contributions from various scales. Let x (j) denote component of data at scale j, () denote the orthonormal basis of wavelets at scale j, containing 3 4 j elements. Then θ (j) 4R

31 Optimal Recovery/Information-Based Complexity X class of objects of interest, subset of IR m Information Operator I n : X IR n samples n pieces of information about unkown signal or image x X. Here I n (x) = ( ξ 1, x,..., ξ n, x ) where ξ j are sampling kernels (nonadaptive, fixed independently of x. Approximate reconstruction operator A n : IR n IR m Analyse l 2 error of reconstruction & find optimal reconstruction algorithm, so use minimax l 2 erros E n (X ) = inf A n,i n sup x A n (I n (x)) 2 x X Evaluate E n (X) and find practical schemes which come close to attaining it

32 Compressive Sensing uses p = 1 Suppose unknown signal x IR M is sparse in some basis. Reconstruct the signal using only a few (= N << M) linear measurements, i.e. take the inner product of x with chosen set of vectors. Use N measurements y to reconstruct M-length sparse signal x having K < N << M nonzero entries. * * * * = A = N M Measurement Matrix 0 0 * * 0 0 * 0

33 Try Euclidean Norm: l 2 Underdetermined linear system. Problem is ill-posed: there are infinitely many solutions ˆx. Classical solution technique uses Least-Squares; note fewer rows than columns in measurement matrix A. ˆx = arg min x 2 subject to y = Ax We have seen that the solution is given by least-square minimum norm ˆx = (A T A) 1 A T y which can be computed very quickly using LU-factorization. Unfortunately, a small l 2 norm for ˆx does NOT imply sparsity.

34 Modern Approach Minimizes l 0 or l 1 Norms x = arg min x p subject to Ax = y Exploit known sparsity of x in some basis. Of all solutions seek the sparsest one x. (p = 0) Define l 0 norm, x 0, to be number of nonzero entries. This gives perfect reconstruction with high probability but combinatorial complexity. (p = 1) Seek solution with the smallest l 1 norm. If N CK, C 3, Donoho and Candes, et. al. have shown perfect reconstruction with high probability, and linear programming complexity.

35 1-D Sparse Signal Recovery: L 1 wins over L 2 Figure: (Left) Original sparse signal. (Center) Very noisy least squares minimum L 2 -norm reconstruction. (Right) Accurate minimum L 1 -norm reconstruction.

36 n-widths Bound Minimax Error Gel fand n-width of set X with respect to l m 2 norm is d n (X, l 2 ) = inf V n sup { x 2 : x V n X } infimum over n-dimensional linear subspace of IR m and Vn is orthocomplement of V n with respect to standard Euclidean inner product. Kolmogorov n-width of X with respect to l m 2 norm is d n (X, l 2 ) = inf V n sup x X inf { x y 2 } y V n The infimum is over n-dimensional linear subspaces of IR m. Measures quality of approximation of X possible by n-dimensional subspaces of V n.

37 Donoho s Conditions CS1-CS3 CS1 Minimal singular value of Φ J exceeds η 1 > 0 uniformly in J < ρn/ log(m). Requires a quantitative degree of linear independence among all small groups of columns. CS2 On each subspace V J the inequality v 1 η 2 n holds uniformly in J < ρn/ log(m). Linear combinations of small groups of columns give vectors that look like random noise when l 1 and l 2 norms are compared. CS3 On each subspace V J, Q c J (v) η 3/ log(m/n) v 1, v V J uniformly in J < ρn/ log(m). For every vector in some V J, quotient norm Q J c is never much smaller than l 1 norm on IR n.

38 Applications to Cone-Beam Tomography? Image reconstruction in circular cone-beam computed tomography by constrained, total-variation minimization by Emil Sidky and Xiaochuan Pan, Phys. Med. Biol. 53 (2008) Authors approximate image reconstruction for circular cone-beam CT to inversion of a finite linear system Mf = g Effects of limited angular range or angular sampling or large cone angles will be incomplete data, and a rectangular system matrix M; g is the data set of measurements. Goal is to recover f we know the problem is ill-posed with many solutions. Classical approach uses Least-Squares; note fewer rows than columns in measurement matrix M. Variationally, f = arg min f 2 subject to Mf = g; solution given by least-square minimum norm f = (M T M) 1 M T g which can be computed very quickly using LU-factorization, but small l 2 norm for f does not imply sparsity.

39 Adaptive Steepest Descent-Projection onto Convex Sets Ignoring positivity constraints, equivalent to solving unconstrained, convex optimization problem: f = arg min Mf g data τ f TV Above can be solved using standard methods such as conjugate gradient or steepest descent. Authors state it is more convenient to use the constrained formulation. Formulation of the optimization problem and constraints on solution: find the discrete image f that minimizes the TV norm subject to the inequality constraints of (A) data fidelity and (B) non-negativity. f = arg min f TV s.t. Mf g 2 ɛ and f 0

40 Steepest Descent To minimize a nonlinear functional, follow Cauchy s lead and try the method of Steepest Descent (1840) 1 Pick starting point x 0 and a small stepsize α > 0. Move a distance α in the direction g 0 in which f decreases most rapidly at x 0. Calculus says this is g 0 = f (x 0 ). New approximation is x 1 = x 0 + αg 0. 2 Recursively, define the sequence x n+1 = x n α n f (x n ) Under what conditions does this sequence converge?

41 Example using Steepest Descent

42 Continuous Gradient Descent Given differential operator F : {H,, } {K, (, )}, use gradient descent to find its zeros. Note different inner products in domain space give different descent directions. If J(u) = 1 2 F(u) 2, gradient is J(u) = F (u) F(u). (Prime denotes Fréchet derivative, star denotes adjoint.) Find z : [0, ) H satisfying dz dt = J(z(t)), z(0) = z 0 Sufficient conditions for convergence include: 1 Convexity: J (u)v, v m v 2 2 Gradient inequality: J(u) F (u) F(u) C F(u)

43 2-D Sparse Signal Recovery: Again L 1 wins over L 2 (Top-Left) Original Shepp-Logan edges. (Top-Right) Minimum L 2 - norm reconstruction. (Bottom) Minimum L 1 -norm reconstruction after 2000, 4000, and 5500 iterations.

44 References Neuberger JW. Sobolev Gradients and Differential Equations Springer-Verlag: Berlin. Carey GF, Richardson WB A note on least-squares methods Communications on Num. Methods in Engineering : Richardson WB, Sobolev Gradient Preconditioning for Image Processing PDE s. Commun. Numer. Meth. Engng : Candes E., Romberg J., and Tao T. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory :