Optimization with nonnegativity constraints

Transcription

1 Optimization with nonnegativity constraints Arie Verhoeven CASA Seminar, May 30, 2007

2 Seminar: Inverse problems 1 Introduction Yves van Gennip February 21 2 Regularization strategies Miguel Patricio March 3 3 Regularization by Hans Groot March 21 Galerkin methods 4 Inverse eigenvalue problems Marco Veneroni April 4 5 Image deblurring Willem Dijkstra April 18 6 Parameter identification Nico van der Aa May 2 7 Total variation regularization Mark van Kraaij May 23 8 Optimization with Arie Verhoeven May 30 nonnegativity constraints 9?? Martijn Slob June 13 10?? Marc Noot June 20

3 Outline 1 Introduction 2 Theory of constrained optimization 3 Numerical variational methods 4 Iterative nonnegative regularization methods 5 Numerical test results 6 Conclusions

4 Outline 1 Introduction 2 Theory of constrained optimization 3 Numerical variational methods 4 Iterative nonnegative regularization methods 5 Numerical test results 6 Conclusions

5 Formulation of the problem Discretized mathematical model: Discrete Fourier Transform: f true i = f true (x i ). g true = K f true. Measurements, e.g. from astronomical imaging, satisfy d i Poisson(g true i ) + Normal(0, σ 2 ). Goal: find f for given d and K.

6 Least Squares method with Tikhonov regularization The least squares solution minimizes the functional: The minimizer is given by J ls (f) = 1 2 Kf d 2 + α 2 f 2. f ls α = [ K T K + αi] 1 K T d. The regularization parameter α is selected to minimize f ls α f true. But this approach does not work in general when f true is unknown.

7 The L-curve method Define X(α) = log Kf α d 2 and Y (α) = log f α 2. With Tikhonov regularization X, Y are smooth functions. Then α can be selected which maximizes the curvature function Ẍ(α)Ẏ (α) Ẋ(α)Ÿ (α) κ(α) = (Ẋ(α)2 + Ẏ. (α)2 ) 3 2 Note that this selected point corresponds to the "corner" of the L-curve, which plots X against Y. Although this method is nonconvergent, it can be used to improve the value of α without knowledge of f true.

8 Nonnegatively constrained minimization For astronomical imaging it is well-known that f i 0. Thus it is more accurate to solve instead min f J ls (f) subject to f 0. Even better is to include the stochastic information of the measurements. Then we solve the following constrained Poisson likelihood minimization problem where J lhd (f) = i=1 min f J lhd (f) subject to f 0, n n (g i +σ 2 )+ ((max{d i, 0}+σ 2 ) log(g i +σ 2 ))+ α 2 f 2. i=1

9 Example 750, 0.1 < x < 0.25, 250, 0.3 < x < 0.32, f true (x) = (x 0.75)(0.85 x), 0.75 < x < 0.85, 0 otherwise.

10 Constrained likelihood minimization

11 Outline 1 Introduction 2 Theory of constrained optimization 3 Numerical variational methods 4 Iterative nonnegative regularization methods 5 Numerical test results 6 Conclusions

12 Optimization Consider the inequality constrained minimization problem Active set of indices: min J(f) subject to c(f) 0. f R n A(f) = {i c i (f) = 0}.

13 KKT conditions Karush-Kuhn-Tucker (first order necessary) conditions for inequality constrained minimization: There exists a vector λ such that m gradj(f ) λ i gradc i(f ) = 0 and i=1 λ i 0, c i (f ) 0, λ i c i(f ) = 0. We define the projection of f onto C as P C (f) = arg min f v. v C The operator P C is well defined and continuous.

14 Nonnegativity constraints Now we consider the problem min J(f) subject to f 0. f R n If J is continuously differentiable and f a local minimizer, it follows that λ = gradj(f ). Then we obtain J f i (f ) 0, i 0, fi J (f ) = 0. f i f A point which satisfies these conditions is a critical point, but it needs not to be a minimizer if.e.g. J is not strictly convex,

15 Nonnegativity constraints II For nonnegativity constraints, we define the feasible set C = {f f 0}. The projected gradient C J : C R n satisfies { J f [ C J] i = i (f) if f i > 0, min{0, J f i (f)} if f i = 0. Thus f is a critical point if and only if C J(f ) = 0. For given C we define P(f) = arg min v f. v 0 If f is a local minimizer, then f = P(f τgradj(f )) for any τ > 0.

16 Outline 1 Introduction 2 Theory of constrained optimization 3 Numerical variational methods 4 Iterative nonnegative regularization methods 5 Numerical test results 6 Conclusions

17 Gradient projection method ν := 0; f 0 := nonnegative initial guess; begin p ν := grad J(f ν ); τ ν := arg min τ>0 J(P(f ν + τp ν )); f ν+1 := P(f ν + τ ν p ν ); ν := ν + 1; end This generalized Steepest Descent method converges linearly to the global minimizer if J is strictly convex, coercive, and Lipschitz continuous.

18 Projected Newton method ν := 0; f 0 := nonnegative initial guess; begin g ν := grad J(f ν ); Identify active set A ν ; H R := reduced Hessian at f ν ; s := H 1 R g ν ; τ ν := arg min τ>0 J(P(f ν + τs)); f ν+1 := P(f ν + τ ν s); ν := ν + 1; end The reduced Hessian equals { δij if i A(f) or j A [H R ] ij = 2 J f i f j otherwise. If the active set can be correctly identified, this algorithm will be locally quadratically convergent.

19 Gradient projection-reduced Newton method ν := 0; f 0 := nonnegative initial guess; begin Gradient Projection Stage p GP := grad J(f ν ); τ GP := arg min τ>0 J(P(f ν + τp GP )); f GP ν := P(f ν + τ GP p GP ); Reduced Newton Stage Identify active set A(f GP ν ); g R := reduced gradient at f GP ν ; ν ; H R := reduced Hessian at f GP s := H 1 R g R ; τ RN := arg min τ>0 J(P(f GP ν f ν+1 := P(f GP ν + τ RN s); ν := ν + 1; end + τs)); We need { 0 if i A(f), [g R (f)] i = J f i (f) otherwise. This algorithm combines the global convergence of Gradient Projection with the locally quadratic rate of Projected Newton. For large-scale problems the linear system H R s = g R could be solved by an iterative method like the Conjugate Gradient method.

20 Outline 1 Introduction 2 Theory of constrained optimization 3 Numerical variational methods 4 Iterative nonnegative regularization methods 5 Numerical test results 6 Conclusions

21 Richardson-Lucy iteration Iterative methods use iteration count as regularization parameter. Consider m J(f) = d i log[kf] i. i=1 Approximation of maximizer by Richardson-Lucy iteration: f ν+1 j = f j ν k j m ( k ij i=1 d i n l=1 k ilf ν l ), where k j = m k lj. We get a sequence of approximations to the maximizer of J(f) subject to m m [Kf] i = d i. i=1 i=1 l=1

22 Modified Reduced Newton Steepest Descent ν := 0; f 0 := nonnegative initial guess; g 0 := K T (Kf 0 d); γ := (g 0, f 0. g 0 ); begin p ν := f ν. g ν ; u := Kp ν ; τ bndry := min{ [f ν ] i /[p ν ] i [p ν ] i < 0}; f ν+1 := f ν + τ ν p ν ; g ν+1 := g ν + τ ν K T u; γ := (g ν+1, f ν+1. g ν+1 ); ν := ν + 1; end

23 Outline 1 Introduction 2 Theory of constrained optimization 3 Numerical variational methods 4 Iterative nonnegative regularization methods 5 Numerical test results 6 Conclusions

24 1D

25 2D

26 Outline 1 Introduction 2 Theory of constrained optimization 3 Numerical variational methods 4 Iterative nonnegative regularization methods 5 Numerical test results 6 Conclusions

27 Summary Nonnegitivity constraints Theory of constrained optimization Variational methods Gradient projection method Projected Newton method Gradient projection-reduced Newton method Gradient projection-cg method Iterative methods Richardson-Lucy iteration Modified Steepest Descent algorithm Numerical test results

28 Conclusions Optimization with nonnegativity constraints often leads to more accurate reconstructions with e.g. less unwanted oscillations. Optimizing Poisson likelihood is more accurate than Least Squares. Iterative methods are preferable if no good a priori value of the regularization parameter is available. Variational regularization methods are more flexible, because they allow the use of prior information about the solution and constraints.

29 Literature. C.R. Vogel: Computational methods for inverse problems, SIAM, Philadelphia, 2002, pp

30 Literature. C.R. Vogel: Computational methods for inverse problems, SIAM, Philadelphia, 2002, pp J. Nocedal and S.J. Wright: Numerical optimization, Springer-Verlag, New York, S.G. Nash and A. Sofer: Linear and nonlinear programming, McGraw-Hill, New York, H.W. Engl, M. Hanke and A. Neubauer: Regularization of inverse problems, Kluwer Academic Publishers, Dordrecht, W.H. Richardson: Bayesian-based iterative methods for image restoration, Journal of the Optical Society of America, 62 (1972), pp B. Lucy: An iterative method for the rectification of observed distributions, Astronomical Journal, 79 (1974), pp

31 Introduction Theory of constrained optimization Numerical variational methods Iterative nonnegative regularization methods Num Questions?