A Simple Explanation of the Sobolev Gradient Method

Transcription

1 A Simple Explanation o the Sobolev Gradient Method R. J. Renka July 3, 2006 Abstract We have observed that the term Sobolev gradient is used more oten than it is understood. Also, the term is oten used with no apparent awareness o its origin. This short note is an attempt to correct both o these situations. 1 Introduction John W. Neuberger coined the term Sobolev gradient and began developing the method in the 1970 s, but his irst journal article on the subject was not published until 1985 ([4]). The primary reerence or his work is now [5]. The Sobolev gradient method is a gradient descent method designed to ind critical points o a (discretization o a) unctional o the orm φ(u) = F (Du), where D is a dierential operator and u is an element o a Sobolev space H or which F (Du) is integrable. The unctional represents a variational ormulation o a dierential equation, and any dierential equation G(Du) = 0 can be treated as a least squares variational problem by taking F (Du) = G(Du) 2 /2 i no better alternative is available. While the primary application is numerical solution o nonlinear partial dierential equations, the method has been extended to geometric modeling problems ([6], [7]), inverse problems ([3]), optimal control problems ([1]), and image processing problems ([8]). Section 2 presents a brie outline o the key ideas in the context o Sobolev spaces, and Section 3 treats the discretized problem as one o numerical optimization. As a numerical optimization method, the key idea is preconditioned gradient descent, and the eicacy o the method is clear without any consideration o unction spaces. Unlike other methods or preconditioning, however, the Sobolev gradient method arises in a very natural and elegant manner rom the Sobolev space setting. Reer to [2] or an extensive discussion o preconditioning. 2 Function Space Setting In order to make the ideas concrete, we apply the method to a simple example problem Poisson s equation: u xx + y = on a two-dimensional 1

2 domain with Dirichlet boundary conditions. Taking u H H 1,2 () and Du = (u, u x, ) L 2 () 3, we have F (Du) = (u 2 x + u 2 y)/2 + u integrable or L 2 (). Then Poisson s equation is the Euler-Lagrange equation associated with φ(u) = F (Du). We show below that it is obtained by setting the L 2 gradient o φ to zero. Note, however, that this involves an implicit assumption that φ has a gradient in L 2 () or, equivalently, that u H 2,2 (). Since the variational ormulation imposes a weaker restriction on u, it is reerred to as a weak ormulation. In order to treat the boundary conditions, we use an initial estimate that satisies the conditions and restrict perturbations o u to the subspace H 0 o unctions that satisy homogeneous boundary conditions. The Fréchet derivative φ (u), when it exists, is the bounded linear unctional deined by φ (u)h = lim [φ(u + αh) φ(u)] α or h H 0. For the example problem, we have φ 1 (u)h = lim [(u x + αh x ) 2 + ( + αh y ) 2 + 2(u + αh) α 0 2α α 0 1 (u 2 x + u 2 y + 2u)] 1 = lim [2α(u x h x + h y + h) + O(α 2 )] α 0 2α = [u x h x + h y + h] h H 0. (1) The Sobolev gradient S φ(u) is deined to be the unique element o H 0 that represents the bounded linear unctional: φ (u)h = S φ(u), h H h H 0. Note that boundedness o φ (u) is related to existence o S φ(u) H 0 by φ (u)h S φ(u) H h H h H 0. To show that our example unctional is C 1 (Fréchet dierentiable or all u), and the Sobolev gradient exists, denote by P the (sel-adjoint) orthogonal projection k k 1 o L 2 () 3 onto Dk = k x : k H 0, and deine π k 2 = k 1 or k y k 3 k 1, k 2, k 3 L 2 (). Then, rom (1), h φ (u)h = h x, u x = P Dh, u x h y = Dh, P u x L 3 2 L 3 2 = h, πp u x H L 3 2 h H 0. 2

3 Hence S φ(u) = πp u x H 0. (Note that D was chosen so that u, v H Du, Dv L 3 2.) A more computationally amenable expression or S φ(u) will be derived in the next section. Our purpose here is to show that the Sobolev gradient exists when the L 2 gradient may not. In order or φ (u) to be bounded in the L 2 norm, u must be in H 2,2 () so that, integrating by parts (or applying Green s theorem) in (1), we obtain φ (u)h = (u xx + y )h = (u xx + y + ), h L2 = φ(u), h L2 h H 0, giving the L 2 gradient φ(u) = (u xx + y ) +. 3 Numerical Setting Consider a gradient descent method or minimizing a unctional φ deined on a Hilbert space H = H 1,2 (). The method o steepest descent is deined by the iteration u k+1 = u k α k φ(u k ) (2) or u 0 H and some means o choosing step-lengths α k. In order that the iterates remain in H, the gradients φ(u k ) must be elements o H; i.e., derivatives must be compatible with the metric (inner product and corresponding norm) on H. More precisely, the directional derivative o φ at u in the direction h deines the Sobolev gradient S φ(u) by φ (u)h = S φ(u), h H h H. Stated dierently, the negative gradient o φ at u is the direction o steepest descent because it maximizes the decrease in φ per unit change in u. The norm used to measure units in the domain o φ strongly inluences the gradient direction. Now consider a inite dierence discretization o the problem. Although variational problems are usually discretized by a inite element method, inite dierencing is simpler and, in some cases, much more eicient ([7]) than inite elements. We retain the above notation but now have u k R N or N grid points in (or N/m grid points i u has m components associated with each grid point). Let H 0 be the subspace o vectors that satisy homogeneous boundary conditions, so that by restricting gradients to H 0, the iterates u k satisy the same boundary conditions satisied by u 0. Simply replacing the Sobolev gradient by the ordinary gradient (vector o partial derivative values) does not lead to an eective method. The ordinary gradient is associated with the Euclidean inner product which (up to a scale actor on a regular grid) is the discretization o the L 2 inner product. Hence, the ordinary gradient is essentially a discretization o the L 2 gradient, and it lacks 3

4 the required smoothness o a discretized Sobolev gradient. The undiscretized unctional φ involves derivatives o u and is thereore not well-deined on L 2 (). As the mesh width decreases, the discretization approaches an ill-deined problem. In short, standard descent methods are completely lacking in integrity or minimizing unctionals that involve derivatives. In order to properly discretize the Sobolev gradient, we must discretize the Sobolev inner product. To this end, denote by D the M by N matrix representing inite dierence approximations to the appropriate derivatives, along with values obtained by averaging i necessary. Assume that D has linearly independent columns so that D T D is positive deinite as well as symmetric. In practice it may be necessary to restrict D T D to H 0 and to ollow its application by projection onto H 0. We now have a discrete Sobolev inner product: g, h H = (Dg) T (Dh) = g T D T Dh. We can use the two inner products to relate the two gradients: and φ (u)h = φ(u) T h = h T φ(u) φ (u)h = S φ(u), h H = S φ(u) T D T Dh = h T D T D S φ(u) or all h H 0, implying that S φ(u) = (D T D) 1 φ(u). (3) The Sobolev gradient is thus computed by solving the linear system with the sparse symmetric positive deinite matrix D T D and the ordinary gradient as right hand side. Note that D T D is a discretized second derivative operator (typically o the orm I or identity I and Laplacian ), and its inverse is thereore a smoothing operator, where smoothness is measured by the number o continuous derivatives. It is shown in [5] that or a very general pair o compatible inner products, the corresponding gradients are related by an operator that has the properties o a Laplacian. We now discuss an alternative perspective the discretized Laplacian as a preconditioner. Assuming φ is twice continuously dierentiable in a neighborhood o u, we have the Taylor series φ(u + αh) = φ(u) + αh T φ(u) + α2 2 ht G(u)h + O(α 3 ), (4) where G(u) denotes the Hessian o φ at u. Let L be a symmetric positive deinite order-n matrix so that g, h L = g T Lh deines an inner product. Then there exists a corresponding gradient L φ(u) and Hessian G L (u) such that φ(u + αh) = φ(u) + α h, L φ(u) L + α2 2 h, G L(u)h L + O(α 3 ) = φ(u) + αh T L L φ(u) + α2 2 ht LG L (u)h + O(α 3 ) (5) 4

5 v1 v2 Figure 1: Ellipses {h R 2 : φ(u ) ht G(u )h = k} or all α R, h R N. Hence, equating terms in (4) and (5), L φ(u) = L 1 φ(u) and G L (u) = L 1 G(u). Let u = u be a critical point at which φ = 0 so that φ(u + αh) = φ(u ) + α2 2 ht G(u )h + O(α 3 ). For small α, the level sets deined by φ(u + αh) = k, k a constant, are approximated by the conic sections {u + h : φ(u ) ht G(u )h = k}. The behavior o φ in the vicinity o u is governed by the eigenvalues o G(u ). I they are all positive, the conic sections are hyperellipsoids, and u is a local minimum. This is depicted or the case N = 2 in Figure 1. The curves have been translated by u, and the (orthonormal) eigenvectors v 1 and v 2 associated with λ 1 = 1 and λ 2 = 4 are displayed. As the condition number o the Hessian λ max /λ min increases, the location o u becomes more sensitive to perturbations o the data, and more diicult to compute accurately. As the condition number approaches 1, on the other hand, the level sets become circular (or approach hyperspherical in general), and the negative gradient direction at every point is toward the minimizer u. Note, however, that unless φ is quadratic, so that G(u) is constant, the level sets deviate rom ellipsoidal with distance rom the critical point. The purpose o the preconditioner L is to reduce the condition number o the Hessian. Since 5

6 each descent step requires solution o the linear system L L φ(u k ) = φ(u k ), it is advantageous to choose preconditioners or which the systems are easily solved. We thereore have a tradeo between cheaply solved systems but with many descent steps due to ineective preconditioning, and more expensive but ewer descent steps. The two extremes are L = I, I denoting the identity (no preconditioning), and L = G(u k ) at step k, in which case G L = I is perectly conditioned, and the descent iteration is u k+1 = u k α k L φ(u k ) = u k α k G(u k ) 1 φ(u k ). With the optimal step-length α k = 1, this is a Newton iteration. Newton s method is clearly the optimal choice when it is viable. However, G(u k ) may not be positive deinite except in a small neighborhood o a local minimum. Also, or large-scale problems such as the discretized partial dierential equations o interest here, the Hessian is oten unavailable or is too expensive to compute or to store. The preconditioner used by the Sobolev gradient method L = D T D on the other hand can always be made positive deinite and, with the exception o extremely complex or high-order dierential operators, is generally well-conditioned, making the linear systems inexpensive to solve. Due to its regular structure, D T D need not be stored at all in most cases. For many problems the best strategy is to start with a Sobolev gradient method and then, i high accuracy is required, switch to a Newton iteration when suiciently close to a local minimum. Reerences [1] A. Jameson and S. Kim, Reduction o the adjoint gradient ormula in the continuous limit, AIAA paper, AIAA Aerospace Sciences meeting, Reno, NV, January [2] Faragó, I., Karátson, J., Numerical solution o nonlinear elliptic problems via preconditioning operators: Theory and applications. Advances in Computation, Volume 11, NOVA Science Publishers, New York, [3] I. Knowles, Variational methods or ill-posed problems, Contemporary Mathematics, 357 (2004), [4] J. W. Neuberger, Steepest descent and dierential equations, J. Math. Soc. Japan 37 (1985), [5] J. W. Neuberger, Sobolev Gradients and Dierential Equations, Springer Lecture Notes in Mathematics #1670, [6] R. J. Renka and J. W. Neuberger, Minimal suraces and Sobolev gradients, SIAM. J. Sci. Comput. 16 (1995),

7 [7] R. J. Renka, Constructing air curves and suraces with a Sobolev gradient method, CAGD 21 (2004), [8] G. Sundaramoorthi, A. Yezzi, and A. Mennucci, Sobolev Active Contours, IEEE Workshop on Variational and Level Set Methods,