Max-Planck-Institut fur Mathematik in den Naturwissenschaften Leipzig Conditions for equality of hulls in the calculus of variations by Georg Dolzmann, Bernd Kirchheim and Jan Kristensen Preprint no.: 1 000

CONDITIONS FOR EQUALITY OF HULLS IN THE CALCULUS OF VARIATIONS GEORG DOLZMANN BERND KIRCHHEIM JAN KRISTENSEN Abstract. We simplify and sharpen several results by K. Zhang concerning properties of quasiconvex hulls of sets and quasiconvex envelopes of their distance functions. The approach emphasizes the underlying geometry and in particular we show thatk pc = K c implies K rc = K c if and only if minfm ng thus answering a question raised in [Z]. This paper addresses a surprising relation between semiconvex hulls of compact sets K M mn rst noted by Zhang [Z], namely that K qc = K c implies K rc = K c and thus K rc = K qc. Our approach simplies the original proof given by Zhang and emphasizes more the underlying geometry. As pointed out to us by D. Preiss [P], this implication constitutes a nonlinear version of the result that subspaces without rank-one directions do not support any nontrivial gradient Young measure (see [BFJK], Theorem 4.1). In particular, based on results by [T, Se, B], this equivalence allows us to show that Zhang's statement holds for the polyconvex hull if and only if minfm ng, see Theorem 3 and the following remark. Thus the rank-one convex and the quasiconvex hulls agree for special sets K with K qc = K c, despite of the fact that rank-one convexity and quasiconvexity are known to be dierent concepts for m 3 ([Sv1]). In general the quasiconvex hull of a compact set is equal to the zero set of the quasiconvexication of the distance function d K p (x) = inffjx ; yj p : y Kg, p [1 1) (see [Z3]), and this motivates to study general properties of distance functions and their semiconvex envelopes. We show in Example 5 that one cannot expect a uniform growth of d qc K p () close to Kqc. Moreover, Example 6 demonstrates that smooth sets need not have smooth quasiconvex hulls. Both results indicate that numerical schemes are likely to produce unreliable results, unless the boundary of the set is well resolved. Our last result, Theorem 9, proves the surprising fact that the distance function itself cannot be used as an indicator of whether a given set is quasiconvex or not. In fact, d K p () is separately convex (i.e. convex in each argument, a property implied by quasiconvexity) on M mn n B(0 R), R arbitrarily large, if and only if K is convex. This was rst proven in [Z3] for p under stronger assumptions on the set K. Our selfcontained proof relies entirely on the well-known fact that the metric projection onto K is unique if and only if K is convex. Notation. Let M mn be the space of all real (m n){matrices. We use jj for the Euclidean norm on M mn, i.e. jxj = tr(x T x). By dim(k) we understand the ane dimension of the set K M mn. We write B(K ") =fx M mn : d K 1 (x) "g U(K ") =fx M mn : d K 1 (x) <"g: A function f : M mn! R is called polyconvex if there exists a convex function g of x and the vector M(x) of all minors of x (i.e. all subdeterminants of x) such that f(x) =g(x M(x)). It is Date: December 31, 1999. 1991 Mathematics Subject Classication. 6B15, 74N15. Key words and phrases. Gradient Young measures, quasiconvex hull, distance function. GD, BK and JK were supported by the EU TMR project on `Phase Transitions in Crystalline Solids' FMRX-CT98-09 (DG 1-BDCN). JK gratefully acknowledges partial support from the Max Planck Institute for Mathematics in the Sciences, Leipzig, and from EPSRC through grant GR/L906. 1

GEORG DOLZMANN BERND KIRCHHEIM JAN KRISTENSEN said to be quasiconvex if for all x M mn and all smooth functions ' : Q =[0 1] n! R m with ' = 0on@Q the inequality Z Z f(x + D'(z))dz f(x)dz Q holds. Finally, f is rank-one convex if f is convex on rank-one lines, i.e., the functions t 7! g(t) = f(x + tr) are convex in t for all x r M mn, rank(r) = 1. A fundamental result in the calculus of variations states that a variational integral I(u) = R f(du)dz is weakly sequentially lower semicontinuous in suitable function spaces if and only if f is quasiconvex. Polyconvexity is a sucient condition for quasiconvexity while rank-one convexity is a necessary one. For any function f : M mn! R we call f qc (x) = supfg(x) : g : M mn! R quasiconvex and g fg the quasiconvex envelope of f the rank-one convex and the polyconvex envelope are dened analogously. In analogy to one possible formula for the (closed) convex hull of a compact set K, denoted by K c,we dene its (closed) quasiconvex hull by K qc = fx : f(x) sup f(y) for all f : M mn! R quasiconvexg: yk A corresponding denition holds for the rank-one convex hull K rc and the polyconvex hull K pc. It is a useful fact that K qc can also be dened as the set of barycenters of gradient Young measures supported on K. In addition to these hulls related to classes of semiconvex functions we dene the closed lamination convex hull L c (K) ofk as the closure of the set L(K) = S 1 i=0 L i(k) where L 0 (K) =K and L i+1 (K) = x = a +(1; )b : [0 1] a b L i (K) rank(a ; b) 1 : These denitions imply immediately that L c (K) K rc K qc K pc K c and it is an open question whether K rc = K qc in M. See [Mu] for a discussion of these notions and their relations. Lemma 1. Let K M mn becompact, contained in the ane subspace X and let H be a hyperplane in X supporting K c. Then (K \ H) qc = K qc \ H and (K \ H) c = K c \ H. Proof. By denition our assumption means that there exists a linear f : X! R such that H = fx X : f(x) =maxf(k c )g. Given any x K qc \ H, we nd a gradient Young measure such that spt() K and x = R zd(z). Since Z Z f(x) = f(y) d(y) max f(k c ) d(y) =f(x) we conclude spt() H \ K. Consequently, x (K \ H) qc. The reverse inclusion is obvious and the same argument with probability measures instead of gradient Young measures proves the result for the convex hull. The following result was rst proven in [Z]. We include a short proof for the convenience of the reader. Theorem. Let K M mn becompact. If K qc = K c then L dim(k) (K) =K c. Proof. Otherwise we nd among all compact sets for which the conclusion fails a set K 0 of minimal ane dimension d 0 1. Let X be the d 0 -dimensional ane subspace containing K 0. We choose x 0 K0 c n L d 0 (K 0 ): If X does not contain any rank-one line then K 0 = K qc 0 by [BFJK, Theorem 4.1] and hence K0 c = Kqc 0 = L 0(K 0 ), acontradiction. Therefore we nd in X a rank-one line ` through x 0 and a point x 1 (` \ @ X K0) c n L d0 ;1(K 0 ) Q

CONDITIONS FOR EQUALITY OF HULLS IN THE CALCULUS OF VARIATIONS 3 (if x 0 @ X K c 0 then we maychoose x 1 = x 0 ). Let H X be a supporting hyperplane of K c 0 through x 1. We have by Lemma 1 for K 1 = H \ K 0 that but K qc 1 = H \ (K 0) qc = H \ K c 0 = K c 1 x 1 K c 1 n L d 0 ;1(K 0 ) K c 1 n L d 0 ;1(K 1 ): This contradicts the minimality ofd 0 and proves the proposition. Motivated by a remark by D. Preiss we notice the following. Theorem was proved using the result from [BFJK] that in subspaces without rank-one directions all sets are quasiconvex. It turns out that this result is also implied by Theorem. Theorem 3. Assume that X is a linear subspace of M mn not containing any rank-one line. Suppose that for every K X compact K pc = K c (K qc = K c ) implies L c (K) =K c. Then every compact subset of X is polyconvex (quasiconvex). Proof. We consider the case of polyconvexity only, the argument in the quasiconvex situation is similar. Assume that the conclusion fails. Then we can x K X compact and x K pc nb(k 3") for some " > 0. Let K ~ = (B(K ") \ X) c n U(x ") then K ~ is compact, and obviously K ~ B(K ") \ X. Since taking the polyconvex hull commutes with translations we obtain ~K c (B(K ") \ X) c ~ K [ (B(x ") \ X) ~ K [ (B(K ") \ X) pc ~ K pc : The opposite inclusion is true in general, hence ~ K c = ~ K pc. On the other hand, x ~ K c n ~ K shows that ~ K c 6= ~ K but by our assumption on X we have ~ K = Lc ( ~ K)= ~ K c, a contradiction. Remark. This equivalence allows us to answer a question raised in [Z] whether the polyconvex version of Theorem holds: if K pc = K c, then L dim(k) (K) =K c. The paper [B] contains an example of a subspace of M 33 without rank-one matrices but with a subset that is not polyconvex. On the other hand, as pointed out in [B] the results from [T] (see also [Se]) imply the following: Whenever a linear subspace X of M m or M m does not contain any rank-one line then all compact subsets of X are polyconvex. This shows that the polyconvex variant of Theorem is true if and only if min(m n). As in [Z] we obtain easily a comparision principle for distance functions. By denition, d qc, but there holds also a reversed verion of this inequality. d qc K\H Proposition 4. Let p [1 1), K M mn be compact and let H be a supporting hyperplane of K. Then there is a non-decreasing, continuous function W H :[0 1)! [0 1) such that W H (0) = 0 and d qc K\H p (x) W H(d qc K p (x)) for all x H: Proof. Assume otherwise. Then there exists a positive " and a sequence fx k g 1 k=1 H such that d qc K\H p (x k) >"but d qc K p (x k) < 1=k for all k 1. Since d qc K p () majorises the convex function x! (maxf0 jxj ;maxfjyj y Kgg) p, we can suppose that x k! x 0 H. Then obviously d qc K p (x 0)=0 but d qc K\H p (x 0) ", which implies (see [Z1]) that x 0 K qc \ H n (K \ H) qc. This contradiction to Lemma 1 nishes our proof. The argument just used also proves that for all quasiconvex, compact sets K and all p [1 1) there exist non-decreasing, continuous functions W K p :[0 1)! [0 1) such that W K p (0) = 0 and d K p (x) W K p (d qc K p (x)) for all x M mn : Our next example shows that the quantiers cannot be interchanged. K

4 GEORG DOLZMANN BERND KIRCHHEIM JAN KRISTENSEN Example 5. For all p [1 1) and all non-decreasing, continuous functions W :[0 1)! [0 1) with W (0) = 0 there exist a compact quasiconvex set K and a sequence fx k g in M such that d K p (x k )=W (d qc K p (x k))!1and d K p (x k )! 0: Proof. In the construction we use the isometric embedding of R into M which identies x with diag(x 1 x ). Choose a sequence fb k g 1 k=1 strictly decreasing to zero such that W (bp k ) < ;p(k+) =k. Now wedene K = f( ;k b k ) : k 1g[f(0 0)g and x k =(3 ;k; b k+1 ): Obviously, d K p (x k ) dist(3 ;k; proj 1 (K)) p ;(k+)p, where proj 1 denotes the projection x 7! x 11. Since (1 0) is a rank-one direction, we can estimate d rc K p(x k ) 1 d K p ( ;k;1 b k+1 )+d K p ( ;k b k+1 ) (b k ; b k+1 ) p b p k and hence d K p (x k )=W (d rc K p (x k)) ;(k+)p =W (b p k ) k. On the other hand K is quasiconvex (even polyconvex) since det(x ; y) > 0 for any x y K, x 6= y, see [Sv]. Example 6. There exists a smooth set U M such that U qc is not smooth, in the sense that Tan(U qc x) (see e.g. [F, Denition 4.3]), is not a linear subspace for a suitable x U qc. This contrasts with the situation for convex hulls, where it is easily seen that the convex hull of a smooth compact set is again smooth. To keep the calculations simple, we use conformal and anticonformal coordinates, i.e., we make the identication x R 4 x1 + x = 3 ;x + x 4 M x + x 4 x 1 ; x : 3 Then det(x) =x 1 + x ; x 3 ; x 4 and jxj =(x 1 + x + x 3 + x 4 ). We dene on R4 the function q f(x) = x 1 + x ; + x 3 + x 4 ; and dene U = fx : f(x) < 0g. Note that even on the larger set of all x with f(x) < 1 the gradient p x rf(x) = 1 + x p ; (x x 1 + x 1 e 1 + x e )+(x 3 e 3 + x 4 e 4 ) is a non-vanishing C 1 function. Hence, @U is a smooth set containing all points x with j(x 1 x )j = j(x 3 x 4 )j =1. This implies of course that 0 ( U) rc and therefore x ( U) rc whenever det(x) =0 and jxj. On the other hand, we have forv =(1 1) R and all x U hv (j(x 1 x )j j(x 3 x 4 )j)i hv ( 0)i ;jvj (j(x 1 x )j j(x 3 x 4 )j) ; ( 0) > ; =0 consequently det(x) > 0 whenever x U. This shows that ( U) pc fx : det(x) 0g and in particular none of the sets ( U) rc ( U) qc and ( U) pc is smooth at the origin. An easy, but slightly longer calculation also shows that the same holds true for the (boundary of) the hulls U rc U qc and U pc of the open set U itself. In the last part of this note we extend results from [Z3] concerning the square of the distance function to the case of arbitrary powers. We apply a more elementary and selfcontained reasoning which does not rely on the precise knowledge of the quasiconvex envelope of the squared distance to a double-well (see [K]). Instead we use the following estimate the proof of which weleave to the reader.

CONDITIONS FOR EQUALITY OF HULLS IN THE CALCULUS OF VARIATIONS 5 Lemma 7. Let K M mn be a compact set and p [1 1). Suppose x M mn and y K are such that jx ; yj = d K 1 (x) > 0. Then we have for all v M mn that lim sup t&0 d K p (x + tv) ; d K p (x) t jx + tv ; yj;jx ; yj x ; y p d K p;1 (x) lim = p d K p;1 (x)hv t&0 t jy ; xj i: As in [Z3], the crucial point in our proof is the nonuniqueness of the metric projection onto nonconvex sets. For the convenience of the reader we give a short proof of this fact following [F]. Theorem 8. Let K M mn be compact and suppose that for every x M mn there exists a unique point (x) K such that j(x) ; xj = d K 1 (x). Then K is convex. Proof. Throughout the proof we write d(x) =d K 1 (x). As d(y) d(x)+jx;yj, d is 1-Lipschitz. Hence, if x k! x and (x k )! y then (x) =y which together with the compactness of K proves that is continuous. We rst assert that for all x = K, and all t nonnegative ; (x)+t(x ; (x)) = (x). Otherwise there exists an x 6 K for which the assertion fails. Using uniqueness of the nearest point it is easy to check that the set of t for which the required equality holds is a closed interval [0 T], T 1. Let ~x = (x)+t (x ; (x)). The continuity of and Peano's existence theorem ([W], x7) allow us to choose a positive " <d(~x) and a curve ' :(;" ")! M mn such that '(0) = ~x and ' 0 (t) = (('(t)) ; '(t))=j('(t)) ; '(t)j. Now consider the real-valued function t 7! (d ')(t). From Lemma 7 we infer that its derivative from the right, (d ') 0 +(t), is bounded from above by ;1 for all t (;" ") and hence j'(p) ; '(q)j d('(p)) ; d('(q)) q ; p if ; "<p<q<". On the other hand, as j' 0 (t)j 1 on (;" "), we also obtain j'(p) ; '(q)j jp ; qj for all p q. Both estimates together imply d('(p)) ; d('(q)) = j'(p) ; '(q)j = q ; p whenever ; "<p<q<": The strict convexity of the Euclidean norm implies that '((;" ")) is a segment, which is due to the direction of ' 0 (0) contained in the ray f(x) +t(x ; (x)) : t > 0g. This shows that d('(p)) = d('(0)) ; p = j'(p) ; (x)j for all p (;" ") andby uniqueness of the metric projection ('(p)) = (x). This contradicts the maximality of T, and the claim is established. We conclude that for any x = K and t arbitrarily large K \ U((x)+t(x ; (x)) tjx ; (x)j) =. In the limit t!1we obtain hx x ; (x)i > h(x) x; (x)i = maxfhy x ; (x)i : y Kg which represents K as the intersection of halfspaces and nishes our proof. We are now in a position to generalize the results in [Z3] for the squared distance function to arbitrary powers under weaker assumptions on K. Theorem 9. Let K M mn be compact, let p [1 1), and assume that d K p is a separately convex function outside of some compact set. If M mn n K is connected, then K is convex. Proof. By assumption we may choose a nite R such that d K p is separately convex whenever restricted to a ball disjoint with K ~ = B(K R). We rst show that the metric projection onto K is unique in each point x = K. ~ Otherwise we nd x 6 K ~ and y1 y K, y 1 6= y, such that jy i ; xj = d K 1 (x), and we may assume that hy ; y 1 e j i < 0 for some canonical basis vector e j. We estimate using Lemma 7 that d K p (x + te j )+d K p (x ; te j ) ; d K p (x) lim sup t&0 t x ; y 1 p d K p;1 (x) he j jx ; y 1 j i + h;e x ; y j jx ; y j i = p d K p;1 (x)he j y ; y 1 jy 1 ; xj i < 0: This implies that d K p is not separately convex near x, acontradiction. Next, note that the metric projection onto the enlarged set K ~ is unique on the whole space mn M. Uniqueness is trivial for points in K, ~ so we may suppose that x = K. ~ Consider z1 z K ~ with

6 GEORG DOLZMANN BERND KIRCHHEIM JAN KRISTENSEN jz i ; xj = d ~K 1 (x) d K 1 (x) ; R, and y i K with jz i ; y i j = R. This implies jx ; y i jd K 1 (x), and thus y 1 = y = K (x) by the claim already established. Moreover, as jx ; K (x)j = jx ; z i j + jz i ; K (x)j we see that both z 1 z are in the segment [ K (x) x] and because furthermore jz 1 ; K (x)j = R = jz ; K (x)j, z 1 = z, so ~K (x) is unique. In particular, due to the foregoing theorem, ~ K is a convex set. Now, we show that@k c K. In fact, given any x @K c there is a unit vector v M mn such that hx vi =maxfhy vi : y Kg. The convexity of ~ K implies that x + Rv ~ K as well, but it is easy to check that B(x + Rv R) \ K B(x + Rv R) \fy : hy vi hx vig = fxg: By denition of ~ K this shows that x K as required. We nish our proof by showing that @K c K and M mn nk connected imply that K is convex. This statement is similar to Lemma.1 in [Z3], but slightly more general. To verify our assertion, choose any y = K and connect it by an arc ' inside M mn n K to a point x suciently far away from the origin to ensure that x = K c. By assumption, ' does not intersect @K c, and we thus infer that y = K c. This concludes the proof of the theorem. References [B] J.M. Ball, Remarks on the paper: \Basic calculus of variations", Pacic J. Math. 116 (1) (1985), 7-10. [BFJK] K. Bhattacharya, N.B. Firoozye, R.D. James, R.V. Kohn, Restrictions on Microstructures, Proc. Royal Soc. Edinburgh A 14 (1994), 843-878. [F] H. Federer, Curvature measures, Trans. Amer. Math. Soc. 93 (1959), 418-491. [K] R.V. Kohn, The relaxation of a double-well energy, Cont. Mech. Therm. 3 (1991), 981-1000. [Mu] S. Muller, Variational methods for microstructure and phase transitions, to appear in: Proc. C.I.M.E. summer school `Calculus of variations and geometric evolution problems', Cetraro, 1996, (S. Hildebrandt and M. Struwe, eds.), http://www.mis.mpg.de/cgi-bin/lecture-notes.pl. [P] D. Preiss, personal communication, 1997 [Se] D. Serre, Formes quadratiques et calcul des variations, J. Math. Pures Appl. 6 () (1983), 177-196. [Sv1] V. Sverak, Rank-one convexity does not imply quasiconvexity, Proc. Royal Soc. Edinburgh Sect. A 10 (199), 93-300 [Sv] V. Sverak, On Tartar's Conjecture, Ann. Inst. Henri Poincare 10 (4) (1993), 405-41. [T] F.J. Terpsta, Die Darstellung biquadratischer Formen als Summe von Quadraten mit Anwendung auf die Variationsrechnung, Math. Annalen 116 (1938), 166-180. [W] W. Walter, Ordinary Dierential Equations, Springer 1998. [Z1] K.W. Zhang, Quasiconvex functions, SO(n) and two elastic wells, Ann. Inst. H. Poincare 14 (1997), 759-785. [Z] K.W. Zhang, On various semiconvex hulls in the calculus of variations, Calc. Var. PDE 6 (1998), 143-160. [Z3] K.W. Zhang, On various semiconvex relaxations of the squared distance function, to appear in Proc. Royal Soc. Edinburgh. Georg Dolzmann, Max Planck Institute for Mathematics in the Sciences, Inselstr. -6, D-04103 Leipzig, Germany E-mail address: georg@mis.mpg.de Bernd Kirchheim, Max Planck Institute for Mathematics in the Sciences, Inselstr. -6, D-04103 Leipzig, Germany E-mail address: bk@mis.mpg.de Jan Kristensen, Mathematical Institute, University of Oxford, 4-9 St Giles, Oxford OX1 3LB, England E-mail address: kristens@maths.ox.ac.uk