METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP

Transcription

1 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP NICOLA ARCOZZI, FAUSTO FERRARI Abstract. We study the smoothness of the distance function from a set in the Heisenberg group, with some applications to the mean curvature of a surface. Contents. Introduction. Notation preliminaries 3 3. The metric normal the distance function for a plane 5 4. The metric normal the oriented metric normal for a smooth surface 7 5. Regularity of the distance function near a smooth surface 6. The Hessian of the distance function for a smooth surface 7 7. Mean Curvature 8. Examples Nonvertical planes The Carnot-Charathèodory sphere the Hessian of the distance function 7 References 9. Introduction In this article we investigate some properties of the function distance from a surface in the Heisenberg group. Let H = H be R 3 with the Heisenberg group structure let d be the associated Carnot-Charathéodory distance see Section for the definitions. If S H is closed, the distance from a point P to S is d E P = inf dp, Q Q E Here, we consider the case where S = Ω is the boundary of a C, open subset of H, in the Euclidean sense. The choice of considering C surfaces in the Euclidean sense is motivated by the fact that such surfaces satisfy a internal/external Heisenberg sphere condition at any noncharacteristic point, as we shall see below. Most results in this paper can be extended to open C surfaces in H. It is convenient to introduce the signed distance from S, { d S P if P Ω δ S P = d S P if P / Ω Observe that, given S, the signed distance δ S is defined modulo a sign, which corresponds to the choice of one of the two open sets having boundary S, or, equivalently, to the choice of an orientation for S. Date: 4/0/003. MSC 49Q5, 53C7, 53C

2 NICOLA ARCOZZI, FAUSTO FERRARI Our motivation for studying distance functions is twofold. In a recent article [] [3] it was shown, in particular, that the function d E is a viscosity solution of the eikonal equation in H, H d E = here, H is the horizontal gradient hence of the infinite Laplacian equation, see [5] [0],H u = X i ux j ux i X j u. i,j= It is interesting to have more infromation on particular solutions of these equations. Another reason for investigating d S is that a useful tool for the study of smooth surfaces S in R n is the fact that the Euclidean distance from S has maximum growth, at least near S, along the segment normal to S. We introduce a definition of metric normal to a smooth surface which plays precisely this role in the Heisenberg group. The metric normal to S at P, P a point of S, is the set N P S containing all points Q such that d S Q = dp, Q. It will be shown that N is an arc of a geodesic an explicit formula for N P S in terms of the parametrization of S will be given. The available definition of horizontal vector normal to a surface in H is related to the metric normal, but it does not exhaust the concept. In fact, there exist different surfaces passing through a point P, having the same horizontal normal vector, but different metric normals, their common feature being that their velocity at P is a multiple of the the horizontal normal vector. Roughly speaking, the possible horizontal normal vectors are in a -to- correspondence with the group of rotations in H, for each choice of the horizontal normal vector, the possible metric normals are in a -to- correspondence with the family of dilations in H. The knowledge of the metric normal allows one, as in R n, to prove the smoothness of the distance function. Theorem.. Let S = Ω be boundary of a C k open subset of H. Then, for any noncharacteristic point P S there is a neighborhood U of P in H such that δ S is C k in U. Moreover, if the surface S is the zero set of a smooth function g : H R, then, on the surface S, H δ S = ± Hg. A point C of S is characteristic for S if the Euclidean tangent space to S at C coincides with the horizontal fiber at C. See Section for precise definitions. The choice of + or depends on the choice of Ω between the two open sets in H having S as boundary. Here, C k is in the Euclidean sense. In Section 6 we compute an explicit expression for the horizontal Hessian see Section 6 for the definition Hess H d S of d S in terms of g. As a corollary, we obtain a new characterization of the mean curvature of a surface in S. Recall that, if S is a smooth surface in H defined by {gx, y, t = 0} P is a non-characteristic point of S, then the mean curvature h S P of S at P [9] is defined as in Euclidean space [8], 3 h S P = X XgP H gp Y gp + Y. H gp Below, we assume that gx, y, t = t fx, y, where f is a C function. Theorem.. Let S = Ω be boundary of a C open subset of H. Let Q = XXf Y f XY f + Y Xf Xf Y f + Y Y f Xf.

3 Then, 4 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 3 Y f Hess H d S = H Q 4 Xf Y f f 5 H Y f [ f 3 ] Xf Y f H Q + 4 Xf f 5 H f 3 In particular, for any non-characteristic P in S, 5 H d S P = h S P. [ Xf H f 5 Q 4 Y f H f 3 ] Xf H Q + 4 Xf Y f f 5 H f 3 This note is structured as follows. In Section we give notation some preliminaries. The Section 3 contains some elementary properties of the metric normal a description of the metric normal to S when S is a Euclidean plane in H. In Section 4 we prove that the metric normal to S at a non-characteristic point is a non-trivial arc of a geodesic, we deduce in Section 5 that d S is smooth in a neighborhood of S {characteristic points of S}. In Section 6 we compute the horizontal Hessian of d S on S. In Section 7 we show that several definitions of mean curvature for a surface d S in H, some of them involving d S, are equivalent. Some examples are given in Section 8. We consider the case when Π = {t = 0} is a plane, we compute Hess H d Π in H, not just on Π. Then, we compute Hess H d Σ, where Σ = {P : dp, O = } is the unit sphere at the origin, with respect to the Carnot-Charathéodory distance. This amounts, in fact, to compute the horizontal Hessian of d O, the distance function itself. In this paper we considered the Heisenberg group with the lowest dimension. This object is interesting in itself. For instance, it is related with the study of the isoperimetric inequality in the Euclidean plane []. More recently, it was realized that sub-riemannian structures modelled on the lowest dimensional Heisenberg group can be used to model the human visual system, see [3], [4], [7] the references quoted therein. In particular, [4] contains some material related to curvature in Heisenberg-like structures.. Notation preliminaries In this section, we collect some basic definitions known facts about the structure the geometry of H. There is a vast literature on sub-riemannian geometry Carnot groups. Justs to quote a few titles, we refer the reader to [], [6], [7], [], [4], [5], [6], [8]. The Heisenberg group H = H is the Euclidean space R 3 endowed with the noncommutative product x, y, t x, y, t = x + x, y + y, t + t + x y xy. Sometimes it is convenient to think of the elements of H as z, t C R. The Carnot- Charathéodory distance in H is the sub-riemannian metric that makes pointwise orthonormal the left invariant vector fields X Y, X = x + y t, Y = y x t. The vector fields X, Y do not commute, [X, Y ] = 4 t. The distance between two points P Q in H is denoted by dp, Q. The span of the vector fields X Y is called horizontal distribution, it is denoted by H. The fiber of H at a point P of H is H P = span{x P, Y P }. The inner product in H P is denoted by,. An important element of H s structure is the dilation group at the origin {δ λ : λ 0}, δ λ z, t = λz, λ t, z = x + iy By left translation, a dilation group is defined at each point P of H. The Heisenberg group is also endowed with a rotation group, which is useful in simplifying some calculations. For θ R, let R θ z, t = e iθ, t be the rotation by θ around the t-axis. Composing with left translation, one could define rotations around any vertical line x, y = a, b. R θ is an isometry of H its differential.

4 4 NICOLA ARCOZZI, FAUSTO FERRARI acts on the fiber H O as a rotation by θ. Under the usual identification between the Riemannian tangent space of H at O, T O H, the Lie algebra h of H, the differential of R θ can be thought of as a rotation on span{x, Y }, the first stratum of h. With respect to the basis {X, Y }, cosθ sinθ dr θ V = V sinθ cosθ whenever V span{x, Y }. With some abuse of notation, we denote dr θ by R θ. The distance between two points P Q in H is defined as follows. Consider an absolutely continuous curve γ in R 3, joining P Q, which is horizontal. That is, γt = atx γt + bty γt lies in H γt. Its Carnot-Charathéodory length is l H γ, l H γ = at + bt / dt The Carnot-Charathéodory distance between P Q, dp, Q, is the infimum of the Carnot- Charathéodory lengths of such curves. The infimum is actually a minimum, the distance between P Q is realized by the length of a geodesic. By translation invariance, all geodesics are left translations of geodesics passing through the origin. The unit-speed geodesics at the origin [], [] are 6 xσ = sinαw cosφσ φ γ O,φ,W σ = yσ = sinαw sinφσ φ tσ = φσ sinφσ φ + cosαw sinφσ φ cosαw cosφσ φ Here, W is a unitary vector in H O αw [0, π is unique with the property γ O,φ,W 0 = W. φ R, the geodesic is length minimizing over any interval of length π/ φ. In the case φ = 0, the geodesic is a straight line in the plane {t = 0}, xσ = cosαw σ, yσ = sinαw σ, we say that the geodesic is straight. If P = z, t z 0, then there exists a unique length minimizing geodesic connecting P O. If P = 0, t, t 0, i.e., if P belongs to the center of H then there is a one parameter family of length minimizing geodesics joining P O, obtained by rotation of a single geodesic around the t-axis. We call curvature of γ O,φ,W the parameter κγ O,φ,W = φ. We will see later how κγ O,φ,W is related to the curvature of a surface in H. The notion of curvature extends to all geodesics by left translations, κl P γ = κγ, where L P is left translation by P H in H. Given points P = z, t P = z, t, they are joined by a unique length minimizing geodesic, unless z = z. Let γ P,φ,α = L P γ O,φ,α The parameter φ is geometric in the following sense. π/ φ is the length of γ P,φ,W sgnφ is positive if only if the t-coordinate increases with σ. Recall that in H the orientation of the t-axis is an intrinsic notion, unlike the Euclidean space. If γ P,φ,W γ P,φ,W have an arc in common, then φ = ±φ, while no such easy relation exists for the parameter W. To change the orientation of a geodesic, observe that γ P,φ,W σ = γ P, φ, W σ Unlike the Euclidean case, a geodesic γ leaving O is not determined by its tangent vector at the origin, γ0 = W = cosαw X O + sinαw Y O. The extra parameter we need is κγ. The parameter κγ, if φ = κγ 0, is related to the dilation group as follows. δ λ γ O,φ,W is a reparametrization of the geodesic γ O,φ/λ,W. That is, all geodesics γ leaving 0 having fixed initial velocity γ0 = v in H O are dilated of each other. The case of the straight geodesics

5 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 5 is the limiting one, corresponding to λ 0. In a precise sense, then, the set of geodesics at O is parametrized by the unit circle in H O by the dilation group, a feature of H with no Euclidean counterpart. This simple fact will be a recurrent theme in the following sections. Let S be a smooth surface in H. We need some differential geometric notions about S. Definition.. Let S be a smooth surface in H let P S be a non-characteristic point. The Euclidean tangent space to S at P is denoted by T P S. The direction tangent to S at P is the -dimensional space V P S = T P S H P. The plane tangent to S at P, Π P S, is the Euclidean plane in H, tangent to S at P in the Euclidean sense. The direction normal to S at P is N P S = H P V P S. When Π P S is not a vertical plane in H, we define a positive direction normal to S at P, denoted by N + P S, as follows. Let C be the characteristic point of Π P S. N P S is the direction normal, with respect to,, H, to that of the straight line CP N + P S is the unit vector in the direction N P S which makes a counterclockwise angle π/ with the halfline CP, having origin in C. The group tangent to S at P [5] is the -dimensional vector space G P S = V P S T, where T = {0, 0, t : t R} is the center of H V P S is the one parameter subgroup of H generated by V P S. One point we want to make in the present paper is that G P S does not seem to capture the complexity of the geodesics set, while T P S does, in a precise sense. The following facts are easily established by direct calculation. Proposition.. Let S be a smooth surface in H, implicitely defined by gx, y, t = 0. Let P S be non-characteristic. Then, 7 V P S = span{y g X Xg Y }, N P S = span{xg X + Y g Y } = span{ H g} 3. The metric normal the distance function for a plane In this section, we list the main results about the distance function for a plane. Definition 3.. Let E be a subset of H, P E. The metric normal to E at P is the set N P S of the points Q H such that dq, E = dq, P. See also [], where a pathological occurrence of the metric normal to the unit sphere was used to study bi-lipschitz functions in H. Proposition 3.. Let P be a plane in H let P P be non-characteristic. characteristic point C, then, If P has a N P P = γ [ π P,/dP,C,N + P P dp, C, π dp, C] If P is a vertical plane, then N P P is the straight geodesic through P, in the direction N P P. Another way to state this result is the following. The projection onto the t = 0 plane of γ is the circle c having as diameter the line joining C, the projection of C, P, the projection of P. Proof. of the proposition. The case when P is a vertical plane is elementary. Consider the case when P has a characteristic point. In the statement of the proposition, everything is invariant under isometries, we can then left-translate everything so that P is the plane t = 0 C = O. Let P 0 be a point in P. Consider a point 0, 0, T on the vertical axis so that a legth-minimizing geodesic from 0, 0, T to 0, 0, T, say γ, intersects P at P. γ = γ P,φ,W is uniquely determined. We need to show that the parameters of γ are those in the proposition s statement, that γ is the metric normal. Step : γ has the right parameter φ. Since γ points upward, φ 0. By symmetry with respect to t = 0, dp, 0, 0, T = π/φ, half-length of γ. Inserting this in the equation for γ,

6 6 NICOLA ARCOZZI, FAUSTO FERRARI assuming after a rotation that P lies on the positive x-axis, we obtain that P = /φ, 0, 0. This shows that dp, 0 = /φ. Step. γ lies in the metric normal. Let S = 0, 0, T. Consider the ball B = BS, dp, S. The points Q where B touches a plane above S, parallel to P, are those for which the geodesic between S Q has parameter φ = π/dp, S i.e., the points having t- coordinates which is greatest. But we just saw that γ has parameter φ = /dp, 0 = π/dp, S, hence P is one of such points. It follows that dp, S minimizes the distance between S P, hence that γ lies on the metric normal. Step 3. γ has the right parameter W. Let c = cs be the projection of γ on P, assuming that γ starts from T goes towards T passing through P. Then, c0 = cπ/φ = 0, cπ/φ = P c is a circle having diameter OP. As s increases, cs runs the circle clockwise. As to the tangent vectors, ċs is the projection on the plane P = {t = 0} of the horizontal vector γs. Suppose P = /φ, 0, 0 lies on the positive x-axis. Then, ċπ/φ = 0,, hence γπ/φ = Y. We only have to show, then, that Y = N + P P. Now, the space tangent to P at P contains just one horizontal direction P is non-characteristic, this direction is that of the x-axis, X. Then, N P P = span{y }, hence, since N + P P has to point upward, N + P P = Y, as wished. Step 4. γ contains the metric normal. Suppose now that Q belongs to both N P P N P P. We show that, then, Q lies on the axis x = y = 0 that P P lie on a Euclidean circle centered at O in P. After a group translation, the plane L Q P touches a closed Carnot Charathèodory ball B centered at the origin in Q P Q P. This may happen only if Q P Q P both lie on the circle c of the points in B having highest or lowest t-coordinate, c = B L Q P. Hence, the characteristic point of P lies on the same vertical line as the center of B, this property is invariant under left translations. As a consequence, Q lies on x = y = 0 P, P are taken one into the other by a rotation around x = y = 0. For each point P in P {O}, consider the geodesic arc β P = γ [ π P,/dP,C,N + P P dp, C, π dp, C] which, by steps -3 lies in N P P. As P varies, the union of the sets β P fills up H {O}. Suppose now that Q belongs to N P P β P. Clearly, Q O, Q belongs to some β P N P P. By the considerations above, then, Q lies on x = y = 0 P is taken to P by a rotation around x = y = 0. The same rotation brings β P onto β P fixes Q, hence Q β P, a contradiction. By Proposition 3. 6 we have an explicit expression for the metric normal to a plane. Theorem 3.. The metric normal to Π = {t = 0} at P = z, t, z = x + iy, is the support of the geodesic arc x + cos σ z + y sin σ z 8 y γ P σ = uσ, vσ, sσ = + cos σ z x sin σ z, σ π z z σ z + sin σ z Let w = u + iv. Then, by 8, 9 The next lemma helps with calculations. w = z cos σ z s = z σ z + sin σ z

7 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 3 7 Lemma 3.. If p is the plane t = ax + by + c, its characteristic point is C = b/, a/, c Proof. The tangent space of p at x, y, t is spanned by, 0, a 0,, b. They are both horizontal iff a = y b = x. 4. The metric normal the oriented metric normal for a smooth surface Lemma 4.. Let S be a smooth surface in H, implicitely defined by gx, y, t = 0. Let P = x 0, y 0, t 0 S. Then, if g t P 0, Π P S has a characteristic point C gy P C = g t P, g xp g t P, t g y P 0 + x 0 g t P + g xp 0 g t P dp, C = Hg [X, Y ]g = Hg t g When g t P = 0, we say that Π P S has characteristic point at infinity. The quantity dp, C is purely non-euclidean. The proposition below says that it is related to another purely non-euclidean feature of H, that is, the role of the dilation group in the determination of geodesics. Theorem 4.. Let Ω be an open subset of H with C boundary S, in the Euclidean sense, locally defined by the equation gx, y, t = 0. Let P be a non characteristic point of S. Then the metric normal N P S to S at P is a proper subarc of the length-minimizing geodesic having N P S as tangent vector at P, with κγ = /dp, C. That is, N P S = suppγ = P suppη left translation by P, η = u, v, s, } { Y gp uσ = dp,c H gp cos σ { dp,c ησ = vσ = dp,c XgP H gp cos σ { dp,c } sσ = dp, σ C dp,c sin σ dp,c + XgP H gp sin σ dp,c + Y gp H gp sin σ dp,c When t gp = 0, i.e. when the Euclidean plane tangent to S at P is vertical, formula becomes XgP ησ = H gp σ, Y gp 3 H gp σ, 0 If P is characteristic S is smooth, then the metric normal at P contains P alone. First, we need the following lemma. Lemma 4.. Let Ω be an open subset of H with C boundary S, in the Euclidean sense. Then, for each non characteristic point P in S, there exist an open neighborhood U of P in S C > 0 such that, for any Q in U there is a closed Heisenberg ball B in Ω, containing Q on its boundary. Given lemma 4., we prove Theorem 4.. Proof. of the Theorem. Let Π P S the Euclidean plane tangent to S at the non characteristic point P, let B = BR, η a Heisenberg ball in Ω, containing P on its boundary γ be the length minizing geodesic between R P γ is unique, otherwise S would not be smooth at P. Choosing, if necessary, a point R on γ closer to P, considering BR, dr, P, we can assume that BR, dr, P B, by Lemma 4.4 below, that BR, dr, P does not intersect neither S nor Π P S it contains P on its boundary. Then, the segment of the geodesic γ with endpoints P R is contained in the metric normal to Π P S at P. By Theorem 3. by the equation of Π P S, we obtain that the equation of γ is that given in. }

8 8 NICOLA ARCOZZI, FAUSTO FERRARI We have now to show that every other point Q of N P S belongs to γ. Let Q be in N P S let ξ be the length minimizing geodesic between Q P. If ξ contains a subarc of γ starting at P, then ξ is contained in γ there is nothing to prove. Suppose that, on the contrary, ξ is a different geodesic. Then, possibly choosing a point Q nearer to P on ξ, we can assume that ξ is a portion of N P Π P S. On the other h, γ is portion of N P Π P S, but in Proposition 3. we showed that the metric normal to a plane is contained in a single geodesic, hence γ ξ are subarcs of the same geodesic. Finally, observe that N P S is a proper subarc of a length minimizing geodesic. Suppose, in fact, that N P S is a complete geodesic starting at P let P be its other endpoint. By 6, P P lie on the same vertical line. Then, R θ N P S is a geodesic whose length realizes the distance between P P, hence between P S. This means that R θ N P S = N P S, which is absurd. Proof. of Lemma 4.. Let P be a non characteristic point on S, let Π P S be the Euclidean plane tangent to S at P let γ be the metric normal to Π P S at P pointing inside Ω. Let s 0 be a number smaller than the length of γ consider the point C s on γ having distance s from P. Let τ s be the left translation carrying C s in O let S, P γ, Ω be the images of S, P γ, Ω, respectively. By continuity by the assumption that S is C, there is r > 0 such that, for each s < δr, there exists a Euclidean ball BEuc s D s, r of radius r contained in Ω having P on its boundary. We will show that there exists a Heisenberg ball BR, η centered in point R of γ, contained in BEuc s D s, r having P on its boundary. The ball τs BR, η = Bτs R, η fulfills the requirements of the lemma for Q = P. Since our procedure is stable with respect to P, it provides the required ball for Q in a neighborhood of P in S, with radius bounded from below by a positive constant. Since P is non characteristic, a subarc of γ starting at P is contained in BEuc s D s, r, but for the point P. It suffices to prove the following claim: BEuc s D s, r contains a Heisenberg ball having P on its boundary. We need an estimate for the Euclidean curvatures of the Heisenberg ball with center at the origin. Lemma 4.3. Let B0, s be the Heisenberg ball centered in 0 with radius s. For every point P B0, s, P = P x, y, t, x, y 0, 0, then there exists a positive number s 0 P such that the principal Euclidean curvatures in P are respectively, as s 0: k E = + o s + 6 φ k E = 3 + o. 4s3 where o 0 as s 0, uniformly w.r.t. φ C, for any fixed C > 0. Proof. From 8 α = φs we get the following parametrization of the Carnot ball of radius s: x = s cos θ sin α α y = s sin θ sin α t = s We can calculate the Euclidean curvatures k E, ke α α sinα. α k E g α = fα f α + s g α of the Carnot ball obtaining respectively

9 where fα = s sin α α where F α = sin α α Hence METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 4 9 We also need this fact. k E = f α g α f α + s g α, gα = s α sinα. As a consequence we get α k E G α = F α F α + sg α k E = F α G α F α + sg α, α sinα Gα =. On the other h as α 0 for fixed φ, α F α = α α3 + oα 4, G α = 3 5 α + oα 3, F α = α + oα 3 G α = 4 5 α + oα. k E = + o s + 6 φ k E = 3 + o. 4s3 Lemma 4.4. Let B = BA, r be a ball in H, Q BA, r γ be a geodesic from A to Q. Then, if A is a point of γ, BA, r da, A is contained in BA, r Q belongs to BA, r da, A. Proof. of lemma 4.4. Let R be a point in BA, r da, A, then dr, A dr, A + da, A = r. We now complete the proof of Lemma 4.. If s is small enough, by Lemma 4.3 there exists an open neighborhood V of P such that B0, s V is contained in B Euc D s, r. Consider now all Heisenberg balls BR, η with R γ. By Lemma 4.4, BR, η B0, s. As η 0, BR, η shrinks to P, hence BR, η is contained in B0, s V B Euc D s, r for η small enough. Clearly, BR, η contains P on its boundary. This proves the claim, hence Lemma 4.. Let S = Ω be the boundary of an open C set in H. The oriented metric normal to S at P S, ν P S, is the geodesic arc ν P S : I H such that δ P Sσ = σ whenever σ I the interval I is maximal with this property. An equation for ν P S can be explicitely be written if we know a parametrization for S. Consider the case t g 0 at P. Recall the geodesic η described in observe that it points upwards i.e., the t-coordinate increases with σ. Then, ν P Sσ = ησ if Ω is below S in a neighborhood of P in S, while

10 0 NICOLA ARCOZZI, FAUSTO FERRARI ν P Sσ = η σ if Ω is above S in a neighborhood of P in S. The case of points P where t g = 0 can be hled as in the Euclidean setting. Theorem 4. has a counterpart for the oriented metric normal. Theorem 4.. Let Ω be an open subset of H with C boundary S, in the Euclidean sense, locally defined by the equation gx, y, t = 0, with g smooth such that g 0 on S. Let P be a non characteristic point of S. Then the oriented metric normal ν P S to S at P is a proper subarc of the length-minimizing geodesic η having equation { } 4 4 tgp σ uσ = 4 tg Y gp cos H gp + XgP sin { ησ = vσ = 4 tg XgP cos 4 tgp σ H gp + Y gp sin } sσ = HgP H gp sin 4 tgp σ H gp 8 tgp { 4 tgp σ 4 tgp σ H gp 4 tgp σ H gp When t gp = 0, i.e. when the Euclidean plane tangent to S at P is vertical, formula becomes XgP ησ = H gp σ, Y gp 5 H gp σ, 0 ν P S has the same orientation as η, or the opposite one, according to the choice of Ω. If P is characteristic S is smooth, then the metric normal at P contains P alone. Next, we give an elementary description of the set of the metric normals endpoints. Let S = Ω be the boundary of an open, connected set in H assume that S is at least C. For P S, let Q be endpoint in Ω other than P of the metric normal N P S when N P S reduces to the point P, we set Q = P. The cut-locus in Ω of S the skeleton of Ω is the set K S of such points Q as P varies over S. Below, N P S refers to the portion of the metric normal at P which lies inside Ω. Here are some properties of K S. Lemma 4.5. Let P S be non characteristic. If N P S is not a straight line, then it is a proper subarc of a maximal length minimizing geodesic starting at P. Proof. Without loss of generality, assume that P = O. Let N P S = γ [0,b], where γ is a maximal length-minimizing geodesic starting at P having length a b. If b = a, then γb = Q belongs to the t-axis, hence R θ γ satisfies lengthr θ γ = lengthγ = dp, Q = dq, S, i.e., for all θ, R θ γ lies on N P S. This implies that R θ γ0 is perpendicular to T O S, for all θ, which is absurd. Lemma 4.6. Let C be a characteristic point of S. Then, N C S = {C}. Proof. Suppose that Q N C S = {C} is different from C. Then, S does not intersect BQ, dq, C meets its closure in C. Hence, S is not smooth at C. Proposition 4.. The cut locus K S of S has the following properties. i K S has empty interior. ii K S contains the characteristic points of S each characteristic point of S is an accumulation point of K S S. Proof. The assertion in i is a consequence of the following. Lemma 4.7. Let Q K S let γ be a geodesic from Q to S such that dq, S = lengthγ. Then γ {Q} is free of points of K S. Proof. of the lemma. Let P be the endpoint of γ in S. Let R be point of K S on γ, other than Q. By definition of K S, there exists a maximal geodesic η through R, having length a > dr, S, }

11 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 5 such that η0 = P S, R = ηb, where b = dr, S > a, for b < c a, dηc, S < c otherwise R would not be the endpoint of N P S. Clearly, dr, P = dr, P. By the triangle inequality, then dq, S dq, P dq, R + dr, P = dq, R + dr, P = dq, P = dq, S but this is possible only if R lies on the geodesic between Q P, so Q η, contradicting the fact that η s length does not realize the distance of its points from S past R. This proves the lemma. The first assertion in ii was proved in Lemma 4.6. Let now C be a characteristic point of S let P n be a sequence of noncharacteristic points of S tending to C. Such sequence exists because the horizontal distribution is not closed under Lie brackets by Frobenius Theorem. The formula for the geodesic normal in Theorem 4. implies that the length of N Pn S is no more that dp n, C n, where C n is the characteristic point of T Pn S. By continuity, dp n, C n dc, C = 0 as n. Let R n K S be the endpoint other than P n of N Pn S. Then, this proves ii. dr n, C dr n, P n + dp n, C dr n, C n + dp n, C 0 Proposition 4.. Let R H K S. Then there is a unique geodesic γ from R to S such that lengthγ = dr, S. i.e., there exists a unique P S such that R N P S. Proof. Suppose there are two such geodesics, γ γ, having the other endpoint, resp. P P, on S. We can not have P = P, otherwise, with a reasoning similar to that of Lemma 4.5, one shows that P is characteristic, hence γ reduces to a point R S. Suppose first that γ is not a straight line. Let Q K S be the endpoint of N P γ other than P. If γ dos not lie in the possibly non length-minimizing prolongment of γ, then, as in the proof of Proposition 4., we have that dp, Q < dp, Q = dq, S, a contradiction. The other possibility is that γ γ have in common the arc between Q R. This leads to a contradiction, too, since it would imply dq, P = dp, R dq, R < dp, R + dq, R = dp, Q, hence Q could not belong to N P. If γ = [P, R] γ = [P, R] are straight lines, either they lie on the prolongment of each other but this is not possible, otherwise R K S, or they have different directions. In the second case, there exists a point Q such that R [Q, P ] dq, S = length[p, Q] otherwise R would be the endpoint of N P, then R K S, as above, we would have dq, P < dp, Q, contradicting the fact that Q N P. A detailed study of the cut-locus K S of its properties closedness, Hausdorff measure, etcetera will be the object of further research. 5. Regularity of the distance function near a smooth surface In this section, we prove the regularity of the distance function for a smooth surface S in H give explicit formulae for its gradient for its Hessian, restricted to the surface itself. Let S be a non-characteristic smooth surface defined by gx, y, t = 0. We consider first the points of S where t gx, y, t 0. Let t = fx, y be the expression of S obtained through the implicit function theorem. Consider the map F : R R H R 3, 6 F u, v, σ = N u,v,fu,v Sσ

12 NICOLA ARCOZZI, FAUSTO FERRARI Lemma 5.. Let Ω be an open subset of H with C boundary S, in the Euclidean sense, locally defined by the equation gx, y, t = 0, t gx, y, t 0, with g in C. Let F be as in 6 let P = F u, v, 0 S. Then, 7 hence, 8 JF u, v, 0 = 0 sign t g Xg 0 sign t g Y g xg tg yg tg sign t g det JF u, v, 0 = dp, C uy g+vxg If P is characteristic, then det JF u, v, 0 = dp, C = 0. The fact that the determinant of the matrix JF u, v, 0 has an intrinsic geometric meaning can be explained as follows. The matrix is expressed with respect to the basis { u, v, σ } of R R, { x, y, t } of H R 3. The first basis has an intrinsic meaning. If we change the second basis into {X, Y, t }, which can be done with a linear transformation having determinant, the matrix representing JF u, v, 0 has an invariant form, which is given in the following theorem. Theorem 5.. Let Ω be an open subset of H with C boundary S, in the Euclidean sense, locally defined as gx, y, t = 0, with g in C g 0. Let F be the function defined in 6 let P = F u, v, 0 S, so that t gp 0. Then, the matrix representing JF u, v, 0 with respect to the basis { u, v, σ } of R R, {X, Y, t } of H R 3 is 9 Proof. of Lemma 5.. Suppose that 0 sign t g Xg 0 sign t g Y g Xg tg Y g tg 0 S = {u, v, t H : t = fu, v}, We will use the expressions Xf = Xg Y f = Y g, where gu, v, t = t fu, v. We have x, y, t = F u, v, τ = u, v, fu, v x, y, t where x, y, t = γ u,v τ γ u,v is the metric normal s left translate by P,i.e., by Theorem 4., x = 4 Xf sin α + Y f cos α y = 4 Y f sin α Xf cos α t = Hf 8 α sin α, where α = we have 4τ H f. Since x = u + x y = v + y t = fu, v + t + vx uy x u = + x u y u = y u t u = f u u, v y + t u + vx u uy u, x v = x v y v = + y v t v = f v u, v + y + t v + vx v uy v

13 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 6 3 x τ = x τ y τ = y τ t τ = +t τ + vx τ uy τ. We compute the derivatives of each coordinate, 4x u = Xf u sin α + Y f u cos α + α u Xf cos α + α u Y f sin α = Xf u sin α + Y f u cos α cos α Xf so x u τ=0 = 0. Analogously y u τ=0 = 0; x τ τ=0 = y u τ=0 = 0; y v τ=0 = 0; y τ τ=0 = 4τ Xf XfXfu H f 3 + Y fy f u sin α 4τ H f 3 XfXfu + Y fy f u, 4x v = Xf v sin α + Y f v cos α 4τ cos αxf H f 3 + sin αy f 4τ H f 3 XfXf u + Y fy f u, Xf H f ; 4x 4 τ = Xf cos α H f + Y f sin α 4 H f, 4y u = Y f u sin α Xf u cos α 4τ cos αy f sin αxf XfXfu H f 3 + Y fy f u, 4y v = Y f v sin α Xf v cos α 4τ cos αy f sin αxf XfXfv H f 3 + Y fy f v, Y f H f. 4y τ = 4τ H f 3 cos αy f sin αxf, 8t u = 4τ XfXf u + Y fy f u α sin α XfXfu + Y fy f u cos α, H f t u τ=0 = 0; 8t v = 4τ XfXf v + Y fy f v α sin α + XfXfv + Y fy f v cos α, H f t v τ=0 = 0,, eventually 8t τ = 4 H f cos α, t τ = 0.

14 4 NICOLA ARCOZZI, FAUSTO FERRARI Hence x u = + 4 Xf u sin α + 4 Y f u cos α 4 cos αxf x u τ=0 = ; x v τ=0 = 0; x τ τ=0 = y u τ=0 = 0; y v τ=0 = ; y τ τ=0 t u = f u + 4 Y f H f 4τ XfXfu H f 3 + Y fy f u sin αxf 4 x v = x v x τ = x τ Xf H f ; y u = y u y v = + 4 Y f v sin α 4 Xf v cos α 4τ cos αy f sin αxf XfXfv 4 H f 3 + Y fy f v, y τ = y τ, 4τ XfXfu H f 3 + Y fy f u τ XfXfu + Y fy f u α sin α XfXfu + Y fy f u cos α H f + vx u uy u y, with t u τ=0 = f u. t τ = Hf cos α + vx τ uy τ, t τ τ=0 = v Xf H f u Y f H f. Moreover t v = f v + 4τ XfXfv + Y fy f v α sin α XfXfv + Y fy f v cos α 4 H f + vx v uy v + x t v τ=0 = f v. As a consequence 0 hence J x y t u v τ τ=0 0 = 0 Xf H f Y f H f f u f v vxf uy f H f detjf u, v, 0 = H f.

15 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 7 5 Replacing in the matrix 0 f u f v by their expressions in terms of g s derivatives, we obtain 0 sign t g Xg JF u, v, 0 = 0 sign t g Y g sign t g as wished. ug tg vg tg uy g+vxg We now consider the set S 0 of those points x 0, y 0, t 0 in S where t gx 0, y 0, t 0 = 0. We consider two cases. Suppose that x 0, y 0, t 0 has a neighborhood U in S such that the metric normal at any point of U is a Euclidean straight line, which is normal in the Euclidean sense to S. Hence, in an open neighborhood of x 0, y 0, t 0, d S is the Euclidean distance, the smoothness of d S follows. The second case is that where x 0, y 0, t 0 is the limit of points x, y, t of S where t gx, y, t 0. Since gx 0, y 0, t 0 0, we can assume that, for x, y, t in a neighborhhod U of x 0, y 0, t 0, y gx, y, t 0. By the Implicit Function Theorem, restricting U if necessary, we can assume that gx, y, t = 0 if only if y = hx, t, where h is defined in an open neighborhood of x 0, t 0 in R. We consider a function H : R R H, defined in an open neighborhood of x 0, t 0, 0, 3 Hx, t, τ = ν x,hx,t,t Sτ Observe that H is a smooth map, in the Euclidean sense, at points where t g = 0 as well, by 3. Now, det JH = h t JF the latter, when τ = 0, is equal to h t dp, C = h t Hg t g Since 0 = g y h t + g t we have that = H h y g hence we have an expression analogous to, Lemma detjhx, hx, t, t = H hx, t. where Xh = Xhx, t y = h x + hh t Y h = Y hx, t y = xh t. On vertical points P, in particular, det JHP = h x + > 0 We are now able to prove Theorem 5.. Let S = Ω, Ω being an open C k subset of H, k, let d S : H [0, be the distance function, d S Q = dq, S Then, for any non-characteristic point P S there is a neighborhood U of P in H such that d S is C k in U. Also, H d S = Hg 5 Corollary 5.. There are C surfaces S such that d S is not even C, in the intrinsec meaning, at its characteristic points.

16 6 NICOLA ARCOZZI, FAUSTO FERRARI It suffices to consider the plane S = {t = 0}. By Theorem 5., H d S x, y, 0 = yx xy x + y which has no limit as x, y, 0 tends to 0, 0, 0. Proof. of Theorem 5.. Consider first the case of a portion of S represented by an equation of the type t = fx, y. The smoothness of d S is an immediate consequence of the Inverse Function Theorem, of the fact that P is non characteristic 8. We write x, y, t = F u, v, τ. We now show 5. Again by the usual Inverse Function Theorem, detjf u, v, 0 uy g+vxg + yg tg Y g uy g+vxg, uy g+vxg, JF x, y, t τ=0 = Y g, Xg yg, tg uy g+vxg + yg tg Xg Y g, Y g yg tg, Hence, the Euclidean gradient of d S on S is d S τ=0 = detjf u, v, 0 uy g + vxg, Then, elementary calculations show that Now, Lemma 4.,, imply y g t g, H d S τ=0 = Xd S X + Y d S Y τ=0 = JF u, v, 0 t g Xg X + Y g Y τ=0 JF u, v, 0 t g = dp, C dp, C = 5 follows. Consider now the case of a characteristic point free portion U of S having an equation of the type y = hx, t. By Lemma 5. by the Inverse function Theorem, d S is smooth in a neighborhood of U. Now, if P U is an accumulation point of points for which 5 holds, then 5 holds for P as well, by continuity. If there is a neighborhood V of P in S such that g t x, y, t = 0 i.e., for Q V, T Q S is a vertical plane, then d S is the Euclidean distance near P H g = g is the Euclidean gradient on V. Hence, 5 holds by the Euclidean theory. As a byproduct, we have the proof of a special case of a theorem of Monti Serra-Cassano [3]. Corollary 5.. Let S = Ω be the boundary of an open C subset of H let d S be the distance function for S. Then, d S satisfies the eikonal equation 6 H d S =, on the surface S. In the following section we improve on this corollary.

17 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP The Hessian of the distance function for a smooth surface Let g be a smooth function from H in R. We define the horizontal Hessian Hess H g the symmetrized horizontal Hessian Hess Hg of g. See, e.g., [5], [0] for a discussion of second order differential operators in the Heisenberg group. XXg Y Xg 7 Hess H g = XY g Y Y g Let XY g = XY g + Y Xg. 8 Hess XXg H g = XY g XY g Y Y g Theorem 6.. Let S be a non-characteristic smooth surface in H, defined by the equation t = fx, y. Let Q = XXf Y f XY f Xf Y f + Y Y f Xf Then, the horizontal Hessian of d S is Y f Hess H d S = H Q 4 Xf Y f f 5 H Y f f 9 [ 3 ] Xf Y f H Q + 4 Xf f 5 H f 3 [ Xf H f 5 Q 4 Y f H f 3 ] Xf H Q + 4 Xf Y f f 5 H f 3 In particular, the symmetrized horizontal Hessian of the distance function d S, restricted to the surface S, is Y f HessH d S = H Q 4 Xf Y f Xf Y f f 5 H f 3 H Q + Xf Y f f 5 H f 30 3 Xf Y f H Q + Xf Y f f 5 H Xf f 3 H Q + 4 Xf Y f f 5 H f 3, on S, 3 H d S = Q H f 3 = XXfY f XY fxfy f + Y Y fxf H f 3. In particular, if C is a characteristic point in S, then Hess H d S P = OdP, C as P C in S. Actually, the formulae hold with g in place of f, if g is a smooth function such that S = {g = 0}. In order to prove the theorem, we need some preliminary results. We denote by Hess k the Euclidean Hessian of a function k from R 3 in R. If K = k,..., k m maps R 3 in R m, we write Hess K = Hess k,..., Hess k m Consider the equation x, y, t = F u, v, τ Let us denote by Hess x, Hess y, Hess t the Euclidean Hessian matrices of xu, v, τ, yu, v, τ, tu, v, τ; let ξ = u, v, τ let ξ x, ξ y, ξ y be the coloumns of the Jacobian matrix of F computed when τ = 0. Let a = ξ x + yξ t b = ξ y xξ t. Lemma 6.. The entries of the horizontal Hessian of d S satisy the following relations. i We have 3 H f XXd S = F u F v HessF a, a Here, c c c 3 denotes the determinant of the 3 3 matrix having columns cj, j =,, 3.

18 8 NICOLA ARCOZZI, FAUSTO FERRARI ii We have 33 H f Y Y d S = F u F v HessF b, b iii XY τ Y Xτ satisfy { XdS XXdS + Y ds XY ds = 0 34 Xd S Y Xd S + Y d S Y Y d S = 0 Proof. The Inverse Function Theorem applied to the equation x, y, t = F u, v, τ gives a set of equations for the second derivatives of τ = d S, w.r.t. x, y, t. Let p, q be any two variables, possibly equal, chosen among x, y, t. Then 35 x u u pq + x v v pq + x τ τ pq = Hessxξ p, ξ q y u u pq + y v v pq + y τ τ pq = Hessyξ p, ξ q t u u pq + t v v pq + t τ τ pq = Hesstξ p, ξ q We solve the linear systems with respect to the second partial derivatives of τ, det x u, x v, Hessxξ p, ξ q y u, y v, Hessyξ p, ξ q t u, t v, Hesstξ p, ξ q 36 τ pq = det x u, x v, x τ y u, y v, y τ t u, t v, t τ. Recall from 8 that det x u, x v, x τ y u, y v, y τ t u, t v, t τ. = H f τ=0 Since XX = xx + 4y xt + 4y tt, since the determinant is linear with respect to each column HessF U, V is bilinear in U V, then 36 implies then, for τ = 0, H f XXd S = F u F v F τ XXdS = F u F v [HessF ξ x, ξ x + 4yHessF ξ x, ξ t + 4y HessF ξ t, ξ t ] = F u F v HessF a, a this shows i. ii can be proved in the same way, starting from the expression Y Y = yy 4x yt + 4x tt. iii follows taking derivatives w.r.t. X Y of the eikonal equation 6. Proof. of Theorem 6.. By i ii in Lemma 6., the expression of XXd S Y Y d S can be computed if we know HessF restricted to τ = 0 JF x, y, t τ=0 = ξ x ξ y ξ t. By iii, this allows us to compute XY d S Y Xd S as well. We apply the classical Implicit Function Theorem to x, y, t = F u, v, τ. Recall from Lemma 5. that the Jacobian of F, 37 JF u, v, 0 = J x y t u v τ τ=0 0 = 0 Xf H f Y f H f f u f v vxf uy f H f,

19 whose determinant is Its inverse matrix is 38 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 9 9 JF x, y, t τ=0 = J u v τ x y t detjf u, v, 0 = H f. τ=0 = + Xffu Xff v H f H Xf f H f Y ff u H + Y ffv f H Y f f fu H f fv H f H f H f. = ξ x ξ y ξ t If gx, y, t = t fx, y, we write XXf = XXg, XY f = XY g, so on. The following relations will be used below. Xf u = XXf Y f u = XY f 39 Xf v = Y Xf Y f v = Y Y f Recalling the derivatives of x, y, t computed in the proof of Lemma 5., we can compute hence x uu = 4 Xf uu sin α + 4 Y f uu cos α + 4 Xf u cos α + 4 Y f u sin αα u + 4 α u sin α Xf 4τ XfXfu H f 3 + Y fy f u 4 cos α Xf 4τ XfXfu H f 3 + Y fy f u u 4τ cos αxf XfXfu 4 H f 3 + Y fy f u 4τ sin αxf XfXfu 4 H f 3 + Y fy f u u, x uu τ=0 = 0. x vu = 4 Xf vu sin α + 4 Y f uv cos α + 4 Xf u cos α + 4 Y f u sin αα v + 4 α v sin α Xf cos α 4 Xf α v 4 4τ XfXfu H f 3 + Y fy f u 4τ XfXfu H f 3 + Y fy f u v 4τ cos αxf XfXfu 4 H f 3 + Y fy f u 4τ XfXfu H f 3 + Y fy f u v sin αxf x vu τ=0 = 0. x vv = 4 Xf vv sin α + 4 Y f vv cos α + Xfv cos α Y f v sin α 4τ cos αxf 4 H f sin αy f 4τ H f 3 XfXf u + Y fy f u v, x vv τ=0 = 0.

20 0 NICOLA ARCOZZI, FAUSTO FERRARI x uτ = 4 Xf 4 H f u cos α 4 Xf 4 H f sin αα u + 4 Y f 4 H f u sin α + 4 Y fα 4 u cos α H f, x uτ τ=0 = Y f H f 3 Xf uy f XfY f u = Y f XXfY f XfXY f. H f 3 x ττ = Xf H f τ cos α Xfα τ sin α H f + Y f H f τ sin α + Y fα τ cos α H f = Xfα τ sin α H f + Y fα τ cos α H f, x ττ τ=0 = 4Y f H f. x vτ = Xf H f v cos α Xfα v sin α H f + Y f H f v sin α + Y fα v cos α H f = Xf H f v cos α + Y f H f v sin α, x vτ τ=0 = Y f H f 3 Xf vy f XfY f v = Y f Y XfY f XfY Y f. H f 3 Moreover t uu = f uu + 4 so that τ XfXfu + Y fy f u α sin α + XfXfu + Y fy f u cos αu H f + vx uu uy uu y u y u, analogously t uu τ=0 = f uu + vx uu uy uu y u y u τ=0 = f uu ; t vu τ=0 = f vu, t vv τ=0 = f vv, t uτ τ=0 = vx uτ uy uτ y τ, t vτ τ=0 = vx vτ uy vτ + x τ, t ττ τ=0 = vx ττ uy ττ.

21 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 0 Hence Hessx τ=0 = Y f 0 0 XXfY f XfXY f H f Y XfY f XfY Y f. XXfY f XfXY f Y XfY f XfY Y f 4 H f Arguing in the same way we get Hessy τ=0 = Xf 0 0 XY fxf Y fxxf H f Y Y fxf Y fy Xf. XY fxf Y fxxf Y Y fxf Y fy Xf 4 H f f uu f vu vx uτ uy uτ y τ Hesst τ=0 = f vu f vv vx vτ uy vτ + x τ. vx uτ uy uτ y τ vx vτ uy vτ + x τ vx ττ uy ττ The horizontal Hessian matrix can be written now directly. A direct calculation shows that Y f a = H f, Xf Y f H f, Xf 40 H f viewed as a column vector b = Xf Y f H f, Xf H f, Y f 4 H f We then compute Hess τ=0 xa, a = Y f Xf H f 6 Q + Y f 4Xf H f 4 Hess τ=0 ya, a = Xf Y f H f 6 Q 4 Xf3 H f 4 Y f Hess τ=0 ta, a = H f 4 Q 4Xf Y f H f + 8 Xf Y f vy f + uxf + 4Xf vy f + uxfq H f 4 H f 6 Recall now that H f XXd S = = 0 Hessx τ=0 a, a 0 Hessy τ=0 a, a v Xf u Y f Hesst τ=0 a, a Y f H f 4 Q 4Xf Y f H f Analogous calculations yield the value of Y Y d S. In order to compute the mixed derivatives, we use 34. Finally, observe that, if S is a C surface, then as P tends to a characteristic point C of S. Hessd S τ=0 = O H f = OdP, C The estimate for the Hessian of d S on S cannot be extended outside S. In Section 8, we see that, for instance, when S is the plane t = 0, the behavior of Hess H d S is worst when we approach 0 near the line x = y = 0.

22 NICOLA ARCOZZI, FAUSTO FERRARI 7. Mean Curvature In this section, we make some considerations about the notion of mean curvature for a surface in H see, e.g.. [9]. Definition 7.. Let S be a smooth surface in H, defined by the equation gx, y, t = 0. The mean curvature of S at P S is XgP Y gp 4 h S P = X + Y H gp H gp The curvature is well defined for non characteristic points only. Here are some characterizations of mean curvature. The first is in terms of the distance function it is an immediate consequence of Theorem 6.. Theorem 7.. Let S = Ω be the boundary of an open, C set in H. characteristic point P in S, 43 H d S P = h S P Then, for any non R. Monti pointed out to us that the same result can be obtained by an explicit calculation, which does not rely on the knowledge of the distance s Hessian. In [9], it is shown that the mean curvature is the trace of a certain matrix M, which is not the Hessian of the distance function. The second is in terms of horizontal curves on S. First, we state an immediate consequence of the existence theorem for O.D.E. s. Lemma 7.. Let S be a smooth surface in H let P S be non-characteristic. Then, modulo restrictions reparametrizations, there exists a unique horizontal curve Γ through P on S. Suppose that S is defined by gx, y, t = 0. A parametrization of the curve is the solution of the O.D.E. s system { Γτ = Y gγτ XΓτ XgΓτ Y Γτ 44 Γ0 = P Theorem 7.. Let S be a smooth surface P S be non-characteristic. Let Γ be the horizontal curve on S through P, let p be the projection of H onto the z-plane, pz, t = z. Then, h S P is the mean curvature of pγ in R at pp. Proof. Suppose first that T P S is a non-vertical plane, that is, that in a neighborhood of P, S admits the parametrization S = {fx, y = t}. The horizontal curve Γ on S through P satisfies equation 44, hence Γ = x, y, t is a solution of ẋ = Y fγ ẏ = XfΓ ṫ = yy fγ + xxfγ Since Xf Y f are independent of t, we solve for x y, independently of t, we find the differential equation for γ = pγ, { ẋ = Y fγ ẏ = XfΓ Suppose that Y f 0 at P otherwise, we can let Xf 0 at P, since P is non characteristic. Parametrizing γ as y = γx, we find y = Xf Y f

23 METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP 3 The curvature of γ in the Euclidean plane is defined as k = d y = dx + y = = Xf Y f y + y 3/ Y f3 d H f 3 dx Y f [ x H f 3 Xf + y Xfy Y f x Y f + y Y fy Xf ] = XXf Y f XY f Xf Y f + Y Y f Xf H f 3 = H d S = h S P by Theorem 7.. We are left with the case when T P S is a vertical plane. If there is an open neighborhood U of P in S such that T Q S is vertical for Q U, then a portion of S containing P is a cylinder i.e., it has an equation of the form ϕx, y = 0 for some C function ϕ of the variables x, y, it is easy to see that, near P, Γ is a curve of the form { ϕx, y = 0 t = tx, y Then, γ = pγ has equation ϕx, y = 0. On the other h, in neighborhood of P in H, d S coincides with the Euclidean distance from S, hence the verification that k = h S P only involves Euclidean considerations which are left to the reader. Suppose now that T P S is a vertical plane, but that in S there is a sequence P n accumulating in P, such that T Pn S is a non-vertical plane. In the Heisenberg group, vectors belonging to different sections of the tangent bundle can be compared by means of left translations. Under the identification H R 3, hence of the tangent space at P H with R 3, the differential of the left translation by Q = a, b, c, L Q : P Q P, is the matrix dl Q = b a which is independent of the point where the differential is calculated. By definition of Carnot- Carathéodory distance, dl Q maps H P in H Q P, isometrically. Fix P H let φ : I H be a smooth, horizontal curve, such that φ0 = P. Let W be a section of the horizontal bundle along φ: W s H φs for s I, W is a smooth map. We define the derivative of W along φ at P as d H φ W s dl φ0 φs W s W 0 := lim ds s=0 s 0 s We give the notion of derivative of a vector field with respect to a vector in the Heisenberg group. Definition 7.. Let W be a section of the horizontal bundle H. Let V H P φ : I H be any horizontal curve such that φ0 = P φ 0 = V. We set H V W P := dh W φs. ds s=0 Lemma 7.. Let W = ax + by be a section of the horizontal bundle H φ : I H be any horizontal curve such that φ0 = P φ 0 = V. Then H V W P = V ap XP + V bp Y P

24 4 NICOLA ARCOZZI, FAUSTO FERRARI where V a V b are the derivatives of the functions a b in the direction V, respectively. Proof. Indeed dl φ0 φs W s W 0 s = dl φ0 φs aφs, bφs, φ saφs φ sbφs W φ0 s aφs, bφs, φ 0aφs φ 0bφs W φ0 = s aφs aφ0 bφs bφ0 = Xφ0 + Y φ0 s s as a consequence dl φ0 φs W φs W φ0 lim = V ap XP + V bp Y P s 0 s Let n S P the unit vector normal to S = Ω at P, in the Heisenberg sense, pointing outside Ω. If gx, y, t = 0 is a local parametrization of S, then XgP n S P = ± H gp X P + Y gp H gp Y P depending on the orientation of S. Theorem 7.3. The linear operator M P S : V H V n SP maps V P S into V P S. M P S acts on V P S as multiplication times h S P. Modulo a sign, the map M P S is the Weintgarten map for surfaces in the Heisenberg group. Proof. If W = n S P, we have H V n S P V P S In fact, differentiating n S P φs = with respect to s, n S P = ν P X P + ν P Y P, we obtain For s = 0, this relation becomes Since n S P = XfP H fp X P + Y fp 45 0 = ν φs d ds ν φs + ν φs d ds ν φs H V n S P = 0 = n S P, H V n S P H H fp Y P we get, for V = α XP + α Y P V P S, [ X XfP H fp Y XfP H fp ] X Y fp H fp Y Y fp H fp [ ] α α The linear map V H V νp is defined on the one-dimensional linear space V P S, hence it acts as multiplication by a constant k. Testing 45 on α, α = Y fp, XfP, i.e. on a basis of V P S, we see that k = Q H f 3 is the mean curvature of S at P. Observe that multiplication by the matrix in 45 acts on V P S as multiplication by the matrix trace.