ON A GENERAL COAREA INEQUALITY AND APPLICATIONS

Annales Academiæ Scientiarum Fennicæ athematica Volumen 27, 2002, 121 140 ON A GENERAL COAREA INEQUALITY AND APPLICATIONS Valentino agnani Scuola Normale Superiore, Piazza dei Cavalieri 7 I-56126 Pisa, Italy; magnani@cibs.sns.it Abstract. We prove a coarea inequality for Lipschitz maps between stratified groups. As a consequence we obtain a Sard-type theorem and the nonexistence of nontrivial coarea formulae between Heisenberg groups. In the case of real valued Lipschitz maps on the Heisenberg group we get a coarea formula using the Q 1 spherical Hausdorff measure restricted to level sets, where Q is the homogeneous dimension of the group. Introduction Recently, several classical problems of geometric measure theory in Euclidean spaces have been studied in general metric spaces, see for instance [1], [2], [3], [8], [19], [22], [30]. Following this path, one starts testing the generality of a geometric concept in spaces which lack some classical features of Riemannian metric spaces, but still keep enough structure. The stratified groups, also known as Carnot groups, are good examples of spaces to be investigated in this perspective. ore precisely they are graded nilpotent simply connected Lie groups. There are many recent contributions on the study of these groups and the more general Carnot Carathéodory spaces: [10], [13], [14], [15], [16], [20], [21], [23], [25], [28], [29], [31] the list is surely not complete). However, several classical facts of geometric measure theory are still not well understood. We mention for instance two open problems as the coarea formula for Lipschitz maps between stratified groups or the question of finding a reliable notion of current. So, the development of new general tools in this context is still at an early stage. In this paper we deal with the problem of coarea formula for Lipschitz maps between stratified groups. Some observations about this problem are in order. We distinguish between the case of real valued maps and the group valued maps, where both are defined on stratified groups. For real valued maps there are different coarea formulae in the literature, as for functions of bounded variation, [13], [16], [22], [25], and for smooth maps, [18]. In the first case the surface measure of the level sets is the perimeter measure, so one can ask whether it is possible to replace this measure with a Hausdorff type measure when the map is Lipschitz, as in the Euclidean case. This problem is raised in Remark 4.9 of [25], where another assumption is the use of the length metric namely the geodesic metric) 2000 athematics Subject Classification: Primary 28A75; Secondary 22E25.

122 Valentino agnani to build the spherical Hausdorff measure. As application of our coarea inequality we answer this question in the case of real valued Lipschitz maps on the Heisenberg group Theorem 3.11), considering the Q 1 spherical Hausdorff measure with respect to an arbitrary R -invariant metric Definition 3.7) and proving that the length metric is R -invariant Proposition 3.15). Another key tool to get the coarea formula is a blow-up theorem for the perimeter measure in the Heisenberg group, see Theorem 4.1 in [15]. The validity of this theorem for general stratified groups would imply the coarea formula in the same groups without relevant changes on our proofs. Unfortunately, the extension of the blow-up theorem to general stratified groups is still not well understood. In the general case of group valued Lipschitz maps the validity of a coarea formula is completely open and it seems that none of the classical methods can be used to solve this problem. Here we give a first partial answer, showing that the following coarea inequality holds 1) Φ Q P A f 1 ξ) ) dφ P ξ) A C P d x f) dφ Q x), where C P d x f) is the coarea factor of d x f see Definition 1.11) and Φ a generalizes the Hausdorff measure to a Carathéodory measure see Definition 1.9). However, it is interesting to observe that in some special cases 1) permits to get the nonexistence of nontrivial coarea formulae for group valued Lipschitz maps see Subsection 2.1). Our technique to prove 1) is based on differentation theorems for measures. Basically we generalize the blow-up method used in Lemma 2.96 of [4], reaching explicit estimates. We use the generalization of the Hausdorff measure Φ a to emphasize the general method adopted. Another application of 1) is a weak version of Sard s theorem. We prove that any Lipschitz map between stratified groups has a negligible set of singular points in almost every level set. We emphasize the attention on the fact that for Lipschitz maps, even in the Euclidean case, one cannot obtain more information. In fact, the classical stronger result, namely Sard s theorem, requires sufficiently smooth maps. Let us give a brief description of the paper. In Section 1 we introduce some known facts of geometric measure theory and some basic notions about stratified groups. In Section 2 we prove the coarea inequality, getting the Sard-type theorem and the nonexistence of nontrivial coarea formulae between different Heisenberg groups. In the last section we obtain the coarea formula for real valued Lipschitz maps on the Heisenberg group. Acknowledgments. I thank Luigi Ambrosio for his precious comments. I am grateful to Fulvio Ricci for his important suggestions about the geometry of the Heisenberg group. I also thank Bernd Kirchheim who proposed me to study the validity of the coarea formula between different Heisenberg groups.

On a general coarea inequality and applications 123 1. Definitions and some results In this section we first introduce some well-known tools of geometric measure theory in metric spaces, then we recall the main notions about stratified nilpotent Lie groups. Definition 1.1. For each metric space X, d) we denote the open ball with center x and radius r by U x,r = {y X dy, x) < r} and U r = U e,r, if some particular element e of the space is understood. Analogously we denote by B x,r = {y X dy, x) r} the closed ball. Throughout the paper we mean by measure on a space X a countably subadditive nonnegative set function defined on all the subsets of the space. The existence of a σ -algebra of measurable sets induced by µ is well known. Definition 1.2 Density points). Consider a metric space with a measure X, d, µ) and a measurable set A X. We define I A) as the set of points x X such that µa B x,r ) 1 µb x,r ) as r 0. We call every element of I A) a density point. Note that in a doubling space we always have µ A \ I A) ) = 0 when A is measurable, see for instance [11]. Now we recall briefly Carathéodory s construction in our particular case, see [11] for the general definition. Definition 1.3 Carathéodory measure). Let X, d) be a metric space and let F be a family of subsets of X. We fix a 0 and define for every t > 0 the measures { } Φ a t E) = β a inf diam D i ) a E D i, diam D i ) t, D i F, Φ a E) = lim Φ a t E), t 0 with E X and β a > 0. We assume that the family F has the following property 2) Θ 1 a H a Φ a Θ a H a, where Θ a > 0 and H a is the Hausdorff measure built with F = PX), β a = ω a /2 a, π a/2 ω a = and Γs) = r s 1 e r dr. Γ1 + a/2) For instance, if F is the family of closed or open) balls and β a = ω a /2 a the corresponding measure, Φ a satisfies the latter estimate with C a = 2 a. Indeed, in this case Φ a is the well-known spherical Hausdorff measure, which we denote by S a. In the sequel we will use a general coarea estimate which holds for Lipschitz maps in arbitrary metric spaces. In fact, after a work of Davies [9], the assumptions in paragraph 2.10.25 of [11] are needless. 0

124 Valentino agnani Theorem 1.4 Coarea estimate). Let f: X Y be a Lipschitz map of metric spaces and consider A X, with 0 P Q. Then the following estimate holds 3) Y H Q P A f 1 ξ) ) dh P ξ) Lpf) ω Q P ω P ω Q H Q A). The symbol denotes the upper integral see for instance [11]). We can easily transform 3) using our measures Φ a from Definition 1.3, obtaining 4) Y Φ Q P A f 1 ξ) ) dφ P ξ) Lpf) ω P ω Q P ω Q Θ Q P, Θ P Θ Q Φ Q A). 1.1. Brief digest on stratified groups. Here we recall some notation and basic facts on stratified groups. Consider a graded nilpotent simply connected Lie group G with Lie algebra G, where G is the direct sum of subspaces V i, i N. An important generating condition is assumed: [V i, V 1 ] = V i+1 and V i = 0 for i > n. The integer n is called the degree of nilpotency of the group and V 1 represents the space of the so-called horizontal directions. The stratified structure of the algebra allows us to define a one parameter group of dilations δ r : G G as δ r v) = n ri u i, where v = n u i, u i V i and r > 0. So δ r δ s = δ rs holds for r, s > 0 and δ r is a homomorphism of the algebra G. The same group of dilations is easily transfered on G by the exponential map exp: G G, which is a diffeomorphism for simply connected Lie groups. Under these assumptions one can define a left invariant distance on G, which is homogeneous with respect to dilations. Namely, we have 1) dx, y) = dux, uy) for every u, x, y G, 2) dδ r x, δ r y) = r dx, y) for every r > 0. A distance with these properties is called homogeneous distance. There are many bi-lipschitz equivalent homogeneous distances one can define. Among homogeneous metrics there is the length metric, that is, for each couple of points x, y G, there exists a rectifiable curve which connects them, whose length is equal to the distance between the points. We fix a scalar product on the space G, so we can define the Lebesgue measure L q, where q is the topological dimension of G. With a slight abuse of notation we denote by L q the measure exp L q defined on G and we do the same for all the Euclidean Hausdorff measures defined on G. The group operation preserves the volume, so L q is left invariant with respect to the translations of the group. By definition of dilation it is not difficult to see that L q B r ) = r Q L q B 1 ), where Q = n i dimv i) and B r is a ball of radius r with respect to the homogeneous distance. Hence, the measure H Q built with the homogeneous distance is proportional to L q they are both Haar measures). Clearly the Hausdorff dimension of the space with respect to the homogeneous distance is Q. Now it is clear that one can identify G with G, thinking as a unique object R q with two different structures: the addition of the vector space with the

On a general coarea inequality and applications 125 Euclidean distance and the operation of the group with the homogeneous distance. In fact, the group operation of G can be translated on G by the exponential map and the Baker Hausdorff Campbell formula yields an explicit polynomial function for the operation. See [6], [17] and [18] for more details. In our study we consider two stratified groups, so we fix another one with homogeneous distance ϱ and topological dimension p. We denote by its Lie algebra, which is the direct sum of subspaces W j, j = 1,..., m. As above, the Lebesgue measure L p and the Hausdorff measure H P with respect to the homogeneous distance ϱ are defined with P = m i dimw i). We will not use different notation to distinguish between dilations of and that one of G. 1.2. Differentiability and coarea factor. We state the extension of Rademacher s theorem on stratified groups and introduce the notion of coarea factor for suitable linear maps between stratified groups. Definition 1.5. Let L: G be a map of stratified groups. We say that L is homogeneous if δ r Lx) = Lδ r x) for every r > 0. Definition 1.6 G-linear maps). We say that a map L: G is G-linear if it is a homogeneous Lie group homomorphism. The G-linear maps generalize the linear maps of Euclidean spaces. Indeed they coincide with linear maps when the stratified groups are Euclidean spaces abelian simply connected Lie groups). An elementary characterization holds: every G-linear map is Lipschitz in the metrics of the groups and conversely any Lipschitz Lie groups homomorphism is a G-linear map see [21]). Definition 1.7. We say that a map f: A G is differentiable at x I A) A if there exists a G-linear map L: G such that ϱ fx) 1 fy), Lx 1 y) ) 5) lim = 0. y A, y x dx, y) The following generalization of Rademacher s theorem on stratified groups is due to Pansu [27], assuming that the domain of the Lipschitz map is open. An extension of this theorem to the general case of arbitrary domains is done in [21] and [31]. It is helpful to note that this improvement cannot be a trivial consequence of a Lipschitz extension theorem, because this one lacks in stratified groups. Theorem 1.8 Differentiability). Every Lipschitz function f: A G is differentiable H Q -almost everywhere. In the case of stratified groups we will consider a particular class of Carathéodory measures, introduced in Definition 1.3. Definition 1.9. We fix a compact neighbourhood D G of the unit element and define the family F 0 = {xδ r D x G, r > 0}. Given a 0 we apply the construction of Definition 1.3 with F equal to F 0 or PG), denoting with Φ a the corresponding measure on G.

126 Valentino agnani Proposition 1.10. The measure Φ a defined above satisfies the estimate 2) and the following ones 1) Φ a δ r E) = r a Φ a E) for E G, r > 0, 2) Φ a t δ r E) r a Φ a t E), for E G, r, t > 0 and r < 1, 3) Φ a xe) = Φ a E), for any x G left invariance). Proof. In case F = PG) clearly Φ a = H a, so 2) is trivial. If F = F 0 it is enough to observe that there exist two positive constants c 1 and c 2 such that B c1 D B c2 and compare Φ a with S a. Properties 1) and 2) follow from the fact that for any s, r > 0 and x G one has diam δ r E) = r diam E) and δ s xδ r D) = δ s xδ sr D F. Finally, by the left invariance of the homogeneous metric property 3) follows. Throughout the paper we will refer to the measures Φ a of Definition 1.9 defined on G and. Next, we adapt the implicit notion of coarea factor in [2] to our framework. Definition 1.11 Coarea factor). Consider a G-linear map L: G, with Q P. The coarea factor C P L) of L is the unique constant such that 6) Φ Q B 1 )C P L) = Φ Q P B 1 L 1 ξ) ) dφ P ξ). In view of the following proposition the definition of coarea factor is well posed. Proposition 1.12. For each G-linear map L: G there exists a unique nonnegative constant C P L) such that 6) holds. oreover, the number C P L) is positive if and only if L is surjective non-singular) and in this case we have ) 7) C P L) = ΦQ P L 1 0) B 1 H q p L 1 0) B 1 )detll ) 1/2. Proof. Consider the dilation δ r restricted to the subspace LG) and note that the jacobian of δ r is r P, where P = m i dim LV i ) ). It follows that H p Br LG) ) = r P H p B1 LG) ), where H p stands for the Euclidean Hausdorff measure p is the topological dimension of LG)). In case L is not surjective it follows that P = m i dim LV i ) ) < m i dimw i ) = P, hence the Hausdorff dimension of LG) is less than P, and by 2) we get C P L) = 0. Now assume that L is surjective. We start proving that Φ Q P is proportional

On a general coarea inequality and applications 127 to the Euclidean Hausdorff measure H q p on the subspace N = L 1 0). Note that N has topological dimension q p and a graded structure N = N 1 N 2 N n, where N i is a subspace of V i. Reasoning as above we have that H q p B r N) = r Q, H q p B 1 N), where Q = n i dimn i). The fact that L is surjective implies that n m, dimv i ) dimw i ) and dimn i ) = dimv i ) dimw i ), i = 1,..., m, so n m Q = i dimn i ) = i dimv i ) dimw i ) ) n + i dimv i ) = Q P. i=m+1 It is clear that Φ Q P N is a left invariant measure, because the metric d on G is left invariant. The measure H q p N is left invariant because translations preserve the volume even if they are restricted to subspaces. It follows that Φ Q P N and H q p N are proportional: 8) Φ Q P N = α Q,P H q p N, where α Q,P = Φ Q P NB 1 )/H q p NB 1 ). For any ξ we can write L 1 ξ) = xn, where Lx) = ξ, so taking into account that left translations are isometries, one concludes that the constant α Q,P remains unchanged if one replaces N with L 1 ξ) in formula 8). As a result we find that the measure νa) = Φ Q P A L 1 ξ) ) dφ P ξ) is positive on open bounded sets, while inequality 4) guarantees that ν is finite on the sets A G with Φ Q -finite measure. By a change of variable involving left translations it is not difficult to see that ν is a left invariant measure on G, so there exists a positive constant C P L) such that ν = C P L)Φ Q. Now we want to compute explicitly the factor C P L). We know that Φ P is proportional to the Lebesgue measure L p on. Thus, we can replace these equalities in the definition of coarea factor obtaining Φ Q P B 1 L 1 ξ) ) dφ P ξ) = α Q,P β Q,P H q p B1 L 1 ξ) ) dl p ξ), where β Q,P = Φ P B 1 )/L p B 1 ). From the classical coarea formula we get Φ Q P B 1 L 1 ξ) ) dφ P ξ) = α Q,P detl L ) 1/2 Φ P B 1 ), finally Definition 1.11 leads us to the claim. Remark 1.13. If G and are Euclidean spaces it follows that C P L) = detll ) 1/2, where L is a linear map. Therefore, the coarea factor coincides with the classical jacobian of the Euclidean coarea formula. For G-linear maps, by 4), we always have 9) C P L) ω Q P ω P Θ Q P Θ P Θ Q ω Q Φ Q LpL). B 1 )

128 Valentino agnani 2. Coarea inequality This section is devoted to the proof of coarea inequality; as a corollary we show a Sard-type theorem for Lipschitz maps between stratified groups. In the sequel we will fix a closed set A G and a Lipschitz map f: A. Indeed one can always extend a Lipschitz map to the closure of its domain when the target is a complete metric space. For any closed set A G the map ξ Φ Q P t A f 1 ξ) ) is a Borel map, hence we can state the following definition. Definition 2.1. For any t > 0 we define a measure on G as follows: for any D G ν t D) = Φ Q P t D A f 1 ξ) ) dφ P ξ). By the estimate 4) the measure ν t is locally finite uniformly in t > 0. The next lemma is a simple variant of Lemma 2.9.3 in [11]. Lemma 2.2. Let ν be a locally finite measure on a doubling space X, µ) which is absolutely continuous with respect to µ and let α be a positive number. Then for any subset A it follows that νa) αµa). { x X lim inf r 0 } νb x,r ) µb x,r ) < α Definition 2.3. For each map f: A and x 0 A, we define the r - rescaled of f at x 0 as the map f x0,r: δ 1/r x 1 0 A) defined as f x0,ry) = δ 1/r fx0 ) 1 fx 0 δ r y) ). Proposition 2.4. Consider a map f: A, a differentiability point x 0 I A) and a sequence of positive numbers r j ) which tends to zero. For every ζ, j N define the compact set K j ζ) = m j B1 f 1 x 0,r m ζ) δ 1/rm x 1 0 A)). Then it follows that j 1 K j B 1 d x0 f) 1 ζ). Proof. Pick an element y j 1 K j, getting a subsequence ϱ l ) of r j ) and a sequence y l ) such that y l B 1 fx 1 0,ϱ l ζ) δ 1/ϱl x 1 0 A), y l y. Thus, by Theorem 1.8 and equation 5) it follows that f x0,ϱ l y l ) d x0 fy), but f x0,ϱ l y l ) = ζ for every l N, then ζ = d x0 fy).

On a general coarea inequality and applications 129 Theorem 2.5 Density estimate). In the above assumptions, for any t > 0 we have 10) lim inf r 0 ν t B x0,r) Φ Q B x0,r) C P d x0 f). Proof. We start considering the quotient ν t B x0,r)r Q = Φ Q P t A Bx0,r f 1 ξ) ) r Q dφ P ξ). The map T x0,r: G G, y x 0 δ r y is the composition of an isometry and a dilation δ r. Thus, choosing r < 1, by property 2) of Proposition 1.10 it follows that Φ Q P t A Bx0,r f 1 ξ) ) = Φ Q P t Tx0,r Ax0,rξ) )) r Q P Φ Q P t Ax0,rξ) ), where A x0,rξ) = {y B 1 fx 0 δ r y) = ξ} δ 1/r x 1 0 A). This implies ν t B x0,r)r Q Φ Q P t Ax0,rξ) ) r P dφ P ξ). Defining R x0,r:, ξ δ 1/r fx0 ) 1 ξ ) = ζ and using property 1) of Proposition 1.10 we obtain R x0,r) Φ P ) = r P Φ P ; hence ν t B x0,r)r Q By the definition of r -rescaled function we have Φ Q P t Ax0,rRx 1 0,rζ) ) dφ P ζ). 11) A x0,r R 1 x 0,rζ) ) = { y B 1 fx 0 δ r y) = fx 0 )δ r ζ } δ 1/r x 1 0 A) = B 1 fx 1 0,rζ) δ 1/r x 1 0 A), ν t B x0,r)r Q Φ Q P t B h B1 f 1 x 0,rζ) δ 1/r x 1 0 A)) dφ P ζ). The family of functions {f x0,r} r>0 is uniformly Lipschitz with bound Lpf) = h on the Lipschitz constants; hence f x0,rb 1 ) B h for any r > 0. Now choose a sequence r j ) such that r j 0 and for each j N define the functions 12) gjζ) t = Φ Q P t B1 fx 1 0,r j ζ) δ 1/rj x 1 0 A)) and the following decreasing sequence of compact sets K j ζ) = m j B1 f 1 x 0,r m ζ) δ 1/rm x 1 0 A)).

130 Valentino agnani In view of Proposition 2.4 we obtain j 1 K j ζ) B 1 L 1 ζ), where L = d x0 f is the differential of f at x 0. By results of paragraph 2.10.20 in [11] it follows that 13) lim sup gjζ) t lim j j ΦQ P t Φ Q P τ Kj ζ) ) Φτ Q P B1 L 1 ζ) ) K j ζ) j 1 with τ < t. Each measure Φ a τ, with τ, a > 0, is finite on bounded sets, then the sequence of nonnegative functions gj t) j N is uniformly bounded by Φτ Q P B 1 ) on B h. This fact together with Fatou s theorem and inequality 13) implies 14) lim sup gjζ) t dφ P ζ) Φ Q P τ B1 L 1 ξ) ) dφ P ζ). j B h B h Joining inequalities 11), 12) and 14), and taking into account the inequality Φ a τ Φ a it follows that lim inf r 0 ν tb x0,r)r Q lim sup ν t B x0,r j )r Q j Φ Q P B 1 L 1 ζ) ) dφ P ζ). j From Definition 1.11 we obtain 15) lim inf r 0 ν tb x0,r)r Q C P d x0 f)φ Q B 1 ). Finally, by inequality 15) and the property 1) of Proposition 1.10 the proof is complete. Theorem 2.6 Coarea inequality). Let A G be a measurable set and consider a Lipschitz map f: A. Then we have Φ Q P A f 1 ξ) ) dφ P ξ) C P d x f) dφ Q x). Proof. We start proving the measurability of gx) = C P d x f). For any t > 0 we consider the Borel function defined on G-linear maps A L Φ Q P t L 1 ) 0) B 1. The limit as t 0 is a measurable function, so by the measurability of x d x f and the representation 7) one concludes this verification. Furthermore, in view of 9) the map g is bounded. Now we define A I A) A as the set of )

On a general coarea inequality and applications 131 differentiability points, hence by Theorem 1.8 we have Φ Q A \ A ) = 0 and by 4) it follows that 16) Φ Q P A f 1 ξ) ) dφ P ξ) Φ Q P A f 1 ξ) ) dφ P ξ). Consider a measurable step function ϕ = k α i1 Ai g, α i 0, k A i = A disjoint union). By estimate 10), for any i = 1,..., k we have lim inf r 0 ν t B x,r ) Φ Q B x,r ) α i for each x A i. Inequality 4) implies the absolute continuity of the measure ν t with respect to Φ Q, so for every i = 1,..., k we can apply Lemma 2.2, getting ν t A i ) α i Φ Q A i ). Since our estimates are independent of t > 0, we can allow t 0. Therefore, summing over i = 1,..., k we find Φ Q P A f 1 ξ) ) dφ P ξ) ϕx) dφ Q x). A By 16) and the measurability of g the proof is complete. 2.1. Applications. Sard s classical theorem states that the image of the singular set of a smooth map is negligible. This statement is easily generalized to the case of Lipschiz maps between stratified groups f: G, when P > Q. In fact, the area formula holds under these assumptions [21], [29]. The subtle question comes up when one deals with the case P < Q. A consequence of Sard s classical theorem is that for almost every element of the target, the intersection of its counterimage with the singular set of the map is empty. A weaker version of this statement adapted for Lipschitz maps is as follows. Theorem 2.7 Sard-type theorem). Let f: A be a Lipschitz map, with A G. Define the set of singular points S 0 = {x A d x f is not surjective}. Then, for H P -a.e. ξ it follows H Q P S 0 f 1 ξ) ) = 0. The proof follows immediately from the coarea inequality. As a result, in almost every fiber the set of non-singular points has full measure. Let us notice that for a regular value t R of a countinuously differentiable function f: G R the set S 0 f 1 t) may be not empty. In this case singular points coincide with characteristic points of the level set. By the fact that real valued continuously differentiable maps on G are Lipschitz with respect to the homogeneous distance, we can apply our weak version of Sard s theorem, obtaining that in a.e. fiber the set of characteristic points is negligible for the Q 1 Hausdorff measure. This observation fits a recent general result due to Balogh [5], where it is proved that any C 1 hypersurface in the Heisenberg group has a negligible set of characteristic points with respect to the Q 1 Hausdorff measure.

132 Valentino agnani Our coarea inequality can also be used to get information about the existence of trivial coarea formulae between stratified groups. In fact, it may happen that for two particular stratified groups, all G-linear maps L: G, with Q P, are not surjective. Hence the coarea factor of the differential is always vanishing. In this case, by 1) we have Φ Q P A f 1 ξ) ) dφ P ξ) = 0, so the coarea formula becomes trivial. This trivialization happens considering coarea formulae between different Heisenberg groups, as the following theorem shows. Theorem 2.8. Let L: H n H m be a G-linear map, with n > m. Then L is singular, i.e. it is not surjective. Proof. We use complex notation to represent the Heisenberg group, writing z, s), w, s) C n R as elements of H n. The groups law reads as follows z, s) w, t) = z + w, s + t + 2 Im z w) ). By the homogeneity of L we have Lz, s) = Az, αs), where A: C n C m real linear map and α R. The homomorphism property implies that is a in particular Lz, s) Lw, t) = L z + w, s + t + 2 Imz w) ), α Imz w) = ImAz Aw ) for any z, w C n. Taking an element u in the kernel of A, for z = u and w = iu we get α = 0, then L is not surjective. In view of Theorem 2.8 we infer that there cannot exist nontrivial coarea formulae between different Heisenberg groups. 3. Perimeter measure and coarea formula on H n Our objective in this section is the proof of the coarea formula for Lipschitz maps in the Heisenberg group, where we replace the perimeter measure of the level sets with the spherical Q 1 Hausdorff measure. But not all homogeneous metrics can be chosen to build the spherical Q 1 Hausdorff measure and fit the coarea formula. In fact, we have found a particular class of homogeneous metrics which can be used for our aim. These metrics possess an invariant property with respect to horizontal isometries Definition 3.5). Now, we recall briefly that H n is a stratified group endowed with a structure of 2n + 1-dimensional real vector space. Identifying the Lie algebra with the group we have the stratification H n = V 1 V 2, where the horizontal space V 1 is generated by the vector fields X j = ξ j + 2ξ n+j ξ 2n+1, Y j = ξ n+j 2ξ j ξ 2n+1, j = 1,... n,

On a general coarea inequality and applications 133 and V 2 is spanned by the vertical direction Z = / ξ2n+1 = 1 4 [X j, Y j ]. In this case an easy computation shows that the Hausdorff dimension of H n is Q = 2n+2, see for instance [6], [17]. We recall the notion of function of bounded variation in this context, see [7], [13], [15], [16]. Definition 3.1. Let Ω H n be an open set and let f: Ω R be a summable function. We say that f is a function of H-bounded variation in Ω if { H f Ω) := sup Ω } f div H ϕ dl 2n+1 ϕ C 1 0 Ω) 2n, sup ϕ 1 <, Ω where div H ϕ = n j=1 X jϕ j + Y j ϕ n+j. The set of all functions of H-bounded variation in Ω will be denoted by BV H Ω). We denote by BV H,loc H n ) the set of locally summable functions which have H-bounded variation when restricted to any relatively compact open set contained in Ω. By the Riesz representation theorem there exists a vector valued Radon measure H f on Ω such that 17) Ω f div H ϕ dl 2n+1 = ϕ, d H f Ω for any ϕ C0Ω) 1 2n. The symbol H f denotes the total variation of the measure H f. If 1 E BV H,loc H n ) we say that E is a set of H-finite perimeter and denote with E H the corresponding variation measure H 1 E, namely the H-perimeter measure. oreover we can express the vector variation of E as H 1 E = ν E H 1 E, where ν E is a E H -measurable function with unit modulus at E H -a.e. point, namely the generalized inward normal of E. Next, we state the BV-coarea formula see [13], [16]). Theorem 3.2. For each g BV H,loc H n ) and every bounded open set U H n we have 18) H g U) = E t H U) dt, where E t = {y H n gx) > t}. R An immediate consequence is that for a.e. t R the set E t has H-finite perimeter in U. By Theorem 1.8 one verifies that all Lipschitz functions have distributional derivatives which coincide almost everywhere with the differential. Furthermore we have 19) D H f L 2n+1 = H f, where D H f = n X ig) 2 + Y i g) 2) 1/2 is the modulus of the differential.

134 Valentino agnani Now we introduce the H-reduced boundary, denoted by H E. This is defined as the set of points x H n such that there exists 20) lim r 0 U x,r ν E d E H E H U x,r ) = ν E x) and ν E x) = 1. By a recent result in [1] we can say that the H-reduced boundary H E is defined independently of the homogeneous metric up to E H -negligible sets and 21) E H H n \ HE) = 0. In fact, all the homogeneous metrics are bi-lipschitz equivalent and the asymptotically doubling property of the perimeter measure stated in Corollary 4.5 of [1] is bi-lipschitz invariant. So, if d is a homogeneous metric in H n, by Theorem 2.8.17 of [11], the measure E H and the family of closed balls form a Vitali relation in the terminology of [11]). It follows that condition 20), in the metric d, holds for E H -a.e. point of H n. The crucial fact we use is a particular consequence of the blow-up Theorem 4.1 in [15]. We state this theorem as follows. Theorem 3.3. Given a set E of H-finite perimeter and x H E, then ν E x,r E x,r H : ν E x)h 2n Π x as r 0. The rescaled set E x,r is defined as E x,r = δ 1/r x 1 E), where Π x is a vertical hyperplane in H n of the form Π x = { exp n ξ ix i + ξ n+i Y i + ξ 2n+1 Z ) H n ξ, α x = 0 } and α x R 2n+1 \ {0}, α x ) 2n+1 = 0. Theorem 3.4. Let d be a homogeneous metric in H n and assume that E is a set of H-finite perimeter. Then for E H -a.e. x H n we have 22) lim r 0 E H U x,r ) r Q 1 = H 2n U 1 Π x ), where the open balls U x,r are defined with respect to the metric d. Proof. By equation 21) it is enough to prove that the limit 22) holds for each point x H E. 2n The measure H Π x is finite on compact sets, so H 2n Π x U t ) = 0 for a.e. t > 0. We fix some t > 0 with H 2n Π x U t ) = 0, so by Theorem 3.3 it follows that ν E x,r d E x,r H ν E x)h 2n U t Π x ) as r 0. U t By a direct calculation, using formula 17) and the homogeneous property of d, that is U tr = δ r U t, we obtain ν E x,r d E x,r H = 1 U t r Q 1 ν E d E H. U x,tr

On a general coarea inequality and applications 135 Finally, using the definition of H-reduced boundary we get E H U x,tr ) lim r 0 tr) Q 1 = 1 1 lim tq 1 r 0 r Q 1 ν E d E H U x,tr = H 2n U t Π x ) t Q 1 = H 2n U 1 Π x ). The latter equality is due to the fact that Π x contains the vertical direction, hence the dilations scale with a power Q 1. This completes the proof. Next we prove that the limit 22) is independent of x for a class of homogeneous metrics. In the following definition we use the complex representation for elements of H n as z, s) C n R. Definition 3.5. We say that T : H n H n is a horizontal isometry if there exists a unitary operator U: C n C n such that for any z, s) H n we have T z, s) = Uz), s ). We define R as the set of all horizontal isometries. Remark 3.6. Let us observe that any horizontal isometry is a G-linear map, i.e. it is a group homomorphism and it commutes with dilations. Furthermore, since any horizontal isometry is in particular an isometry of R 2n+1 with respect to the Euclidean norm, the Hausdorff measures H a on Hn, a 0, are preserved. Definition 3.7. A homogeneous metric d on H n is called R -invariant if T U 1 ) = U 1 for every T R, where U 1 is the unit open ball in the metric d. Lemma 3.8. Given an R -invariant metric d on H n, there exists Υ d > 0 such that 23) Υ d = H 2n U 1 Π α ), for any Π α = { exp n ξ ix i + ξ n+i Y i + ξ 2n+1 Z ) H n ξ, α = 0 } with α R 2n+1 \ {0} and α 2n+1 = 0. Proof. Observing that Π α is independent of the length of α, it is enough to observe that for any α, β R 2n+1 \ {0}, with α = β = 1, α 2n+1 = β 2n+1 = 0, there exists a horizontal isometry T : H n H n such that T Π α ) = Π β. Thus, by the R -invariance, we have H 2n U 1 Π β ) = H 2n T U 1 Π α ) ) = H 2n U 1 Π α ). Theorem 3.9. Let E be a set of H-finite perimeter and let d be an R - invariant metric on H n. Thus, we have 24) E H = γ Q S Q 1 HE, where S Q 1 is the spherical Hausdorff measure in the metric d and γ Q = Υ d /ω Q 1.

136 Valentino agnani Proof. From Theorem 3.4 and Lemma 3.8, observing that all the blow-up hyperplanes Π x are of the form supposed in Lemma 3.8, we have the limit E H U x,r ) lim r 0 r Q 1 = Υ d, for E H -a.e. x H n. Note that any homogeneous metric on stratified groups has the property diamb r ) = 2r, r > 0; it is enough to consider a point x expv 1 ), observing that δ 2 x = exp2 ln x) and using the homogeneity of dilations. Thus, we can apply Theorems 2.10.172) and 2.10.181) of [11] to the measure E H restricted to H E and use equation 21); so the proof is complete. Remark 3.10. The left-hand side of formula 24) is independent of the metric, so it is clear that the constant γ Q takes into account the change due to the distance. The following theorem is the main result of this section. Theorem 3.11 Coarea formula). Let f: H n R be a locally Lipschitz map and let A H n be a measurable set. Then we have 25) D H f dl 2n+1 = γ Q S Q 1 f 1 t) A ) dt, A where S Q 1 is the spherical Hausdorff measure with respect to any R -invariant distance and γ Q = Υ d /ω Q 1 the number Υ d is as in Theorem 3.9). Proof. We compute the coarea factor for a G-linear map and apply the coarea inequality. Consider a non-singular G-linear map L: H n R. It is not difficult to see that Lξ) = n ξ iα 1i + ξ n+i α 2i, where ξ = ξ i ),...,2n+1. Now define α = α 1, α 2 ) R 2n and notice that for any t R 26) E t H = α H 2n E t, with E t = {x H n Lx) > t} and D H L = α 0. This is clear because the level sets of L are just vertical hyperplanes, hence in particular they are C and without characteristic points, so the perimeter measure can be computed explicitly see for instance [7] or [15]). By equations 24) and 26) we get α H 2n E t \ HE t ) = E t H E t \ HE t ) = 0. The inequality S Q 1 C H 2n proved in [26] gives S Q 1 E t \ H E t) = 0, then 27) E t H = γ Q S Q 1 L 1 t). We restrict L to the bounded open set U and apply formulas 18), 19) and 27) getting 28) D H L dl 2n+1 = γ Q S Q 1 L 1 t) U ) dt. U R R

On a general coarea inequality and applications 137 Now we choose D = B 1 in Definition 1.9 and β Q 1 = γ Q see Definition 1.3), then Φ Q 1 = γ Q S Q 1. The same procedure is possible to get Φ Q = L 2n+1 and Φ 1 = L 1. Thus, by formula 28) and Definition 1.11 we find C 1 L) = D H L, so the coarea inequality 1) gives 29) γ Q S Q 1 f 1 t) A ) dt D H f dl 2n+1, R for any measurable set A H n. Using formulas 18), 19) and 24) and observing that H E t E t we get the opposite inequality for bounded open sets U H n D H f dl 2n+1 = γ Q S Q 1 HE t U) dt U R 30) γ Q S Q 1 f 1 t) A) dt, where E t = {y H n fy) > t}. As a result, inequalities 29) and 30) imply the coarea formula on open bounded sets of H n. Therefore, by Borel regularity of the spherical Hausdorff measure we finish the proof. Corollary 3.12. In the above assumptions, given a summable map u: H n R it follows that ) 31) u D H f dl 2n+1 = γ Q u ds Q 1 dt. A R R A f 1 t) A Proof. The proof follows by standard approximation arguments, taking increasing sequences of characteristic functions and applying the Beppo Levi-monotone covergence theorem. Remark 3.13. The homogeneous metric used in [15] is defined as d z, s), z, s ) ) = S z, s) 1 z, s ) ), where S z, s) ) = max{ z, s 1/2 }. By the homomorphism property of horizontal isometries it follows that this distance is R -invariant. The R -invariance of this distance and Theorem 3.9 give the representation of the perimeter measure proved in [15]. In general, for an arbitrary homogeneous distance we have E H = θs Q 1 E, where θx) = H 2n U 1 Π x )/ω Q 1 and Π x is the blow-up plane at the point x. Hence, it is clear that θx) may be not constant if the distance is not R -invariant. In the Euclidean coarea formula the geodesic distance Euclidean norm) is involved, so the natural question is whether the geodesic distance in H n namely the length metric) enjoys the R -invariance, which allows us to get formula 25). We adopt the general definition of the Carnot Carathéodory distance applied to H n, [12], [24].

138 Valentino agnani Definition 3.14. We say that an absolutely continuous curve γ: [0, t] H n is horizontal if for almost every τ [0, t] and every ξ R 2n+1 we have 32) γ τ), ξ 2 n ) 2 ) 2. Xj γτ), ξ + Yj γτ), ξ j=1 For each x, y H n we denote by C x,y the set of all horizontal curves joining x to y. From the definition of horizontal ) curve ) it follows that γ τ) is a linear combination of vectors X j γτ), Yj γτ) and its norm is bounded by the norm of the vector ) X j γτ), Yj γτ) ))j=1,...n. Chow s theorem implies that the set C x,y is not empty for all x, y H n see for instance [17]), hence we can define the following distance d C x, y) = inf { t γ : [0, t] H n, γ C x,y }. The metric d C is called the Carnot Carathéodory distance. Some remarks on this definition are in order. From a standard argument using Arzelá Ascoli s theorem it follows that for each couple of points x, y H n there exists a curve which connects them and whose length is equal to d C x, y); hence d C is a length metric. Furthermore, d C is a homogeneous distance on H n. Proposition 3.15. All the horizontal isometries are indeed isometries with respect to the Carnot Carathéodory metric. In particular, the Carnot Carathéodory metric is R -invariant. Proof. We have to prove that given T R, for any x, y H n it follows that d C x, y) = d C T x, T y). We consider γ: [0, t] H n, with γ C x,y. The map l z : H n H n denotes the left translation correspondent to an element z H n. At a differentiability point τ, by the left invariance of the vector fields X j, Y j, inequality 32) becomes dlγτ) c τ), ξ 2 n dlγτ) X j 0), ξ 2 + dlγτ) Y j 0), ξ 2, ξ R 2n+1, where γs) = l γτ) exp cs) for any s [0, t], cτ) = 0. Then we have c τ), ξ 2 n Xj 0), ξ 2 + Yj 0), ξ 2 = ξ 2, ξ R 2n+1, where ξ = ξ i ),...,2n. Now we consider the composition Γ = T γ. By Definition 3.5 the map T restricted to R 2n is in particular a real isometry, hence we have n ξ 2 = T ξ ) 2 = Xj 0), T ξ) 2 + Yj 0), T ξ) 2,

On a general coarea inequality and applications 139 then c τ), ξ 2 T ξ ) 2 for any ξ R 2n+1. It follows that T c τ), ξ 2 ξ 2 = n Xj 0), ξ 2 + Yj 0), ξ 2, ξ R 2n+1, then dlγτ) T c τ), ξ 2 n dlγτ) X j 0), ξ 2 + dlγτ) Y j 0), ξ 2, ξ R 2n+1. Now, the homomorphism property of T permits writing Γs) = l Γτ) T exp cs) for any s [0, t]; thus Γ τ) = dl Γτ) T c τ). Hence, the left invariance of X j, Y j implies Γ τ), ξ n ) 2 ) 2, Xj Γτ), ξ + Yj Γτ), ξ ξ R 2n+1. The last inequality holds at any differentiability point τ ; thus we have proved that T γ C T x),t y). The converse is similar, taking T x), T y) and using T 1 R, completing the proof. In view of the last proposition and Theorem 3.11 we obtain the coarea formula 25) with respect to the length metric d C. References [1] Ambrosio, L.: Some fine properties of sets of finite perimeter in Ahlfors regular metric measure spaces. - Adv. ath. to appear). [2] Ambrosio, L., and B. Kirchheim: Rectifiable sets in metric and Banach spaces. - ath. Ann. to appear). [3] Ambrosio, L., and B. Kirchheim: Currents in metric spaces. - Acta ath. 185, 1 80, 2000. [4] Ambrosio, L., N. Fusco, and D. Pallara: Functions of Bounded Variation and Free Discontinuity Problems. - Oxford University Press, 2000. [5] Balogh, Z..: Size of characteristic sets and functions with prescribed gradients. - Preprint. [6] Bellaïche, A.: The tangent space in sub-riemannian geometry. - In: Subriemannian Geometry, Progr. ath. 144, edited by A. Bellaïche and J. Risler, Birkhäuser Verlag, Basel, 1996. [7] Capogna, L., D. Danielli, and N. Garofalo: The geometric Sobolev embedding for vector fields and the isoperimetric inequality. - Comm. Anal. Geom. 2, 1994, 203 215. [8] Cheeger, J.: Differentiability of Lipschitz functions on metric measure spaces. - Geom. Funct. Anal. 9, 1999, 428 517. [9] Davies, R.O.: Increasing sequence of sets and Hausdorff measure. - Proc. London ath. Soc. 20, 1970, 222 236. [10] Danielli, D., N. Garofalo, and D.. Nhieu: Traces inequalities for Carnot Carathéodory spaces and applications. - Ann. Scuola Norm. Sup. 27, 1998, 195 252.

140 Valentino agnani [11] Federer, H.: Geometric easure Theory. - Springer-Verlag, 1969. [12] Fefferman, C., and D.H. Phong: Subelliptic eigenvalue problems. - In: Proceedings of the Conference on Harmonic Analysis, Chicago, 1980, edited by W. Beckner et al., Wadsworth, pp. 590 606, 1981. [13] Franchi, B., R. Serapioni, and F. Serra Cassano: eyers Serrin type theorems and relaxation of variational integrals depending on vector fields. - Houston J. ath. 22, 1996, 859 889. [14] Franchi, B., R. Serapioni, and F. Serra Cassano: Sets of finite perimeter on the Heisenberg group. - C. R. Acad. Sci. Paris 329, 1999, 183 188. [15] Franchi, B., R. Serapioni, and F. Serra Cassano: Rectifiability and perimeter in the Heisenberg group. - ath. Ann. to appear). [16] Garofalo, N., and D.. Nhieu: Isoperimetric and Sobolev inequalities for Carnot Carathéodory spaces and the existence of minimal surfaces. - Comm. Pure Appl. ath. 49, 1996, 1081 1144. [17] Gromov,.: Carnot Carathéodory spaces seen from within. - In: Subriemannian Geometry, Progr. ath. 144, edited by A. Bellaïche and J. Risler, Birkhäuser Verlag, Basel, 1996. [18] Heinonen, J.: Calculus on Carnot groups. - Ber. Univ. Jyväskylä ath. Inst. 68, 1995, 1 31. [19] Kirchheim, B.: Rectifiable metric spaces: local structure and regularity of the Hausdorff measure. - Proc. Amer. ath. Soc. 121, 1994, 113 123. [20] Korányi, A., and H.. Reimann: Foundation for the theory of quasiconformal mappings on the Heisenberg group. - Adv. ath. 111, 1995, 1 87. [21] agnani, V.: Differentiability and area formula on stratified Lie groups. - Houston J. ath. to appear). [22] iranda,.: Functions of bounded variation on good metric measure spaces. - Preprint. [23] onti, R.: Some properties of Carnot Carathéodory balls in the Heisenberg group. - Atti Accad. Naz. Lincei Cl. Sci. Fis. at. Natur. em. 9) at. Appl. to appear). [24] Nagel, A., E.. Stein, and S. Wainger: Balls and metrics defined by vector fields I: Basic properties. - Acta ath. 155, 1985, 103 147. [25] onti, R., and F. Serra Cassano: Surface measures in Carnot Carathédory spaces. - Calc. Var. Partial Differential Equations to appear). [26] Pansu, P.: Geometrie du group d Heisenberg. - These pour le titre de Doctour 3éme cycle, Université Paris VII, 1982. [27] Pansu, P.: étriques de Carnot Carathéodory quasiisométries des espaces symétriques de rang un. - Ann. ath. 129, 1989, 1 60. [28] Pauls, S.D.: The large scale geometry of nilpotent Lie groups. - Comm. Anal. Geom. to appear). [29] Pauls, S.D.: A notion of rectifiability modeled on Carnot groups. - Preprint. [30] Semmes, S.: Fractal geometry, decent calculus, structure among geometries. - Preprint. [31] Vodop yanov, S.K.: P -differentiability on Carnot groups in different topologies and related topics. - Proc. Anal. Geom. 603 670, Sobolev Institute Press, Novosibirsk, 2000. Received 24 October 2000