Journal of Applied Analysis Vol. 5, No. 2 (1999), pp. 239 247 AN INTERSETION FORMULA FOR THE NORMAL ONE ASSOIATED WITH THE HYPERTANGENT ONE M. ILIGOT TRAVAIN Received May 26, 1998 and, in revised form, September 11, 1998 Abstract. The aim of this work is to present a new result about estimation of the hypertangent normal cone of an intersection without using directionally Lipschitz assumptions. 1. Introduction During the last three decades, studies in Optimization have led to the introduction and the investigations of various types of tangent cones, generalized derivatives, subdifferentials and other mathematical objects of interest. The investigations concern for a large part the relationships beetween these objects and their calculus. One can consult the books of larke [7], Aubin Frankowska [2], Rockafellar Wets [16] and references therein. In [11], Ioffe has systematically described a family of tangent cones and summarized their properties. Among all the tangent cones of the family, three only are convex: the hypertangent cone, the interiorly hypertangent cone and the larke tangent cone. 1991 Mathematics Subject lassification. Primary 49J52. Key words and phrases. Nonsmooth analysis, hypertangent cone, normal cone. ISSN 1425-6908 c Heldermann Verlag.
240 M. iligot Travain The properties of the interiorly hypertangent cone and the larke tangent cone have been intensively studied and have led to calculus with the larke subdifferential. Many properties of the hypertangent cone can be found in Rockafellar [14], Borwein and Strojwas [4] and [5]. This paper is concerned with an intersection formula for the normal cone associated to the hypertangent cone. First let us recall some facts. The larke tangent cone to a set of a Banach space X at z is defined as { } T (z) = v X x n z, tn 0, v n v, n N, x n + t n v n where x n z means that xn z and ( n N,xn ). The interiorly hypertangent cone is I (z) = { v X x n z, tn 0, v n v, It can be defined as n 0 N, n N,n n0 xn +tnvn }. I (z) = {v X ε > 0, B(z, ε) +]0, ε[b(v, ε) } where B(z, ε) is the closed ball of radius ε centered at z. These cones are convex and I (z) T (z). The author must point out that what Rockafellar calls hypertangent cone is called here the interiorly hypertangent cone to emphasize that this cone is always open. Let and D be two closed sets of a Banach space and z a point in D. Rockafellar has shown that if X is finite dimensional and if the following tranversality hypothesis is satisfied: int (T (z)) T D (z), (1) where int (T D (z)) denotes the interior of T D (z), then we have the intersection formula N D (z) N (z) + N D (z) (2) where N (z) is the larke normal cone to at z, that is the negative polar cone of T (z), N (z) = {x X v T (z), x, v 0}. Aubin and Ekeland [1], Ioffe [9] (see also Borwein and Ward [6]) have shown that under the following hypothesis (strictly weaker than (1)): we obtain the same intersection formula. T (z) T D (z) = X, (3)
An intersection formula for the normal cone 241 In an infinite dimensional setting, the same result is obtained by Rockafellar in [14] under the hypothesis (1) and an additional hypothesis on one of the sets. He supposes that (for example) is epi-lipschitz at z which means that I (z). Ioffe has shown that if (3) holds and one of the sets is epi-lipschitz then (2) is valid, see [10]. In fact in an infinite dimensional setting, this result is equivalent to the result of Rockafellar because the following equivalence is valid when is epi-lipschitz: int (T (z)) T D (z) I (z) T D (z) I (z) T D (z) = X T (z) T D (z) = X. Borwein and Strowjas ([4]) have proved in a remarkable result that (2) still holds whenever H (z) H D (z) = X (4) where H (z) denotes the hypertangent cone to at z defined by ([15]) H (z) = {v X ε > 0, B(z, ε) +]0, ε[v }. There exists many extensions of these results, using other normal cones and other tranversality conditions. One can refer to larke and Rassi [8], Ioffe [10], Jourani and Thibault [12], Ward [17] and references therein. Our aim in this paper is to show (additionally to Borwein and Strojwas [4]) that under (4) another result can be proved, say N H D(z) N H (z) + N H D (z) where N H (z) denotes the negative polar cone of H (z). (We must point out that this formula is not a (direct) consequence of the one by Borwein and Strowjas.) 2. The hypertangent cone It is easily seen that the hypertangent cone can also be defined by H (z) = { v X x n z, tn 0, n 0 N, n N,n n0 xn +tnv }. It is convex but not necessarily closed and, in general, I (z) H (z) T (z). Note that the interiorly hypertangent cone is always open and the larke tangent cone is always closed. Let us recall the following result:
242 M. iligot Travain Theorem 1 ([15]). Let be a subset of a Banach space X and z. Suppose that is epi-lipschitz at z (i.e. I (z) ), then int(t (z)) = int(h (z)) = I (z). Note also that even if is epi-lipschitz, I (z) may be different from H (z) but the following result due to Borwein and Strojwas holds, where core(a) = {x X v X, ε > 0, x+] ε, ε[v A}. Theorem 2 ([5, orollary 2.2 p. 106]). Let be a closed subset of a Banach space X. Then core (H (z)) = int (H (z)) = I (z). In particular, is epi-lipschitz at z if and only if core (H (z)). We give a short proof of this result but before we need some notations and facts. We denote by H ε (z), where ε > 0, the following set, It is not necessarily convex. Moreover H ε (z) = {v E B(z, ε) +]0, ε[v }. It is closed whenever the set is closed. H (z) = ε>0 H ε (z) and H ε 1 (z) Hε 2 (z) if ε 1 ε 2 > 0. Let them recall that a subset A of a Banach space X is nowhere dense if int(cl(a)) =. The subset A is said to be meager if it is the union of a denumerable family of nowhere dense subsets. So if B A and A is meager then B is meager and the union of a denumerable family of meager subsets is meager. Let λ 0 and a X, then a subset A is meager if and only if a + λa is meager. A fundamental result (due to Baire) is the following: every nonempty open subset of a Banach space is nonmeager. We will need the following result: let A be a subset of a Banach space X, if core(a) the A is nonmeager. In fact, let z core(a) then n N n(a z) = X (N = N\{0}) so there exists n 0 such that n 0 (A z) is nonmeager so A is nonmeager. Note that we always have I (z) int(h (z)) core(h (z)). If core (H (z)) then H (z) is nonmeager so using the fact that H (z) = ε>0,ε Q Hε (z),
An intersection formula for the normal cone 243 where Q denotes the set of rational numbers, it follows that there exists ε 0 > 0 such that H ε 0 (z) is nonmeager. So its interior is nonempty because it is closed and hence there exists w X and η > 0 such that B(z, ε 0 )+]0, ε 0 [B(w, η) which means that w I (z) and is epi-lipschitz at z. So, due to Theorem 1, int (H (z)) = I (z) and core (H (z)) = int (H (z)) which concludes the proof. 3. Intersection formula The negative polar cone N H (z) of the hypertangent normal cone will be called the normal cone associated to the hypertangent cone, so N H (z) = {x X v H (z), x, v 0}. Remark 1. It is important to note that, in general, for a subset of a Banach space X, cl(h (z)) may be different from T (z) (z ) (if A is a subset of a Banach space, we denote cl(a) the closure of A), so N H (z) and N (z) are not necessarily equal. For example if = {(x, y, z) R3 x = z2 } then H (0) = {(0, y, 0) y R} and T (0) = {(0, y, z) (y, z) R2 }, so, clearly, N H (z) N (z). Let us introduce some complementary notations. If A is a subset of a Banach space, we denote by co[a] the convex hull of A, co[a] the closed convex hull of A (where the space is endowed with the strong topology). It will be convenient to set B r = B(0, r) for r > 0. We begin by proving the following lemma which will be needed in the sequel. Lemma 1. For all r > 0, ε > 0, co [H(z) ε B r ] H ε/(1+r) (z). Given v co [H ε (z) B r], there exists {(v i, λ i )} i=1,...,n such that n n λ i 0, λ i = 1, v i H(z) ε B r and v = λ i v i. i=1 Fix x z + B ε/(1+r) and t ] 0, ε/(1 + r) [. Using the fact that v 1 H ε (z), we get x + tλ 1 v 1. Suppose that x + t(λ 1 v 1 +... + λ i v i ), i 1. We have x + t(λ 1 v 1 +... + λ i v i ) z x z + t(λ 1 v 1 +... + λ i v i ) < ε 1 + r + ε 1 + r r = ε i=1
244 M. iligot Travain and v i+1 H ε (z), t ]0, ε[, so x + t(λ 1v 1 +... + λ i v i + λ i+1 v i+1 ). Finally x + t(λ 1 v 1 +... + λ n v n ) and v H ε/(1+r) (z). We are going to establish the main result. Theorem 3. Let and D be two closed sets of a Banach space X and z D. If then H (z) H D (z) = X N H D(z) N H (z) + N H D (z). (5) Remark 2. The inclusion (5) cannot be (directly) derived from the inclusion (2) (see the remark at the beginning of the Section 3). One always has N (z) + N D (z) N H (z) + N H D (z) but N D(z) N H D (z). In order to prove the preceding theorem, we need the following lemmas. First a slight generalization of a classical result. When A is a subset of a product X Y..., we denote by A X the projection of A on X. Lemma 2. Let X and Y be two Banach spaces and let be a closed convex set of the product X Y. Suppose that X is nonmeager. Then the interior of X is nonempty. In order to prove this lemma, we use another result. Lemma 3 ([13, Lemma 1]). Let X and Y be two Banach spaces and let be a closed convex set of a product X Y. Suppose that Y is bounded. Then int [cl [ X ]] = int [ X ]. One can write X = n N n X, where n = (X nb) and B is the closed unit ball of Y. If X is nonmeager then there exists n 0 such that n 0 X is nonmeager and then, using the Lemma 3, we can deduce that the interior of n 0 X is nonempty which ensures that the interior of X is also nonempty because n 0 X X. The following technical lemma will be also needed. We give the proof for completeness. Lemma 4. Let A and B be two closed convex sets of a Banach space X. Then A + B is the projection of a closed convex set of the product X X.
An intersection formula for the normal cone 245 Denote à = {(x, y) X X x y A}. à is a closed set of X X because A is closed. Let us verify that ] A + B = [à (X B) If z A + B then there exists x A and y B such that z = x + y so z y A, i.e. (z, y) [à à and (z, y) X B and hence (z, y) (X B) ]. ] If z [à (X B) then there exists y B such that (z, y) à which X means that z y A and z = z y + y A + B. Now à and X B are closed and convex which concludes the proof. Before stating the last lemma, recall that for a function f : X R {± } defined on a Banach space X, it is usual to set dom f = {x X f(x) < + } and [f r] = {x X f(x) r}. The Fenchel subdifferential of f at z such that f(z) R is F f(z) = {x X x X, x, x z + f(z) f(x)}. If A is a set, δ A is the indicator function defined by δ A (x) = 0 if x A, δ A (x) = + if x / A. The following lemma is elementary and well known. Lemma 5. Let f and g be functions from X to R {+ } such that f(0), g(0) Rand f(x) + g(x) f(0) + g(0) for all x X and denote for u X, Then X. ϕ(u) = inf f(x) + g(x u). x X F ϕ(0) = F f(0) F g(0). First note the simple but important following fact H (z) H D (z) H D (z). Now let x be any element of N H D (z). Then for all v H (z) H D (z), x, v 0 so for all v X ( δh (z) x ) (v) + δ HD (z)(v) 0. Denote ( ϕ(u) = inf δh (z) x ) (v) + δ HD (z)(v u). v X Then, using Lemma 5, we obtain ϕ(0) = 0 and F ϕ(0) = ( N H(z) x ) ND H (z). If r > 0, [ δ H (z) x r ] [ δ 2 HD (z) r ] [ϕ r]. 2
246 M. iligot Travain Then for η > 0 small enough, [H (z) B η ] [H D (z) B η ] [H (z) B η ] H D (z) [ϕ r]. Now, using the assumption H (z) H D (z) = X, we deduce n N [H (z) B n ] [H D (z) B n ] = X and hence there exists n 0 N such that [H (z) B n0 ] [H D (z) B n0 ] is nonmeager. So for any λ > 0, the set λ ([H (z) B n0 ] [H D (z) B n0 ]) = [H (z) B λn0 ] [H D (z) B λn0 ] is nonmeager. Fixing λ > 0 so that λn 0 η, we obtain that [H (z) B η ] [H D (z) B η ] is nonmeager. As ε>0,ε Q [Hε (z) B η ] [H ε D(z) B η ] = [H (z) B η ] [H D (z) B η ], there exists ε > 0 such that [H ε (z) B η] [HD ε (z) B η] is nonmeager and hence so is the set co [H ε (z) B η] co [HD ε (z) B η]. Furthemore by Lemma 1 and (6) we have co [H ε (z) B η ] co [H ε D(z) B η ] [ H ε/(1+η) (z) B η ] [ H ε/(1+η) D (z) B η ] [H (z) B η ] [H D (z) B η ] [ϕ r] and hence co [H ε (z) B η] co [H ε D (z) B η] is nonmeager and contained in [ϕ r]. Using Lemmas 2 and 4, we deduce that the interior of co[h ε (z) B η ] co [H ε D (z) B η] is nonempty and hence the interior of [ϕ r] is nonempty too. Finally, using the fact that dom ϕ = H (z) H D (z) = X, we obtain that 0 int ([ϕ r]) so, by convexity of ϕ, F ϕ(0) and hence x N H (z) + N H D (z). Acknowledgements. The author is grateful to Professor L. Thibault for his hints. (6) References [1] Aubin J.P., Ekeland I., Applied Nonlinear Analysis, Wiley Interscience Publ., New York, 1984. [2] Aubin J.P., Frankowska H., Set Valued Analysis, Systems ontrol Found. Appl., Birkhäuser Boston, Boston, 1990. [3] Borwein J.M., Strowjas H.M., Tangential approximations, Nonlinear Anal. T.M.A. 9 (1985), 1347 1366. [4] Borwein J.M., Strowjas H.M., The hypertangent cone, Nonlinear Anal. T.M.A. 13(2) (1989), 125 144.
An intersection formula for the normal cone 247 [5] Borwein J.M., Strowjas H.M., Directionally Lipschitzian mappings on Baire spaces, anad. J. Math. 1 (1984), 95 130. [6] Borwein J.M., Ward D.E., Nonsmooth calculus in finite dimensions, SIAM J. ontrol Optim. 25 (1987), 1312 1340. [7] larke F.H., Optimization and Nonsmooth Analysis, lassics Appl. Math., SIAM, Philadelphia, 1990. [8] larke F.H. and Rassi N., Formules d intersection en analyse non lisse, Ann. Sci. Math. Québec 14 (1990), 289 316. [9] Ioffe A.D., Approximate subdifferential and applications I: The finite dimensional theory, Trans. Amer. Math. Soc. 281 (1984), 289 316. [10] Ioffe A.D., Approximate subdifferential and applications III: The metric theory, Mathematika 36 (1989), 1 38. [11] Ioffe A.D., On the subdifferential theory, in Fermat days 85: Mathematics for Optimization, Math. Studies Series, North Holland, Amsterdam, 1986. [12] Jourani A., Thibault L., Extensions of the subdifferential calculus rules in Banach spaces and applications, anad. J. Math. 48 (1996), 834 848. [13] Robinson S.M., Regularity and stability for convex multivalued functions, Math. Oper. Res. 1 (1976), 130 143. [14] Rockafellar R.T., Directionally Lipschitzian functions and subdifferential calculus, Proc. London Math. Soc. 39 (1979), 331 355. [15] Rockafellar R.T., Generalized directional derivatives and subgradients of nonconvex functions, anad. J. Math. 32 (1980), 257 280. [16] Rockafellar R.T. and Wets R.J-B., Variational Analysis, Grundlehren Math. Wiss. 317, Springer, New York, 1998. [17] Ward D.E., onvex subcones of the contingent cone in nonsmooth calculus and optimization, Trans. Amer. Math. Soc. 302 (1987), 1312 1340. Marc iligot Travain Departement of Mathematics University of Avignon 33 rue Pasteur 84000 Avignon France