c 2012 Society for Industrial and Applied Mathematics

Transcription

1 SIAM J. CONTROL OPTIM. Vol. 50, No. 4, pp c 01 Society for Industrial and Applied Mathematics VARIATIONAL INEQUALITIES FOR SET-VALUED VECTOR FIELDS ON RIEMANNIAN MANIFOLDS: CONVEXITY OF THE SOLUTION SET AND THE PROXIMAL POINT ALGORITHM CHONG LI AND JEN-CHIH YAO Abstract. We consider variational inequality problems for set-valued vector fields on general Riemannian manifolds. The existence results of the solution, convexity of the solution set, and the convergence property of the proximal point algorithm for the variational inequality problems for set-valued mappings on Riemannian manifolds are established. Applications to convex optimization problems on Riemannian manifolds are provided. Key words. variational inequalities, Riemannian manifold, monotone vector fields, proximal point algorithm, convexity of solution set AMS subject classifications. Primary, 49J40; Secondary, 58D17 DOI / Introduction. Various problems posed on manifolds arise in many natural contexts. Classical examples are given by some numerical problems such as eigenvalue problems and invariant subspace computations, constrained minimization problems, and boundary value problems on manifolds, etc.; see, for example, [1, 13, 17, 6, 37, 38, 39]. Recent interests are focused on extending some classical and important results for solving these problems on linear spaces to the setting of manifolds. For example, some numerical methods such as Newton s method, the conjugate gradient method, the trust-region method, and their modifications for optimization problems on linear spaces are extended to the Riemanninan manifolds setting [, 11, 16, 4]. A theory of subdifferential calculus for functions defined on Riemannian manifolds is developed in [3, 0], where these results are applied to show existence and uniqueness of viscosity solutions to Hamilton Jacobi equations defined on Riemannian manifolds and to study constrained optimization problems and nonclassical problems of calculus of variations on Riemannian manifolds. The important notions of monotonicity in linear spaces were extended to Riemannian manifolds and have been studied extensively in [1,, 7, 8, 9, 30, 31, 40], while weak sharp minima for constrained optimization problems on Riemannian manifolds are explored recently in [3], where various notions of weak sharp minima are extended and their complete characterizations are established. Variational inequalities on R n are powerful tools for studying constrained optimization problems and equilibrium problems, as well as complementary problems, and have been studied extensively; see, for example, the survey [18] and the book [3]. Variational inequality problems on Riemannian manifolds were first introduced and studied in [6] by Németh for univalued vector fields on Hadamard manifolds. Received by the editors May 3, 011; accepted for publication (in revised form) May 18, 01; published electronically August 8, Department of Mathematics, Zhejiang University, Hangzhou 31007, People s Republic of China (cli@zju.edu.cn). This author s research was partially supported by the National Natural Science Foundation of China (grant ) and by Zhejiang Provincial Natural Science Foundation of China under grant Y Center for General Education, Kaohsiung Medical University, Kaohsiung 8070, Taiwan (yaojc@ kmu.edu.tw). This author s research was partially supported by the National Science Council of Taiwan under grant NSC M MY3. 486

2 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 487 Németh established in [6] some basic results on existence and uniqueness of the solution for variational inequality problems on Hadamard manifolds and proposed an open problem on how to extend the existence and uniqueness results from Hadamard manifolds to Riemannian manifolds. This problem was solved completely in [5] by Li et al. The famous proximal point algorithm for optimization problems and for variational inequality problems on Hilbert spaces were extended to the setting of Hadamard manifolds, respectively, in [14] and [1], where the well-posedness and convergence results of the proximal point algorithm on Hadamard manifolds were established. Another basic and interesting problem for variational inequalities is the convexity of the solution set, which, unlike in the linear cases, seems nontrivial even for univalued vector fields on Hadamard manifolds. Our interest in the present paper is the study of variational inequality problems for set-valued vector fields on general Riemannian manifolds (not necessarily Hadamard manifolds). The purpose of the present paper is to explore the existence of the solution and the convexity of the solution set, as well as the convergence property of the proximal point algorithm for the variational inequality problems for set-valued mappings on Riemannian manifolds. Compared with the corresponding ones for linear space, the problems considered here for the general Riemannian manifold cases are much more complicated, and most of the known techniques in linear space setting does not work; for example, even in a Hadamard manifold we do not have the following property (which holds trivially in a linear space and plays a crucial role in the study of the convexity problem of the solution set): The image of any linear segment in the tangent space under an exponential map is a geodesic segment in the underlying manifold. Most of the main results obtained in the present paper extend and improve the corresponding ones for univalued vector fields on Hadamard manifolds and/or Riemannian manifolds, while the convexity results of solution sets are completely new for set-valued vector fields on Hadamard and Riemannian manifolds. The paper is organized as follows. The next section contains some necessary notation, notion, and preliminary results. The existence and uniqueness of the solution set of the variational inequality problems for set-valued mappings on general Riemannian manifolds are established in section 3. In sections 4 and 5, the convexity results of the solution set and the convergence results of the proximal point algorithm of the variational inequality problems are provided for set-valued mappings on Riemannian manifolds of curvature bounded above. Applications of our results on convergence of the proximal point algorithm to convex optimization problems on Riemannian manifolds are presented in section 6.. Notation and preliminary results. Webeginwithsomenecessarynotation, notions, and preliminary results about Riemannian manifolds that will be used in the next sections. The readers are referred to some textbooks for details, for example, [5, 1, 33, 36]. Let M be a finite-dimensional Riemannian manifold with the Levi Civita connection on M. Letx M, and let T x M denote the tangent space at x to M. Wedenote by, x the scalar product on T x M with the associated norm x, where the subscript x is sometimes omitted. For x, y M, letc :[0, 1] M be a piecewise smooth curve joining x to y. Then the arc-length of c is defined by l(c) := 1 ċ(t) dt, 0 while the Riemannian distance from x to y is defined by d(x, y) :=inf c l(c), where the infimum is taken over all piecewise smooth curves c :[0, 1] M joining x to y. Recall that a curve c :[0, 1] M joining x to y is a geodesic if (.1) c(0) = x, c(1) = y, and ċċ =0on[0, 1],

3 488 CHONG LI AND JEN-CHIH YAO and a geodesic c :[0, 1] M joining x to y is minimal if its arc-length equals its Riemannian distance between x and y. By the Hopf Rinow theorem (cf. [1]), if M is additionally complete and connected, then (M,d) is a complete metric space, and there is at least one minimal geodesic joining x to y. Moreover, the exponential map at x exp x : T x M M is well-defined on T x M. Clearly, a curve c :[0, 1] M is a minimal geodesic joining x to y if and only if there exists a vector v T x M such that v =d(x, y) andc(t) =exp x (tv) foreacht [0, 1]. Let γ be a geodesic. We use P γ,, to denote the parallel transport on the tangent bundle TM (defined below) along γ with respect to, which is defined by P γ,γ(b),γ(a) (v) =V (γ(b)) for any a, b R and v T γ(a) M, where V is the unique vector field satisfying γ (t)v =0forallt and V (γ(a)) = v. Then, for any a, b R, P γ,γ(b),γ(a) is an isometry from T γ(a) M to T γ(b) M. We will write P y,x instead of P γ,y,x in the case when γ is a minimal geodesic joining x to y and no confusion arises. For a Banach space or a Riemannian manifold Z, weuseb Z (p, r) andb Z (p, r) to denote, respectively, the open metric ball and the closed metric ball at p with radius r, thatis, B Z (p, r) ={q Z : d(p, q) <r} and B Z (p, r) ={q Z : d(p, q) r}. We often omit the subscript Z if no confusion arises. Recall that, for a point x M, the convexity radius at x is defined by { } each ball in B(x, r) is strongly convex (.) r x := sup r>0:. and each geodesic in B(x, r) is minimal Clearly, if M is a Hadamard manifold, then r x =+ for each x M. Let TM := ({x} T x M) x M denote the tangent bundle. Then TM is a Riemannian manifold with the natural differential structure and the Riemannian metric; see, for example, [1]. Thus, the following proposition can be proved by a direct application of the definition of parallel transports (cf. [1, Lemma.4] in the case when M is a Hadamard manifold). Proposition.1. Let z 0 M, and define the mapping P z0 : TM T z0 M by P z0 (z,u):=p z0,zu for each (z,u) TM. Then P z0 is continuous on x B(z 0,r z0 ) ({x} T xm). We denote by Γ x,y the set of all geodesics c := γ xy :[0, 1] M satisfying (.1). Note that each c Γ x,y can be extended naturally to a geodesic defined on R in the case when M is complete. Definition. below presents the notions of the different kinds of convexities, where items (a) and (b) are known in [41], while item (c) is known in [7]; see also [3, 5]. Definition.. Let A be a nonempty subset of M. ThesetA is said to be (a) weakly convex if, for any p, q A, there is a minimal geodesic of M joining p to q lying in A; (b) strongly convex if, for any p, q A, there is just one minimal geodesic of M joining p to q and it is in A;

4 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 489 (c) locally convex if, for any p A, there is a positive ε>0 such that A B(p, ε) is strongly convex. Clearly, the following implications hold for a nonempty set A in M: (.3) The strong convexity = the weak convexity = the local convexity. Remark.1. Recall (cf. [36]) that M is a Hadamard manifold if it is a simple connected and complete Riemannian manifold of nonpositive sectional curvature. In a Hadamard manifold, the geodesic between any two points is unique and the exponential map at each point of M is a global diffeomorphism. Therefore, all convexities in a Hadamard manifold coincide and are simply called the convexity. Recall from [39, p. 110] (see also [36, Remark V.4.]) that a point o M is called a pole of M if exp o : T o M M is a global diffeomorphism, which implies that for each point x M, the geodesic of M joining o to x is unique. For the existence result of the solutions of the problem (3.1), the notion of the weak pole of A in the following definition was introduced in [5]. Definition.3. Apointo A is called a weak pole of A if for each x A, the minimal geodesic of M joining o to x is unique and lies in A. Clearly, any subset with a weak pole is connected. Let p A. For the following proposition, which is known in [36, pp ], we use ɛ(p) to denote the supremum of the radius r such that the set A B(p, r) is strongly convex, that is, ɛ(p) :=sup{r >0: A B(p, r) is strongly convex}. Proposition.4. Suppose that M is a complete Riemannian manifold. Let A M be a nonempty closed connected and locally convex subset. Then, there exists a connected (embedded) k-dimensional totally geodesic submanifold N of M possessing the following properties: (i) A = N and γ xy ([0, 1)) N for any p A, x B(p, ɛ(p)) N, y B(p, ɛ(p)) A, andγ xy Γ x,y ; (ii) γ xy (t) / A for any y / N and any t (1,t 0 ],wheret 0 > 1 is such that γ xy ([0,t 0 ]) B(p, ɛ(p)). Following [36, p. 171], the sets int R A := N and R A := A \ N are called the (relative) interior and the (relative) boundary of A, respectively. Let int A denote the topological interior of A, thatis,x int A if and only if B(x, δ) A for some δ>0. Some useful properties about the interiors are given in the following proposition. Let x A. WeuseF x A to denote the set of all feasible directions by (.4) F x A := {v T x M : there exists t >0 such that exp x tv A for any t (0, t)}. Remark.. Assume that M is complete and A is locally convex. Following [36, p. 171], we define the set C ( x) by C ( x) := {v T x M \{0} : there exists t (0, r x ) such that exp x t(v/ v ) N} {0}. Then one checks by definition that (.5) C ( x) F x A C ( x) for any x A and (.6) C ( x) = F x A for any x N.

5 490 CHONG LI AND JEN-CHIH YAO Proposition.5. Suppose that M is a complete Riemannian manifold and A is a closed connected and locally convex subset of M. Let N =int R A. Then the following assertions hold: (i) x int R A F x A = T x N B( x, δ) exp x (T x N) A for some δ>0. (ii) x int A F x A = T x M. (iii) If int A, thenint R A =inta. (iv) If x int R A is a weak pole of A, thenγ x x ((0, 1]) int R A for any x A. Proof. In view of Remark., one easily sees that assertion (i) holds by [36, pp ], while assertion (ii) is clear by the definition of the interior. As for assertion (iii), we note that if x int A, thent x N = T x M by (ii), and so N is an m-dimensional totally geodesic submanifold N of M, wherem := dimm. This means that, for any x N, T x N is of dimension m, andsot x N = T x M. Thus (iii) is seen to hold by assertion (ii). Finally, assertion (iv) is known in [5, Proposition 4.3]. 3. Existence and uniqueness results. Throughout the whole paper, we always assume that M is a complete Riemannian manifold. Let A M be a nonempty set, and let Γ A x,y denote the set of all γ xy Γ x,y such that γ xy A. LetV : A TM be a set-valued vector field on A, thatis, V (x) T x M for each x A. Thevariational inequality problem considered here on Riemannian manifold M is of finding x A such that (3.1) v V ( x) satisfying v, γ xy (0) 0 for each y A and each γ xy Γ Ā x,y. Any point x A satisfying (3.1) is called a solution of the variational inequality problem (3.1), and the set of all solutions is denoted by S(V,A). The following theorem on the existence of solutions of the variational problem (3.1) for continuous (univalued) vector fields on A was proved in [5], which plays an important role for the study of this section. Theorem 3.1. Let A M be a compact, locally convex subset of M, andlet V be a continuous vector field on A. Suppose that A has a weak pole o int R A. Then S(V,A), that is, the variational inequality problem (3.1) admits at least one solution x. Write A R := A B(o, R), where R > 0ando A. Consider the following variational inequality problem: Find x R A R and v R V (x R ) such that (3.) v R, γ xry(0) 0 for each y A R and each γ xry Γ AR x R,y. The following proposition establishes the relationship between the solutions for problems (3.) and (3.1), which will be used frequently for our study in what follows. Proposition 3.. Let A M be a nonempty subset, and let V be a set-valued vector field on A. Then x A is a solution of the problem (3.1) if and only if there exist R>0 and a point o A such that x B(o, R) and x is a solution of the problem (3.), that is, (3.3) x S(V,A) [ R >0, o A such that x S(V,A R ) B(o, R)]. Proof. The necessity part is trivial. Hence we need only prove the sufficiency part. To this end, suppose there exist o A and R>0 such that x S(V,A R ), and so x B(o, R). Let y A and γ xy Γ Ā x,y. Then there exists δ>0 such that δ γ xy (0) < min{r x,r d(o, x)},

6 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 491 where r x is the convexity radius at x. Set z =exp x (δ γ xy (0)). Then z A R,and γ x,z Γ x,z is unique and contained in A R ; hence it satisfies that γ xz (0) = δ γ xy (0). Since x S(V,A R ), it follows that there exists v V ( x) such that v,δ γ xy (0) 0. Consequently, v, γ xy (0) 0. This shows that x S(V,A) because y A and γ xy Γ Ā x,y are arbitrary. Below we will extend the existence theorem to the case of set-valued vector fields on A. Let (X, d X )and(y,d Y ) be metric spaces. Let X Y be the product metric space endowed with the product metric d defined by d((x 1,y 1 ), (x,y )) := d X (x 1,x )+d Y (y 1,y ) for any (x 1,y 1 ), (x,y ) X Y. Let Γ be a set-valued mapping from X to Y. Let gph(γ) denote the graph of Γ defined by gph(γ) := {(x, y) X Y : y Γ(x)}. Recall that Γ is upper semicontinuous on X if for any x 0 X and any open set U containing Γ(x 0 ), there exists a neighborhood B(x 0,δ)ofx 0 such that Γ(x) U for any x B(x 0,δ). In the following definition, we extend this notion to set-valued vector fields on manifolds; see, for example, [1]. Definition 3.3. Let A M be a nonempty subset and V be a set-valued vector field on A. Letx 0 A. V is said to be (a) upper semicontinuous at x 0 if for any open set W satisfying V (x 0 ) W T x0 M, there exists an open neighborhood U(x 0 ) of x 0 such that P x0,xv (x) W for any x U(x 0 ) A. (b) upper Kuratowski semicontinuous at x 0 if for any sequences {x k } A and {u k } TM with each u k V (x k ),relationslim k x k = x 0 and lim k u k = u 0 imply u 0 V (x 0 ). (c) upper semicontinuous (resp., upper Kuratowski semicontinuous) on A if it is upper semicontinuous (resp., upper Kuratowski semicontinuous) at each x 0 A. Remark 3.1. By definition, the following assertions hold: (i) The upper semicontinuity implies the upper Kuratowski semicontinuity. The converse is also true if A is compact and V is compact-valued. (ii) A set-valued vector field V on A is upper semicontinuous at x 0 A if and only if the set-valued mapping V : A TM is upper semicontinuous at x 0 ;thatis, for any open subset W of TM containing V (x 0 ), there exists a neighborhood U(x 0 ) of x 0 such that V (x) W for all x U(x 0 ) A. The following proposition is known in [4, Theorem 1]. Proposition 3.4. Let X be a metric space without isolated points, and let Y be anormedlinearspace. LetΓ:X Y be an upper semicontinuous mapping such that Γ(x) is compact and convex for each x X. Then for each ɛ>0 there exists a continuous function f ɛ : X Y such that gph(f ɛ ) U(gph(Γ),ɛ), whereu(gph(γ),ɛ) is an ɛ-neighborhood of gph(γ): U(gph(Γ),ɛ):={(x, y) X Y :d((x, y), gph(γ)) <ɛ}. Consider a compact subset A of M, and let {B(x i,r i ): i =1,,...,m} be an open cover of A, thatis,a m i=1 B(x i,r i ), where, for each i =1,,...,m, r i := rx i and r xi is the convexity radius at x i. Fixingsuchanopencoverandx A, wedenote by I(x) the index-set such that i I(x) if and only if x B(x i,r i ).

7 49 CHONG LI AND JEN-CHIH YAO Let V(A) denote the set of all upper Kuratowski semicontinuous set-valued vector fields V satisfying that V (x) is compact and convex for each x A. By Proposition 3.4, we can verify the following existence result of ɛ-approximations for set-valued vector fields on A. Lemma 3.5. Suppose that A is a connected compact subset of M and that V V(A). Letε>0. Then, there exists a continuous vector field V ε such that for any x M, there exist a subset {x ε i : i I(x)} A and a point yε conv{p x,xi P xi,x ε i V (xε i ): i I(x)} satisfying that (3.4) V ε (x) y ε <ε and d(x ε i,x) <ε for each i I(x). Proof. By the well-known finite partition theorem of unity (see [1], for example) and without loss of generality, we may assume that there exist m nonnegative continuous functions {ψ i : A R : i =1,...,m} such that (3.5) supp(ψ i ):={x A : ψ i (x) 0} B(x i,r i ) and (3.6) m ψ i (x) =1 foreachx A. i=1 Let i =1,,...,m and consider the mapping F i : A B(x i,r i ) Tx i M defined by F i (x) :=P xi,xv (x) for each x A B(x i,r i ). Then F i is upper semicontinuous by Proposition.1 and F i (x) is compact convex for each x A B(x i,r i ). Then by Proposition 3.4, there exists a continuous mapping f ε i : A B(x i,r i ) T xi M such that (3.7) gph( f ε i ) U(gph(F i),ɛ). Let fi ε : A TM be defined by fi ε(x) := f i ε(x) foreachx A B(x i,r i )and fi ε(x) := 0 otherwise. Define the vector field V ε : A M by V ε (x) = m i=1 ψ i (x)p x,xi fi ε (x) for each x A. It is easy to check that V ε is continuous on A. Now we prove that V ɛ is as desired. To do this, let x A and let I + (x) be the subset of {1,,...,m} such that ψ i (x) > 0 for i I + (x). Then I + (x) I(x) by (3.5). Furthermore, we have by (3.7) that, for each i I + (x), there exist x ɛ i A B(x i,r i )andyi ɛ F i(x ɛ i ) such that (3.8) d(x, x ɛ i)+ f ε i (x) y ɛ i <ɛ. Write y ɛ := i J(x) ψ i(x)p x,xi y ɛ i.thenyε conv{p x,xi P xi,x ε i V (xε i ): i I+ (x)} by the definition of F i. Moreover, by (3.8), together with the definitions of V ɛ (x) andy ɛ, one sees that x ɛ i B(x, ɛ) foreachi I+ (x) and V ɛ (x) y ɛ ψ i (x) fi ε (x) yi ɛ <ɛ; i J(x)

8 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 493 hence (3.4) is shown and the proof is complete. Theorem 3.6. Let A M be a compact and locally convex subset of M. Suppose that A has a weak pole o int R A and that V V(A). Then the variational inequality problem (3.1) admits at least one solution. Proof. Fix an open cover {B(x i,r i ): i =1,,...,m} of A and let n N. Then, applying Lemma 3.5 to 1 n in place of ɛ, we have that there exists a continuous vector field V n : A TM with the properties mentioned in Lemma 3.5 for V n in place of V ɛ. Applying Theorem 3.1 to V n,wehavethats(v n,a). Take x n S(V n,a). Then, we can choose (3.9) ( {x n i : i I( x n)} B x n, 1 ) and v n conv{p xn,x n i P xi,x nv (xn i i ): i I( x n)} satisfying (3.10) V n ( x n ) v n < 1 n. Without loss of generality, we may assume that I( x n )={1,,..., m} for some natural number m m and v n := m i=1 tn i P x n,x i P xi,x nvn i i with each vi n V (x n i ), each tn i 0, and m i=1 tn i =1. Recalling that A is compact, V is upper semicontinuous, and each V (x) iscompact, we see that, for each i =1,..., m, the sequence {vi n : n N} is bounded. Thus, without loss of generality, we may assume that (3.11) x n x, t n i t i and v n i v i for each i =1,..., m (using subsequences if necessary). This, together with (3.9), implies that (3.1) d(x n i, x) d(x n i, x n )+d( x n, x) 0 for each i =1,..., m. It follows from Proposition.1 and (3.11) that P xn,x i P xi,x nvn i i v i for each i. Furthermore, we have that v i V ( x) foreachi =1,..., m since V is upper Kuratowski semicontinuous, and so (3.13) v := lim n v n V ( x) as V ( x) is closed and convex. Consider the variational inequality problem (3.) with R = r x /ando = x. Since x n S(V n,a) S(V n,a R )and x n x, it follows that, for n large enough, V n ( x n ), exp 1 x n y 0 for each y A R. Since V n ( x n ) v by (3.10) and (3.13), and since exp 1 x n y exp 1 x y (noting that x n x), we conclude by taking limits that v, exp 1 x y 0 for each y A R. This shows that x S(V,A R )asv V ( x), and so x S(V,A) by Proposition 3.. The proof is complete. Note that if A is a convex subset of a Hadamard manifold M, thenint R A is nonempty and each point of A is a pole. Hence the following corollary, which extends

9 494 CHONG LI AND JEN-CHIH YAO the corresponding existence result in [6] to the setting of set-valued vector fields, now is a direct consequence of Theorem 3.6. Corollary 3.7. Suppose that M is a Hadamard manifold, and let A M be a compact convex set. Let V V(A). Then the variational inequality problem (3.1) admits at least one solution. To extend the existence result on solutions of the variational inequality problem (3.1) to the case when A is not necessarily bounded, we introduce the coerciveness condition for set-valued vector fields on Riemannian manifolds. Recall that γ ox denotes the unique minimal geodesic joining o to x when o is a weak pole of A and x A. Definition 3.8. Let A M be a subset of M containing a weak pole o of A. Let V be a vector field on A. V is said to satisfy the coerciveness condition if (3.14) sup v 0 V (o),v x V (x) v x, γ xo (0) v 0, γ xo (1) d(o, x) as d(o, x) + for x A. Definition 3.9. We say that a locally closed convex set A has the BCC (bounded convex cover) property if there exists o A such that, for any R>0, thereexistsa locally convex compact subset of M containing A B(o, R). Clearly, a locally closed convex set A has the BCC property if and only if, for any bounded subset A 0 A, there exists a locally convex compact subset of M containing A 0. Moreover, by [5, Proposition 4.5], if M is complete and if the sectional curvature of M is nonnegative everywhere or nonpositive everywhere, then any locally convex closed subset of M has the BCC property. Theorem Let A M be a locally convex closed set with a weak pole o int R A,andletV V(A) satisfy the coerciveness condition. Suppose that A has the BCC property. Then the variational inequality problem (3.1) admits at least a solution. Proof. Take H > V (o) := inf v V (o) v. Then, by the assumed coerciveness condition, there is R>0 such that (3.15) sup ( v x, γ xo (0) v 0, γ xo (1) ) Hd(o, x) foreachx A with d(o, x) R. v 0 V (o),v x V (x) Hence, for each x A with d(o, x) R, one has that (3.16) sup v x V (x) v x, γ xo (0) Hd(o, x)+ V (o) γ xo (1) =( V (o) H)d(o, x) < 0. By assumption, there exists a locally convex compact subset K R of M containing A B(o, R). Then ÂR := A K R M is a locally convex compact set. Moreover, note that, for any x ÂR, the unique minimal geodesic γ ox joining o to x lies in B(o, R) and thus in A B(o, R) because o is a weak pole of A. Thisimpliesthat γ ox is in K R andtheninâr. Hence o int R  R is a weak pole of  R. Thus Theorem 3.6 is applied (with ÂR in place of A) togetthats(v,âr). Since A R ÂR, it follows that S(V,ÂR) S(V,A R ). Furthermore, by (3.16), we have that S(V,A R ) B(o, R). It follows from Proposition 3. that S(V,ÂR) S(V,A R ) S(V,A). This together with the fact that S(V,ÂR) completes the proof. Thus the following corollary is direct from Theorem Corollary Let A M be a locally convex closed set with a weak pole o int R A,andletV V(A) satisfy the coerciveness condition. Suppose that M is

10 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 495 complete and that the sectional curvature of M is nonnegative everywhere or nonpositive everywhere. Then the variational inequality problem (3.1) admits at least one solution. Below we consider the uniqueness problem of the solution of the variational inequality problem (3.1). The notions of monotonicity on Riemannian manifolds in the following definition are important tools for the study of the uniqueness problems and have been extensively studied in [8, 9, 10, 15, 19, 7, 8, 9, 30] for univalued vector fields and in [1,, 40] for set-valued vector fields. Definition 3.1. Let A M be a nonempty weakly convex set and let V be a set-valued vector field on A. The vector field V is said to be (1) monotone on A if for any x, y A and γ xy Γ A x,y the following inequality holds: (3.17) v x, γ xy (0) v y, γ xy (1) 0 for any v x V (x), v y V (y); () strictly monotone on A if for any x, y A and γ xy Γ A x,y the following inequality holds: (3.18) v x, γ xy (0) v y, γ xy (1) < 0 for any v x V (x), v y V (y); (3) strongly monotone on A if there exists ρ>0 such that for any x, y A and γ xy Γ A x,y the following inequality holds: (3.19) v x, γ xy (0) v y, γ xy (1) ρl (γ xy ) for any v x V (x), v y V (y). Now we have the following uniqueness result on the solution of problem (3.1). Theorem Let A 0 A M. Suppose that A 0 is weakly convex and V is strictly monotone on A 0 satisfying S(V,A) A 0. Then the variational inequality problem (3.1) admits at most one solution. In particular, if A is weakly convex and V is strictly monotone on A, then the variational inequality problem (3.1) admits at most one solution. Proof. Suppose on the contrary that problem (3.1) admits two distinct solutions x and ȳ. Then x, ȳ A 0.SinceA 0 is weakly convex, there exists a minimal geodesic γ xȳ Γ A0 x,ȳ. Then there exist v V ( x) andū V (ȳ) such that (3.0) v, γ xȳ (0) 0 and ū, γ xȳ (1) 0. It follows that v, γ xȳ (0) ū, γ xȳ (1) 0. This contradicts the strict monotonicity of V on A 0, and the proof is complete. It is routine to verify that if V is strongly monotone, then it satisfies the coerciveness condition. Therefore, the following corollary is straightforward. Corollary Let A M be a closed weakly convex set with a weak pole o int A, andletv V(A) be a strongly monotone vector field on A. Suppose that A has the BCC property (e.g., M is complete and the sectional curvature of M is nonnegative everywhere or nonpositive everywhere). Then the variational inequality problem (3.1) admits a unique solution.

11 496 CHONG LI AND JEN-CHIH YAO 4. Convexity of solution sets. As assumed in the previous section, let A M be a nonempty closed subset, and let V be a set-valued vector field on A. This section is devoted to the study of the convexity problem of the solution set of the variational inequality problem. For this purpose, we introduce the notions of r-convexity and prove some related lemmas. Definition 4.1. Let A be a nonempty subset of M and let r (0, + ). Theset A is said to be (a) weakly r-convex if, for any p, q K with d(p, q) <r, there is a minimal geodesic of M joining p to q lying in A; (b) r-convex if, for any p, q A with d(p, q) < r, there is just one minimal geodesic of M joining p to q and it is in A. Let κ 0, and assume that M is of the sectional curvature bounded above by κ. The Riemannian manifolds of the sectional curvature bounded above by κ possess some useful properties which are listed in the following proposition. Note by [6, Theorem 1A.6] that any Riemannian manifold of the curvature bounded above by κ is a CAT(κ) space (the reader is referred to [6] for details). Thus Proposition 4. below is a direct consequence of [6, Propositions 1.4 and 1.7]. Recall that the model space Mκ n is the Euclidean space E n if κ = 0 and the Riemannian manifold obtained from the n-dimensional sphere (S n, d) by multiplying the distance function by the constant 1 κ if κ>0. Then Mκ n is of the constant curvature κ, and the following law of cosines in Mκ n holds: d( x, ȳ) =d( x, z) +d(ȳ, z) d( x, z)d(ȳ, z)cosᾱ if κ =0, (4.1) cos( κd( x, ȳ)) = cos( κd( x, z)) cos( κd(ȳ, z)) + sin( κd( x, z)) sin( κd(ȳ, z)) cos ᾱ if κ>0, where x, ȳ, z Mκ n and ᾱ := x zȳ is the angle at z of two geodesics joining z to x, ȳ; see, for example, [6, p..4]. By definition, a geodesic triangle Δ(x, y, z) inm consists of three points {x, y, z} in M (the vertices of Δ(x, y, z)) and three geodesic segments joining each pair of vertices (the edges of Δ(x, y, z)). Recall that a triangle Δ( x,ȳ, z) in the model space Mκ is said to be a comparison triangle for Δ(x, y, z) X if d(x, y) =d( x,ȳ), d(x, z) =d( x, z) and d(z, y) =d( z,ȳ). Define D κ := π κ if κ>0andd κ := + if κ =0. Proposition 4.. Suppose that M is of the curvature bounded above by κ. Then the following assertions hold: (i) For any two points x, y M with d(x, y) <D κ, there exists one and only one minimizing geodesic joining x to y. (ii) For any z M, the distance function x d(z,x) is convex on B(z, Dκ ); that is, for any two points x, y B(z, Dκ ) and any γ xy Γ x,y, the function t d(z,γ(t)) is convex on [0, 1]. (iii) Any ball in M with radius less than Dκ is strongly convex; in particular, we have that r x Dκ for each x M, wherer x is the convexity radius defined by (.). (iv) For any geodesic triangle Δ(x, y, z) in M with d(x, y) +d(y, z) +d(z,x) < D κ, there exists a comparison triangle Δ( x, ȳ, z) in Mκ for Δ(x, y, z) such that, for γ yz Γ y,z and γȳ z Γȳ, z, d(γ yz (t),x) d( γȳ z (t), x) for each t [0, 1] and xzy x zȳ,

12 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 497 where xzy and x zȳ denote the inner angles at z and z of triangles Δ(x, y, z) and Δ( x, ȳ, z), respectively. Clearly, by assertion (i) of Proposition 4., if M is of the curvature bounded above by κ, then a subset of M is weakly D κ -convex if and only if it is D κ -convex. Let z A and λ>0. Define the vector field V λ,z : A TM by V λ,z (x) =λv (x) E z (x) for each x A, where E z is the vector field defined by (4.) E z (x) :={u exp 1 x z : u =d(x, z), exp x tu A t [0, 1]} for each x A. Note that E z (x) for any z, x A with d(x, z) <rif A is weakly r-convex. Moreover, if A is r-convex, then we have that (4.3) E z (x) =exp 1 x z for any z, x A with d(x, z) <r. Consider the variational inequality problem (3.1) with V λ,z in place of V,and recall that S(V λ,z,a) is its solution set. Then we define the set-valued map J λ : M A by J λ (z) :=S(V λ,z,a) for each z A, that is, x J λ (z) if and only if there exist v V (x) andu E z (x) such that (4.4) λv u, γ zy (0) 0 for each y A and each γ zy Γ A z,y. Clearly, the following equivalence holds for any z A: (4.5) z S(V,A) z J λ (z). Throughout this whole section, we assume that V is monotone on A. For a subset Z of some normed space, it would be convenient to use the notion Z to denote the distance to the origin from Z: { infu Z u if Z, Z := 0 if Z =. Lemma 4.3. Let r (0, + ), andleta be a weakly r-convex subset of M. Let λ>0 and x A be such that λ V (x) <r. Then the following assertion holds: (4.6) B(x, r) J λ (x) B(x, λ V (x) ). In particular, if A is weakly convex, then we have that (4.7) J λ (x) B(x, λ V (x) ). Proof. Letz B(x, r) J λ (x). Since A is weakly r-convex, it follows by definition that there exist v V (z) andu E z (x) such that (4.4) holds. Let γ zx be the geodesic defined by γ zx (t) :=exp z tu for each t [0, 1]. Then by (4.), d(x, z) = u, γ zx (0) = u, andγ zx Γ A z,x. This, together with (4.4), implies that (4.8) d (x, z) = u, u = u, γ zx (0) λ v, γ zx (0).

13 498 CHONG LI AND JEN-CHIH YAO Since V is monotone, it follows that d (x, z) λ v, γ zx (0) λ v x, γ zx (1) λ v x d(x, z) for each v x V (x). Hence d(x, z) λ V (x), and the proof is complete. Lemma 4.4. Suppose that M is of the sectional curvature bounded above by κ and that A is D κ -convex. Let y 0 A and λ>0 be such that λ V (y 0 ) < Dκ. Then J λ (y 0 ) is nonempty. Proof. LetR>0 be such that λ V (y 0 ) <R<r:= D κ. Then A R = B(y 0,R) A is compact and locally convex. Take o int R A R such that d(y 0,o) < r R. Then, for each x A R, d(x, o) d(x, y 0 )+d(y 0,o) R + r R <r. By Proposition 4. (i), (ii), the minimal geodesic joining o to x is unique and lies in A R. This means that o is a weak pole of A R. Thus, Theorem 3.6 is applicable to concluding that S(V λ,y0,a R ). By Lemma 4.3 (applied to A R in place of A), we have that d(y R,y 0 ) λ V (y 0 ) <R for any y R S(V λ,y0,a R ). This, together with Proposition 3. (applied to o := y 0 ), implies that S(V λ,y0,a R ) S(V λ,y0,a)=j λ (y 0 ). This shows that J λ (y 0 ), and the proof is complete. Lemma 4.5. Suppose that M is of the sectional curvature bounded above by κ and that A is D κ -convex. Let y 0 A, x 0 S(V,A), andλ>0 be such that (4.9) λ V (y 0 ) +d(y 0,x 0 ) < D κ. Then the following estimates hold for each z J λ (y 0 ): (4.10) { d(y0,x 0 ) d(z,x 0 ) +d(y 0,z) if κ =0, cos( κd(y 0,x 0 )) cos( κd(z,x 0 )) cos( κd(y 0,z)) if κ>0 and (4.11) d(z,x 0 ) d(y 0,x 0 ). Proof. Without loss of generality, we assume that κ = 1, and thus D κ = π. Let z J λ (y 0 ). By assumption (4.9), d(y 0,x 0 ) < π and λ V (y 0) < π. Since A is π-convex, it follows from Lemma 4.3 that (4.1) d(y 0,z) λ V (y 0 ) < π. Thus one sees that if (4.10) is shown, then cos d(y 0,x 0 ) cos d(z,x 0 ) by (4.10); hence (4.11) holds. Thus we need only prove (4.10). To this end, note by (4.1) that d(z,x 0 ) d(z,y 0 )+d(x 0,y 0 ) λ V (y 0 ) +d(x 0,y 0 ).

14 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 499 Therefore, (4.13) d(x 0,y 0 )+d(y 0,z)+d(z,x 0 ) (λ V (x 0 ) +d(x 0,y 0 )) π. Consider the geodesic triangle Δ(x 0,y 0,z) with vertices x 0,y 0,z. By Proposition 4.(i), the geodesic joining each pair of the vertices of Δ(x 0,y 0,z) is unique. Let α := x 0 zy 0 denote the angle of the geodesic triangle at vertex z, thatis,α satisfies that (4.14) exp 1 z x 0, exp 1 z y 0 = exp 1 z x 0 exp 1 z y 0 cos α. By Proposition 4.(iii), there exists a comparison triangle Δ( x 0, ȳ 0, z) in the model space S such that (4.15) d( x 0, ȳ 0 )=d(x 0,y 0 ), d( x 0, z) =d(x 0,z) and d(ȳ 0, z) =d(y 0,z). Moreover, we have that α ᾱ := x 0 zȳ 0, the spherical angle at z of the comparison triangle. By (4.1), we have that (4.16) cos d( x 0, ȳ 0 )=cosd( x 0, z)cosd(ȳ 0, z)+sind( x 0, z) sin d(ȳ 0, z)cosᾱ. In view of (4.15), to complete the proof, it suffices to show that cos ᾱ 0. Note that the unique geodesic γ zx0 joining z to x 0 is in Γ A z,x 0,andsoE y0 (z) =exp 1 z y 0 is a singleton. Note also that x 0 S(V,A) A and z J λ (y 0 ) A. It follows from definition that there exist v x0 V (x 0 )andv z V (z) such that v x0, exp 1 x 0 z 0 and λv z exp 1 z This, together with the monotonicity of V, implies that exp 1 z y 0, exp 1 z x 0 λ v z, exp 1 z y 0, exp 1 z x 0 0. x 0 λ v x0, exp 1 x 0 z 0. Combining this with (4.14) gives that cos α 0. Since ᾱ α, wehavecosᾱ cos α 0 and complete the proof. The main theorem of this section is as follows. Theorem 4.6. Suppose that M is of the sectional curvature bounded above by κ and that A is closed and D κ -convex. Then the solution set S(V,A) is D κ -convex. Proof. Let x 1,x S(V,A) be such that d(x 1,x ) <D κ. Let c Γ x1,x be the minimal geodesic. Write y 0 := c( 1 ). To complete the proof, it is sufficient to show that y 0 S(V,A). Note d(x i,y 0 ) < Dκ for each i =1,. We can take λ>0such that λ V (y 0 ) +d(x i,y 0 ) < D κ for each i =1,. Since A is D κ -convex, Lemmas 4.4 and 4.5 are applicable to getting that J λ (y 0 ) and (4.17) d(z,x i ) d(y 0,x i ) for each z J λ (y 0 )andi =1,. Take z 0 J λ (y 0 ), and let γ i be the minimal geodesic joining z 0 to x i for i =1,. Define { γ1 (1 t) t [0, γ(t) := 1 ], γ (t 1) t [ 1, 1].

15 500 CHONG LI AND JEN-CHIH YAO Then applying (4.17), we estimate the length of γ by l(γ) =l(γ 1 )+l(γ )=d(z 0,x 1 )+d(z 0,x ) d(x 1,y 0 )+d(y 0,x )=d(x 1,x ) l(γ). This implies that γ is the minimal geodesic joining x 1 and x. Hence γ = c thanks to Proposition 4.(i). This implies that z 0 c([0, 1]) and thus y 0 = z 0 J λ (y 0 )(as d(z 0,x 1 )=d(z 0,x )). Consequently, y 0 S(V,A) by Lemma 4.3, and the proof is complete. Note that a Hadamard manifold is of the sectional curvature bounded above by κ =0andD κ =+. Therefore, from the above theorem we have immediately the following corollary, which was claimed in [6] for any continuous univalued vector field V, but the proof provided there is not correct. Corollary 4.7. Suppose that M is a Hadamard manifold and that A is convex. Then the solution set S(V,A) is convex. For a general Riemannian manifold, we have the following result. Theorem 4.8. Suppose that M is a complete Riemannian manifold and A is locally convex. Then the solution set S(V,A) is locally convex. Proof. Without loss of generality, we assume that S(V,A) isnonempty. Let x S(V,A). Then there exists 0 < r <r x such that B( x, r) A is strongly convex. Since B( x, r) is compact, it follows that the sectional curvature on B( x, r) is bounded, and so there exists some κ>0such that the sectional curvature on B( x, r) is bounded above by κ. This, together with Proposition.4, implies that N := B( x, r) is a totally geodesic submanifold of M with the bounded curvature above by κ. Moreover, by Proposition.5, T x N = T x M for each x N. This means that the restriction V N of V to N is a vector field on N. Let 0 <R< r and set A R := B( x, R) A. Then A R N is compact in N. Consider the variational inequality problem (3.1) with N and V N in place of M and V, and let S N (V N,A R ) denote the corresponding solution set. Then S N (V N,A R ) B( x, R) =S(V,A R ) B( x, R) S(V,A) B( x, R), where the equality holds because N is a totally geodesic submanifold of M. Furthermore, by Proposition 3., we have that S(V,A R ) B( x, R) S(V,A) B( x, R); hence S N (V,A R ) B( x, R) =S(V,A) B( x, R). Note that A R is strongly convex in N, and thus so is D κ -convex. Thus Theorem 4.6 is applied (to A R and N in place of A and M) togetthats N (V,A R )isd κ -convex in N. This, together with Proposition 4., yields that S N (V,A R ) B( x, r) is strongly convex in N for any 0 <r Dκ. Taking 0 <r min{r, Dκ } and noting that N is a totally geodesic submanifold of M, we have that S(V,A) B( x, r) =S N (V,A R ) B( x, r) is strongly convex in M. Thuswe showed that S(V,A) is locally convex as x S(V,A) is arbitrary. 5. Proximal point methods. As in the previous section, we assume throughout this whole section that A M is a nonempty closed subset and V V(A) is monotone on A. This section is devoted to the study of convergence of the proximal point algorithm for the variational inequality problem (3.1). Let x 0 D(A) and {λ n } (0, + ). The proximal point algorithm (with initial point x 0 )consideredin this section is defined as follows. Letting n =0, 1,,... and having x n,choosex n+1 such that (5.1) x n+1 J λn (x n ) for any n N. Let x M and r>0. Recall from [1, p. 7] that B(x, r) isatotally normal ball around x if there is η>0 such that for each y B(x, r) wehaveexp y (B(0,η))

16 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 501 B(x, r), and exp y ( ) is a diffeomorphism on B(0,η) T y M. The supremum of the radii r of totally normal balls around x is denoted by r(x), i.e., (5.) r(x) :=sup { r>0 B(x, r) is a totally normal ball around x }. Here we call r(x) the totally normal radius of x. By [1, Theorem 3.7] the totally normal radius r(x) is well defined and positive. The next proposition is take from [40, Lemma 3.6] and is useful in what follows. Proposition 5.1. Let z M, andletx, y B(z,r(z)) with x y. Then, for any u T x M and v T y M, one has that (5.3) ( d ds d(exp x su, exp y sv) ) s=0 = 1 d(x, y) ( u, exp 1 x y + v, exp 1 y x ). We still need the following two lemmas, the first of which was proved in [34]. Recall that κ 0. Lemma 5.. Suppose that M is of the curvature bounded above by κ. Let x, y, z M be such that κ max{d(x, y), d(x, z), d(z,y)} r< π. Then the following assertion holds: (5.4) d(exp x s exp 1 x z,exp y s exp 1 y z) sin(1 s)r d(x, y) for each s [0, 1]. sin r Lemma 5.3. Let z A, andlete z betheset-valuedvectorfielddefinedby(4.). Suppose that M is of the sectional curvature bounded above by κ and A is D κ -convex. Then E z is strongly monotone on B(z, r) for each 0 < r < Dκ 4. Consequently, E z is strictly monotone on B(z, Dκ 4 ). Proof. By Proposition 4., the geodesic joining x to y is unique, and exp 1 x y is a singleton for any x, y B(z, Dκ ). This means that the vector field E z =exp 1 x y is univalued on B(z, Dκ Dκ ). Let 0 < r < 4 and let x, y B(z, r). It suffices to show that, if the geodesic joining x to y is contained in A, then the following inequality holds: (5.5) exp 1 x z,exp 1 x y exp 1 y z,exp 1 y x κ r cot( κ r)d (x, y). To show (5.5), note by definition that r(z) Dκ 4 and so B(z, r) B(z,r(z)). Thus Proposition 5.1 is applicable to getting that ( d (5.6) ds d(exp x s exp 1 x z,exp y s exp 1 y z) = 1 d(x, y) ( exp 1 x z,exp 1 x ) s=0 y + exp 1 y z, exp 1 y x ). On the other hand, let r := κ r. Then π κ max{d(x, y), d(y, z), d(z,x)} r<. Then we apply Lemma 5. to conclude that ( ) d ds d(exp x s exp 1 x z,exp y s exp 1 y z) lim s 0 + sin(1 s)r s=0 sin r 1 d(x, y) =(r cot r)d(x, y). s

17 50 CHONG LI AND JEN-CHIH YAO This, together with (5.6), implies that 1 d(x, y) ( exp 1 x z,exp 1 x y + exp 1 y z, exp 1 y x ) r(cot r)d(x, y). Hence (5.5) holds, and the proof is complete. The following corollary is useful. Corollary 5.4. Suppose that M is of the sectional curvature bounded above by κ and that A is weakly convex. Let y 0 A and λ>0 be such that λ V (y 0 ) < Dκ 4. Then J λ (y 0 ) is a singleton. Proof. Let A 0 := A B(y 0, Dκ 4 ). By Lemmas 4.3 and 4.4, we have that J λ (y 0 ) A 0.Furthermore,V λ0,x 0 is strictly monotone on A 0 because E x0 is strictly monotone on B(x 0, Dκ 4 ) by Lemma 5.3, and V is monotone on A by assumption. Hence the assumptions of Theorem 3.13 are satisfied, and the conclusion follows. Recall that the proximal point algorithm (5.1) is well-defined if, for each n N, J λn (x n ) is a singleton. For the following proposition, which provides a sufficient condition ensuring the well-posedness of the algorithm, we define inductively the sequence of (set-valued) mappings {J n } by J n+1 := J λn J n for any n =0, 1,,..., where J 0 := I, the identity mapping. Proposition 5.5. Suppose that M is of the sectional curvature bounded above by κ and A is weakly convex. Let x 0 A be such that (5.7) λ n V (J n (x 0 )) D κ 4 Then algorithm (5.1) is well-defined. Proof. By assumption (5.7), for each n =0, 1,,... λ 0 V (x 0 ) = λ 0 V (J 0 (x 0 )) D κ 4. Thus applying Corollary 5.4 (to λ 0 and x 0 in place of λ and y 0 ), we get that J λ0 (x 0 ) is a singleton. Thus x 1 is well-defined. Similarly, we have from (5.7) again that λ 1 V (x 1 ) = λ 1 V (J 0 (x 0 )) D κ 4 and so x is also well-defined by Corollary 5.4. Thus, the well-defindedness of algorithm (5.1) can be established by mathematical induction, and the proof is complete. Remark 5.1. Let {λ n } (0, + ). We have the following assertions: (a) If M is a Hadamard manifold, algorithm (5.1) is well-defined because D κ = + and condition (5.7) holds automatically. (b) If algorithm (5.1) generates a sequence {x n } satisfying that (5.8) λ n V (x n ) D κ for each n =0, 1,..., 4 then condition (5.7) holds and algorithm (5.1) is well-defined. Recall that a sequence {x n } in complete metric space X is Fejér convergent to F X if d(x n+1,y) d(x n,y) for each y F and each n =0, 1,,...

18 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 503 The following proposition provides a local convergence result for the proximal point algorithm. Theorem 5.6. Suppose that M is of the sectional curvature bounded above by κ and A is weakly convex. Let V V(A) be a monotone vector field satisfying S(V,A). Letx 0 A be such that d(x 0, S(V,A)) < Dκ 8 and the algorithm (5.1) is well-defined. Suppose that {λ n } satisfies both condition (5.8) and (5.9) lim inf n d(x n+1,x n ) λ n =0. Then {x n } converges to a solution of the variational inequality problem (3.1). Proof. WriteF := S(V,A) B(x 0, Dκ 4 ). Then F. Letˆx 0 S(V,A) such that d(x 0, ˆx 0 ) < Dκ 8. We first prove that the following two assertions hold for any x F and any n =0, 1,...: (5.10) cos( κd(x n,x)) cos( κd(x n+1,x)) cos( κd(x n+1,x n )), (5.11) d(x n+1,x 0 ) d(ˆx 0,x 0 ) < D κ 4 For this purpose, let x F.Then and d(x n+1,x) d(x n,x) d(x 0,x). λ 0 V (x 0 ) +d(x 0,x) < D κ. Hence, Lemma 4.5 is applicable (with x 0,x in place of y 0,x 0 ). Thus, by (4.10) and (4.11), we obtain that cos κd(x 0,x) cos κd(x 1,x)cos κd(x 1,x n ) and d(x 1,x) d(x 0,x). That is, assertion (5.10) and the second assertion in (5.11) are shown. In particular, we have that d(x 1, ˆx 0 ) < d(ˆx 0,x 0 ) because ˆx 0 F by the choice of ˆx 0. It follows that d(x 1,x 0 ) < d(x 1, ˆx 0 )+d(ˆx 0,x 0 ) d(ˆx 0,x 0 ) < D κ 4, and the first assertion in (5.11) is also proved. Thus assertions (5.10) and (5.11) hold for any x F and n = 0. Then one can use mathematical induction to show that they hold for any x F and any n =0, 1,... To verify the convergence of the sequence {x n },wenotethat{x n } is Fejér convergent to F by (5.11); hence {x n } is bounded. This, together with assumption (5.9), implies that there exists a subsequence {n k } such that x nk x for some x A and (5.1) exp 1 x nk +1 x n k λ nk = d(x n k +1,x nk ) λ nk 0. Then, by the first assertion in (5.11), we have d( x, x 0 ) = lim k d(x nk,x 0 ) d(ˆx 0,x 0 ) < D κ 4 ; hence x B(x 0, Dκ 4 ). Moreover, we conclude by (5.10) and (5.11) that lim cos( κd(x nk +1,x nk )) = 1, n

19 504 CHONG LI AND JEN-CHIH YAO and so lim n d(x nk +1,x nk )=0. This means that x nk +1 x. Let R := Dκ 4 and o := x. Without loss of generality, we assume that {x nk }, {x nk +1} are in B( x, R). By definition, x nk +1 J λnk (x nk ), and so there exists v nk +1 V (x nk +1) such that (5.13) λ nk v nk +1 exp 1 x nk +1 x n k, exp 1 x nk +1 z 0 for each z A R := A B(o, R). Since x nk +1 x and each V (x nk +1) is compact, it follows that {v nk +1} is bounded. Thus we may assume that v nk +1 v for some v T x M (using subsequences if necessary). Noting that V is upper Kuratowski semicontinuous at x, wehavethat v V ( x). Letting k, we get from (5.1) and (5.13) that v, exp 1 x z 0 for each z A R. This shows that x S(V,A R ). Hence, x S(V,A) by Proposition 3., and so x F. Applying the second assertion in (5.11), we obtain that the sequence {d(x n, x)} is monotone, which, together with lim k d(x nk, x) =0,showsthat{x n } converges to x and completes the proof. Theorem 5.7. Suppose that M is of the sectional curvature bounded above by κ and that A is weakly convex. Let V V(A) be a monotone vector field satisfying S(V,A). Let x 0 A be such that d(x 0, S(V,A)) < Dκ 8 and algorithm (5.1) is well-defined. Suppose that {λ n } (0, + ) satisfies (5.14) n=0 λ n = and λ n V (x n ) D κ 4 for each n =0, 1,... Then the sequence {x n } converges to a solution of the variational inequality problem (3.1). Proof. By Theorem 5.6, it suffices to verify that (5.9) holds. To this end, let n N. We assume that κ = 1 for simplicity. Let F = S(V,A) B(x 0, Dκ 4 ), and let x F. Applying Lemma 4.5, we have that (5.10) and (5.11) hold. Clearly (5.10) is equivalent to the following inequality: (5.15) ( ) ( ) ( ) sin d(x n,x) sin d(x n+1,x) +cosd(x n+1,x)sin d(x n+1,x n ). By (5.11), d(x n+1,x) π 4 ; hence ( ) 1 cos d(x n+1,x)sin d(x n+1,x n ) Combining this with (5.15) yields that ( ) d(xn+1,x n ) = d(x n+1,x n ) λ n n=0 λ n n=0 = π n=0 ( ) d(xn+1,x n ) = π π d(x n+1,x n ). ( ( ) ( )) 1 1 sin d(x n,x) sin d(x n+1,x) <. Therefore, (5.9) follows because n=0 λ n =. Combining Theorem 5.7 and Proposition 5.5, together with Remark 5.1(b), we immediately get the following corollary.

20 VARIATIONAL INEQUALITIES ON RIEMANNIAN MANIFOLDS 505 Corollary 5.8. Suppose that M is of the sectional curvature bounded from above by κ and that A is weakly convex. Let V V(A) be a monotone vector field with S(V,A). Let x 0 A be such that d(x 0, S(V,A)) < Dκ 8. Suppose that {λ n } (0, + ) satisfies condition (5.7) and (5.16) λ n =. n=0 Then algorithm (5.1) is well-defined, and the generated sequence {x n } converges to a solution of the variational inequality problem (3.1). In the special case when M is a Hadamard manifold, condition (5.7) and condition d(x 0, S(V,A)) < Dκ 8 hold automatically. Therefore the following corollary is direct and extends [1, Theorem 5.1], which is proved for univalued continuous monotone vector fields V under the stronger assumption that the sequence {λ n } satisfies inf n λ n > 0. Corollary 5.9. Suppose that M is a Hadamard manifold and that A is convex. Let V V(A) be a monotone vector field with S(V,A). Letx 0 A, and suppose that {λ n } (0, + ) satisfies condition (5.16). Then algorithm (5.1) is well-defined, and the generated sequence {x n } converges to a solution of the variational inequality problem (3.1). Consider the special case when A = M. Then the proximal point algorithm (5.1) is reduced to the following for finding a singularity of V : (5.17) 0 λ n V (x n+1 ) E xn (x n+1 ) for each n =0, 1,,..., which, in Hadamard manifold M, is equivalent to the following: (5.18) 0 λ n V (x n+1 ) exp 1 x n+1 x n for each n =0, 1,,... This algorithm was presented and studied in [1] for set-valued vector fields on Hadamard manifolds. The following corollary, which is a direct consequence of Theorem 5.7, extends [1, Corollary 4.8], which was proved on Hadamard manifold M under the stronger assumption that the sequence {λ n } satisfies inf n λ n > 0. Corollary Suppose that M is of the sectional curvatures bounded above by κ. Let V V(M) be a monotone vector field with V 1 (0) := {x M : 0 V (x)}. Let x 0 M be such that d(x 0,V 1 (0)) < Dκ 8,andlet{λ n} (0, + ) satisfy conditions (5.7) and (5.16). Then the sequence {x n } generated by the algorithm (5.17) is well-defined and converges to a point x V 1 (0). One natural question is, Does there exist a positive number sequence {λ n } satisfying conditions (5.7) and (5.16)? The following remark answers this affirmatively. Remark 5.. Let x 0 A be such that d(x 0, S(V,A)) < Dκ 8, and set Λ := sup { V (x) : x B(x 0, Dκ 4 )}.Let{λ k } ( ) 0, Dκ 4Λ satisfy (5.16). Then, by the proofs for Proposition 5.5 and Theorem 5.6, algorithm (5.1) is well-defined, and the generated sequence {x n } is contained in B(x 0, Dκ 4 ). This and the choice of {λ k}, together with the definition of Λ, means that conditions (5.7) and (5.16) are satisfied. 6. Application to convex optimization. As illustrated in [10] and the book [35], lots of nonconvex optimization problems on the Euclidean space can be reformulated as convex ones on some proper Riemannian manifolds. This section is devoted to an application in convex optimization problems on Riemannian manifolds of the convergence results for the proximal point algorithm established in the previous section.