journal of coplexity 6, 459473 (2000) doi:0.006jco.2000.0544, available online at http:www.idealibrary.co on On the Counication Coplexity of Lipschitzian Optiization for the Coordinated Model of Coputation Mehran Mesbahi Departent of Aerospace Engineering and Mechanics, University of Minnesota, Minneapolis, Minnesota and George P. Papavassilopoulos 2 Departent of Electrical Engineering-Systes, University of Southern California, Los Angeles, California Received January 20, 999 We consider the proble of approxiating the axiu of the su of Lipschitz continuous functions. The values of each function are assued to reside at a different eory eleent. A single processing eleent is designated to approxiate the value of the axiu of the su of these functions by adopting a certain protocol. Under certain assuptions on the class of perissible protocols, we obtain the iniu nuber of real-valued essages that has to be transferred between the processing eleent and the eory eleents in order to find the desired approxiation of this axiu. In particular, we exploit the optiality of the nonadaptive protocols for the Lipschitzian optiization proble, studied in the context of inforation-based coplexity, to prove our ain result. 2000 Acadeic Press Key Words: Lipschitzian optiization; counication coplexity; coordinated odel of coputation.. INTRODUCTION Five people are each given a nuber between and 00. A questioner coes along and wants to figure out the su of these five nubers by To who correspondence should be addressed. E-ail: esbahiae.un.edu. 2 E-ail: yorgosbode.usc.edu. 459 0885-064X00 35.00 Copyright 2000 by Acadeic Press All rights of reproduction in any for reserved.
460 MESBAHI AND PAPAVASSILOPOULOS asking each person questions of the type: ``Is your nuber greater than or equal to x?,'' for soe integer x. In return, the questioner expects to receive a ``yes'' or a ``no'' response fro the participant. The questioner has been told by an inforer that the su of the nubers held by the participants does not exceed 00. Suppose that the questioner asks the first person whether the nuber held by hi or her is greater than or equal to 75 and receives in return a ``yes'' answer. Knowing that the total su is not greater than or equal to 00, it would not be wise for the questioner to ask the second person ``Is your nuber greater than or equal to 75?'' since the reply is definitely a ``no.'' In fact, the questioner can use the previous responses to forulate the question for the next person in soe intelligent anner. In a sense, the questioner can adapt the next question by incorporating the answers for the previous questions in its forulation. It is also conceivable that the questioner first figures out the exact nuber that each person has, without considering other people's nubers, and then sus the up. In this case, the total nuber of questions that the questioner has to ask would just be five ties the nuber of questions needed to figure out one person's nuber. In fact one ight suspect that the questioner cannot really do better than this, in ters of the iniizing the total nuber of questions asked, at least in the worst case. In this paper we consider a siilar proble. There are function storage devices, or eory eleents, storing the values of the functions f,..., f, and each function is known to be in the class of Lipschitz continuous functions with odulus k; the class of such functions will be denoted by F k. There is a processing eleent which is designated to approxiate the value of z :=ax x j= f j(x) by asking each eory eleent about the value of the function residing in that eory eleent, at a given point. We are interested to know, under the above restrictions, the iniu nuber of questions that the processing eleent has to ask the eory eleents, in order to be able to approxiate the value of z within an accuracy 0<=<, for all possible f j # F k ( j=,..., ). Although in this case, the processing eleent cannot find z by figuring out the axiu of each function f j ( j=,..., ) separately, our result indicates that in ters of the total nuber of questions needed to be asked fro the eory eleents, the processing eleent cannot do any better than this, at least in the worst case. The iniu nuber of questions needed to approxiate the axiu of the su of functions in F k as described above, shall be referred to as the counication coplexity of the k-lipschitzian optiization for the coordinated odel of coputation. The proble of deterining the counication coplexity is iportant in several settings. First, is the area of parallel and distributed coputation (Bertsekas and Tsitsiklis, 989; Hwang, 994). In this setting, one can
LIPSCHITZIAN OPTIMIZATION 46 consider the processors as having partial inforation regarding the coputational task at a given tie, and hence they counicate aong theselves in order to solve the proble in a distributed anner. It is believed that the aount of counication needed to coplete the coputation in the distributed anner is one of the ain factors that deterine the efficiency of the parallelis eployed (Gentlean, 978; Saad 986). The counication requireents becoe very iportant in the context of very large integrated circuits (VLSI) (Aho et al., 983; Ullan, 984). In particular, it is known that the nuber of bits that is needed to be exchanged between the different parts of the chip is related to the product of the area of the chip and the coputation tie (Ullan, 984). The issue of counication coplexity is also of relevance in the setting of distributed data acquisition and control. In this case, one can consider the processing eleent as the controller which has access to the state of the environent through two or ore sensors. The sensors, due to their liited coputational power can only send functionals of the state of the environent upon receiving a correspondence fro the controller. If the counication aong the controller and the sensors is costly (for exaple due to the congestion of the network), the issue of counication coplexity becoes iportant. In this paper, we will show that for the Lipschitzian optiization proble, the ethodology developed in the context of inforation-based coplexity can be extended to the coordinated odel of coputation. This will be done ainly by utilizing the results pertaining to the optiality of the nonadaptive protocols for the case where there is a single pair of processing and eory eleents. The organization of the paper is as follows. We first provide a very brief survey of the works that have been done in the area of counication coplexity. In Section 2, we provide the iniu aount of notation and preliinaries which enables us to state, ore forally, the proble and the ain result discussed in the paper. Section 3 is devoted to the proof of the ain result... Related Works The study of counication coplexity was initiated by Abelson (980) where functions of the for f: R _R n R, f # C 2 (the class of twice continuously differentiable functions) were studied. In this setting, x # R and y # R n are known by different processors, and each processor can transit functions of their data which are also assued to be in C 2. The counication coplexity is defined to be the iniu nuber of essages that has to be exchanged between the processors in order to exactly evaluate f(x, y). It should be noted that functions considered by Abelson (980) have a very special structure; naely, it is assued that there exists a counication
462 MESBAHI AND PAPAVASSILOPOULOS protocol which can be eployed by the processors in order to obtain the exact value of f(x, y). The work that has been done in the spirit of Abelson (980) includes that of Luo and Tsitsiklis (99). Another strea of work on the counication coplexity was initiated by the work of Yao (979). This line of work is concerned with obtaining the counication coplexity of evaluating a Boolean function f(x, y), where f: X_Y [0, ], and X and Y are finite sets. It is assued that x # X and y # Y are known by two different processors. The counication coplexity is then defined to be the iniu nuber of bits that has to be exchanged between the processors in order to exactly evaluate f(x, y), for all possible values of x # X and y # Y. In this setting, X and Y have a siple structure and the counication coplexity, in essence, is an indication of the behavior of f on the lattice X_Y. We refer the reader to the survey of Orlitsky and El Gaal (988) for a suary of this approach and any possible extensions. More closely related to the approach of the present paper is the work done in the area of inforation-based coplexity (Traub et al., 988; Neirovsky and Yudin, 983), and in particular the work of Sukharev (992). This line of work is concerned with the efficiency of algoriths for probles defined on the infinite-diensional spaces, such as the function integration proble, approxiation, and optiization. In this context, the processing eleent can obtain the values of the function (a eber of an infinite-diensional space) through an oracle. In general, for these probles only approxiate solutions can be obtained. Therefore there is a presence of the paraeter = in all the coplexity results. It turns out that in any situations, the cost of the oracle calls doinates the cost of the entire algorith. One is thus led to consider the counication coplexity, i.e., the iniu nuber of oracle calls needed by the algorith in order to be able to approxiate the solution (the iniizer, the value of the integral, etc.) within an error =. The work of Sukharev (992) is concerned with the sae issue, but his approach relies ore heavily on the iniax odels and various notions of adaptive algoriths. In fact in the introduction of Sukharev (992) it is stated that the ain feature of the work is that ``the process of coputation has been regarded as a controlled process and the algorith as a control strategy.'' Consequently, the optial algoriths are obtained by eploying ethods of operations research, gae theory, and syste analysis. 2. PRELIMINARIES AND THE MAIN RESULT In this section, we provide certain notions which enable us to state the proble considered in the paper ore forally. We then state the ain result.
LIPSCHITZIAN OPTIMIZATION 463 FIG.. The coordinated odel of coputation. We consider a odel of coputation, referred to as the coordinated odel of coputation, where a single processing eleent (PE) is connected to eory eleents (MEs) via dedicated channels. This odel is shown in Fig.. Each eory eleent ME j has access to the value of the Lipschitz continuous function f j ( j=,..., ), with Lipschitz odulus k, at an arbitrary point of its doain. Let us denote by F k the class of Lipschitz continuous functions with constant k, defined on the p-diensional unit cube [0, ] p. The value of f j at the point x will be denoted by x( f j ), j=,...,. ThePE is allowed to specify x to each ME in an arbitrary anner and receive in return the value x( f j ) with infinite precision. The operation of specifying x by the PE and receiving the value x( f j ) fro the ME j is counted as one counication operation. The objective of the PE is to approxiate : x #[0,] p j= z := ax f j (x) (2.) within an accuracy 0<=<. Let us denote the total inforation gathered by the PE after n such inforation gathering operations by I n ; i.e., I n contains the values of different f j 's at various points of the doain [0, ] p. The PE then applies a functional ; to I n and coes up with an estiate of z (2.). We shall refer to ; as the terinal operation. The process of gathering the inforation I n, and applying the terinal operation ;, will be referred to as the counication protocol. Under the aforeentioned restrictions on the counication protocol, let us define the counication coplexity of the k-lipschitzian optiization for the coordinated odel of coputation (with eory eleents) as 2 (=, k) :=[in n: ; (I n )&z =, \f # F k ]. (2.2)
464 MESBAHI AND PAPAVASSILOPOULOS The counication coplexity 2 (=, k) is the iniu nuber of questionanswer sessions that the PE needs to conduct in order to coe up with an 0<=< approxiation of ax [0, ] p j= f j. The PE coes up with this approxiation by applying ; to the inforation gathered during the n operations of the for x( f j ). Moreover, the approxiation should be valid for all possible f j # F k ( j=,..., ). The ain result of the paper can now be stated as follows. Theore. For the k-lipschitzian optiization and for 0<=<, 2 (=, k)=2 (=, k). (2.3) In the rest of the paper, we shall present the proof of Theore. First however, we need soe ore preliinaries. 2.. More Preliinaries Consider the PEME's configuration shown in Fig.. We assue that f j :[0,] p R, f j # F k, and that ME j has access to the values of f j at any point x #[0,] p, which will be denoted by x( f j ) (j=,..., ). As it is custoary in (eleentary) functional analysis, we think of x both as a point in [0, ] p and as a functional on F k, where F k := [f:[0,] p R, f(x)&f( y) k &x& y&, \x, y #[0,] p ], (2.4) and &}& denotes the infinity (supreu) nor. Each of the channels shown in Fig., between the PE and the MEs, can carry a real nuber, x( f j ), in response to the x subitted by the PE to the ME j. The point subitted to the ME j at tie i will be denoted by x i ( f j ). Let I n =(x,...,, x ( f k ),..., ( f kn )), where [k,..., k n ][,..., ] n. The PE can stop the inforation gathering process at tie n and apply the terinal operation ; to I n, in order to approxiate z (2.). The basic question considered in the present work is the inial value of n, as a function of 0<=<, such that for all possible choices of f j # F k ( j=,..., ), ; (I n )&z =. In order for the PE to coe up with the point x i to be subitted to soe ME at tie i, it eploys the previously gathered inforation by using soe strategy. Let us denote by x~ i the strategy eployed by the PE at tie i to coe up with the new point x i that is to be subitted to soe ME. The (n+) tuple :~ n :=(x~,..., x~ n, ; ) will be called an nth degree (deterinistic) protocol for approxiating z (2.). The protocol :~ n can in fact be described by the following sequence of appings and functionals,
LIPSCHITZIAN OPTIMIZATION 465 x~ #x : F k R; x #[0,] p, x~ 2 :[0,] p _R [0, ] p (which yields x 2 ); x 2 : F k R, n& n& b x~ n :[0,] p _}}}_[0,] p _R_}}}_R[0, ] p (which yields ); n : F k R, ; :[0,] p _}}}_[0,] p _R_}}}_RR. Note that in general, the PE uses all the previous points chosen, and all the corresponding function values at those points, to coe up with the new point. If the new point to be subitted to the ME j is independent of the previous questions, for all j=,..., and for all tie instances i, then we call the protocol (strongly) nonadaptive. In this case, x~ i #x i : F k R, for all i. For the nonadaptive protocols, since the point x i subitted to the ME j solely depends on F k, it follows that the sae point should also be chosen for all j ( j=,..., ) and that the nuber of points specified for all MEs should be equal. Without loss of generality, for the nonadaptive case, we shall assue that the points are subitted to the MEs by the PE in the round-robin anner; i.e., the MEs are indexed fro through, and the points are subitted to the MEs starting fro ME through ME, and then back to ME, and so on. For the nonadaptive protocols, let l denote the nuber of points specified to each one of the MEs; n=l. The judicious choice of the terinal operation in the nonadaptive case would be n ; (I n )= ax k=0,..., l& : j= x k+ j ( f j ). (2.5) For our purpose, the role played by the nonadaptive protocols is of central iportance; the results pertaining to the nonadaptive protocols are easily extendible fro = to the case where >, as will be shown in the next section. Let A n and A n denote the class of nth degree adaptive and nonadaptive protocols, respectively. Then for a specific set of f,..., f # F k, residing at ME j ( j=,..., ), the error associated with using the protocol :~ n to approxiate z (2.) is defined to be e(:~ n, k, f,..., f ):= z&; (I n ). (2.6)
466 MESBAHI AND PAPAVASSILOPOULOS The least guaranteed error for the class of nth degree adaptive protocols for the k-lipschitzian optiization will then be E adaptive (n, k) := inf :~ n # A n and for the nonadaptive case it will be sup e(:~ n, k, f,..., f ) (2.7) f,..., f # F k E non-adaptive (n, k) := inf sup e(:~ n, k, f,..., f ). (2.8) :~ n # A n f,..., f # F k Although E adaptive (n, k) (n, k) by definition (since A n A shall nevertheless use the following notation: n ), we E (n, k) :=in[ (n, k), E adaptive (n, k)]. (2.9) Clearly the only interesting scenario would be when E (n, k)= (n, k), which is the case if and only if (n, k)=e adaptive (n, k). We now define siilar definitions for the counication coplexity of the k-lipschitzian optiization. In particular, let us define the following: and, 2 adaptive (=, k):=in[n: E adaptive (n, k)=], (2.0) 2 nonadaptive (=, k):=in[n: (n, k)=], (2.) 2 (=, k):=in[n: E (n, k)=]. (2.2) Our approach for proving Theore is along the proofs of the following two propositions. Proposition 2. Proposition 3. 2 nonadaptive (=, k)=2 (=, k). (2.3) 2 nonadaptive (=, k)=2 (=, k). (2.4)
LIPSCHITZIAN OPTIMIZATION 467 Theore will then follow by cobining the stateents of Propositions 2and3. 3. THE PROOF OF THE MAIN RESULT In this section, we present the proofs of Propositions 2 and 3. The stateent of Theore then follows iediately fro the results of these two propositions. Proposition 2 deals with the proble of extending the result pertaining to the counication coplexity for = to the case of >. When =, the coordinated odel of coputation shown in Fig. reduces to the oracle-type achine considered in the context of inforation-based coplexity. The following result of Sukharev (992) plays a central role in our analysis. We reind the reader that we are dealing only with the Lipschitzian optiization proble. Theore 4 (Sukharev (992, Theore.2, p. 25)). (n, k)=e (n, k)= k 2 wn p x. For all k>0, An obvious iplication of Theore 4 is that for all >, and for fixed k>0, (n, k)=e (n, k)=e (n, k). As a corollary to Theore 4 we also obtain Corollary 5. For all k>0, 2 nonadaptive (=, k)=2 (=, k)= \ 2=+ k p. We now present the proof of Proposition 2. Proof (Proposition 2). As it was pointed out in Introduction, for the nonadaptive protocols we shall fix the operation ;, as defined in (2.5). In this case, the points subitted to each ME j ( j=,..., ) by the PE depend solely on the functional class F k. If we use the optial nonadaptive protocol for the single ME case on all the MEs and use the terinal operation (2.5), we obtain a nonadaptive protocol for the ME case and thus,
468 MESBAHI AND PAPAVASSILOPOULOS (n, k)= (l, k) (l, k) = (l, k), since this protocol is now exactly an lth degree protocol for the single ME case as applied to a function in F k. Consequently, 2 nonadaptive (=, k)2 nonadaptive \ =, k + =2nonadaptive (=, k). Consider now a situation where the functions residing in the MEs are all identical and, oreover, where the PE is aware of this fact when subitting the questions to the MEs. The best achievable error bound in this case would be (l, k), indicating that this scenario is exactly the sae as the one where the PE is counicating with one ME, with the knowledge that the function residing in that ME is a eber of F, k. Thereby, inf sup e(:~ n, k, f,..., f )= inf sup e(:~ l, k, f ) :~ n # A n f = f 2 =}}}=f # F k :~ l # A l f # F k which translates to Thus, = (l, k) (n, k), (l, k) (n, k). 2 nonadaptive \ =, k + =2nonadaptive (=, k)2 nonadaptive (=, k). K Using Corollary 5, we also conclude that 2 nonadaptive (=, k)=2 (=, k). We now present the proof of Proposition 3. Proposition 3 states that for the k-lipschitzian optiization on the coordinated odel of coputation, the PE cannot do any better than using an optial nonadaptive protocol (in ters of reducing the aount of counication needed in the worst case).
LIPSCHITZIAN OPTIMIZATION 469 Proof (Proposition 3). It suffices to show that for all n, (n, k)=e adaptive (n, k). The proof is essentially a straightforward generalization of the proof of Sukharev for the single eory case. We present the proof in three steps. First, soe notations are introduced. Consider a nonadaptive protocol with the fixed terinal operation defined as in (2.5). Then the protocol is erely specified by the points x i subitted to each PE. In particular, define :=(x,..., ) and write (n, k)=inf sup e(, k, f,..., f ), f,..., f # F k where e(, k, f,..., f ) is the error of the approxiation when the terinal operation is fixed (siilarly we shall use the notation e(x~ n, k, f,..., f ) for the adaptive case). Define the set and F(, f,..., f n ):=[( f $,..., f $ ) x i ( f $ j )=x i ( f j ); i= j od ] e F (, f,..., f )= : ( f $,..., f $ )#F(, f,..., f n ) The three steps of the proof are as follows: e(, k, f $,..., f $ ).. There exist functions f j ( j j) such that e F (, f,..., f ) e F (, f,..., f ), for all f j # F k. 2. Provided that () holds, e F (, f,..., f ) has a generalized saddle point, that is, inf sup e F (, f,..., f )= sup f,..., f f,..., f inf e F (, f,..., f ). 3. Provided that (2) holds, the stateent of the Proposition is then proved. We provide the proof for each part.. Fix and let l=n. Forf $ j # F k ( j) define, f j := \ : j f $ j & ax k=0,..., l& : x k+ j ( f j ), j= ++
470 MESBAHI AND PAPAVASSILOPOULOS where f + =ax( f, 0). Note that for each j, f j # F k, since if g # F k,then ax(g, 0)#F k, and g # F k. Now, Thereby, and consequently, x i ( f j)=0, in, i= j od. e(, k, f $,..., f $ )=e(, k, f,..., f )e F (,0,...,0), for all f $ j # F k ( j). 2. In general one has, e F (, f $,..., f $ )e F (,0,...,0) sup f,..., f inf e F (, f,..., f )inf sup e F (, f,..., f ). f,..., f To show that the inequality also holds in the reverse direction, we observe that, sup f,..., f inf e F (, f,..., f )inf e F (, f,..., f ) inf sup e F (, f,..., f ). f,..., f 3. We now show that in view of () and (2) above, (n, k)e adaptive (n, k). By the definition of e F (, f,..., f ), one has (n, k)=inf sup e F (, f,..., f ). f,..., f For any $>0, there exists f $ j ( j) such that inf e F (, f $,..., f $ )=inf inf sup e F (, f,..., f )&$ f,..., f sup e(, k, f,..., f )&$, f,..., f
LIPSCHITZIAN OPTIMIZATION 47 which iplies that for all, e F (, f $,..., f $ )inf sup e(, k, f,..., f )&$ f,..., f For any fixed adaptive protocol :~ n =(x~ n, ; ), let $ be the realization of the strategy x~ n for the particular functions f $ j ( jn). Then, sup e$(x~ n, k, f,..., f )e F (, f $,..., f $ ) $ f,..., f inf Since :~ n =(x~ n, ; )and$>0 were arbitrary, sup e F (, f,..., f )&$. f,..., f Hence, inf sup e(:~ n, k, f,..., f )inf :~ n f,..., f sup e(, k, f,..., f ). f,..., f (n, k)e adaptive (n, k). K Having proved Propositions 2 and 3, we now copare the equations (2.3) and (2.4) and obtain, 2 (=, k)=2 (=, k), which is the stateent of Theore. As a corollary of Theore, we also obtain an expression for the counication coplexity of the k-lipschitzian optiization for the coordinated odel of coputation. Corollary 6. For 0<=<, p 2 (=, k)= 2=+ \k. (3.5) Proof. Since E (n, k)=k2 wn p x by Theore 4, 2 (=, k)=w(k2=) p X. The stateent of the corollary now follows using Theore. K
472 MESBAHI AND PAPAVASSILOPOULOS 4. CONCLUSION We have addressed the proble of deterining the counication coplexity of Lipschitzian optiization for the coordinated odel of coputation. The ain concept that has been exploited in this direction is the optiality of a nonadaptive protocol aong the class of all perissible protocols. This result can be viewed as a generalization of the result of Sukharev for the oracle-type achines considered traditionally in the context of inforationbased coplexity. There are several directions along which this work can be continued. For exaple, it would be of interest to consider the counication coplexity for ore general distributed configurations, such as the case where ore than one processing eleent is present. ACKNOWLEDGMENTS The research of G. P. Papavassilopoulos has been supported in part by the National Science Foundation Grant CCR-9222734. The authors thank the referee for the correction to the original proof of Proposition 2. REFERENCES. H. Abelson, Lower bounds on inforation transfer in distributed coputation, J. Assoc. Coput. Mach. 27 (980), 384392. 2. A. V. Aho, J. D. Ullan, and M. Yannakakis, On notion of inforation transfer in VLSI circuits, in ``Proceedings of 5th STOC,'' pp. 3339, 983. 3. D. P. Bertsekas and J. N. Tsitsiklis, ``Parallel and Distributed Coputation,'' Prentice Hall, New York, 989. 4. W. M. Gentlean, Soe coplexity results for atrix coputations on parallel achines, J. Assoc. Coput. Mach. 25 (978), 25. 5. K. Hwang, ``Advanced Coputer Architecture,'' McGrawHill, New York, 994. 6. Z. Luo and J. N. Tsitsiklis, On the counication coplexity of solving a polynoial equation, SIAM J. Coput. 20 (99), 936950. 7. A. S. Neirovsky and D. B. Yudin, ``Proble Coplexity and Method Efficiency in Optiization,'' Wiley, New York, 983. 8. A. Orlitsky and A. El Gaal, Counication coplexity, in ``Coplexity in Inforation Theory'' (Y. S. Abu-Mostafa, Ed.), pp. 206, Springer-Verlag, BerlinNew York, 988. 9. Y. Saad, Counication coplexity of Gaussian eliination algorith on ultiprocessors, Linear Algebra Appl. 77 (986), 35340. 0. A. G. Sukharev, ``Miniax Models in the Theory of Nuerical Methods,'' Kluwer Press, DordrechtNorwell, MA, 992.
LIPSCHITZIAN OPTIMIZATION 473. J. F. Traub, G. W. Wasilkowski, and H. Wazniakowski, ``Inforation-Based Coplexity,'' Acadeic Press, San Diego, 988. 2. J. N. Tsitsiklis and Z. Q. Luo, Counication coplexity of convex optiization, J. Coplexity 3 (987), 23243. 3. J. Ullan, ``Coputational Aspects of VLSI,'' Coputer Science Press, Rockville, MD, 984. 4. A. C. Yao, Soe coplexity questions related to distributed coputing, in ``Proceedings of th STOC,'' pp. 20923, 979.