J. Parallel Distrib. Comput.

Transcription

1 J. Parallel Dstrb. Comput. 7 (20) Contents lsts avalable at ScenceDrect J. Parallel Dstrb. Comput. ournal homepage: Game-theoretc statc load balancng for dstrbuted systems Satsh Penmatsa a, Anthony T. Chronopoulos b, a Department of Math. & Computer Scence, Unversty of Maryland Eastern Shore, Prncess Anne, MD 2853, Unted States b Department of Computer Scence, Unversty of Texas at San Antono, One UTSA Crcle, San Antono, TX 78249, Unted States a r t c l e n f o a b s t r a c t Artcle hstory: Receved 3 June 2009 Receved n revsed form 28 November 200 Accepted 30 November 200 Avalable onlne 7 December 200 Keywords: Load balancng Dstrbuted systems Cooperatve game Non-cooperatve game Expected response tme Farness In ths paper, we present a game theoretc approach to solve the statc load balancng problem for sngleclass and mult-class (mult-user) obs n a dstrbuted system where the computers are connected by a communcaton network. The obectve of our approach s to provde farness to all the obs (n a sngleclass system) and the users of the obs (n a mult-user system). To provde farness to all the obs n the system, we use a cooperatve game to model the load balancng problem. Our soluton s based on the Nash Barganng Soluton (NBS) whch provdes a Pareto optmal soluton for the dstrbuted system and s also a far soluton. An algorthm for computng the NBS s derved for the proposed cooperatve load balancng game. To provde farness to all the users n the system, the load balancng problem s formulated as a non-cooperatve game among the users who try to mnmze the expected response tme of ther own obs. We use the concept of Nash equlbrum as the soluton of our non-cooperatve game and derve a dstrbuted algorthm for computng t. Our schemes are compared wth other exstng schemes usng smulatons wth varous system loads and confguratons. We show that our schemes perform near the system optmal schemes and are superor to the other schemes n terms of farness. 200 Elsever Inc. All rghts reserved.. Introducton A dstrbuted system often conssts of heterogeneous computng and communcaton resources. Due to the possble dfferences n the computng capactes and uneven ob arrval patterns, the workload on dfferent computers n the system can vary greatly [5,3]. Improvng the performance of such a system by an approprate dstrbuton of the workload among the computers s commonly known as load balancng. The load balancng schemes can be ether statc or dynamc [46]. The statc schemes ether do not use any system nformaton or use only the average system behavor whereas the dynamc schemes consder nstantaneous system states (runtme state nformaton) n the ob allocaton calculatons. However, as the overhead costs for the exchange of system state nformaton ncrease, the statc schemes can perform equally well or better compared to dynamc schemes [53]. The lower complexty or the mnmal runtme overhead of the statc schemes s also an added advantage. Another maor drawback of the dynamc schemes s Ths work was supported n part by the Natonal Scence Foundaton under grant number CCR Ths research of A.T. Chronopoulos was partly supported by a NSF grant (HRD ) to the Unversty of Texas at San Antono. Correspondng author. E-mal addresses: spenmatsa@umes.edu (S. Penmatsa), atc@cs.utsa.edu (A.T. Chronopoulos). URLs: (S. Penmatsa), (A.T. Chronopoulos). ther senstvty to naccurate nformaton used for ob allocaton purposes. Some dynamc allocatons can result n extremely poor system performance even when the nformaton accuracy s only slghtly less than 00% [49,48]. Also, obs n a dstrbuted system can be dvded nto dfferent classes based on ther resource usage characterstcs and ownershp. Based on the number of ob classes consdered, we have a sngle-class or mult-class (mult-user) ob dstrbuted system. In ths paper, we consder the statc load balancng problem for both sngle-class obs and mult-user obs n a dstrbuted computer system that conssts of heterogeneous host computers (nodes) nterconnected by a communcaton network. Jobs arrve at each computer accordng to a tme-nvarant exponental process. Load balancng s acheved by transferrng some obs from nodes that are heavly loaded to those that are dle or lghtly loaded. A communcaton delay wll be ncurred as a result of sendng a ob to a dfferent computer for processng. Snce all the obs belongng to the same user (or same class) usually have equal prorty or are under the same admnstratve doman, we use a cooperatve game to formulate the load balancng problem for sngle-class obs. In a mult-class (or mult-user) ob envronment, obs belong to varous users and a user prefers to have her/hs obs executed frst (or faster) than others. Because of ths selfsh nature, we use a non-cooperatve game to model the load balancng problem for mult-user obs. The expected (mean) response tme of a ob or a user or the system used n ths paper s defned as the total tme to execute a ob or all the users obs or /$ see front matter 200 Elsever Inc. All rghts reserved. do:0.06/.pdc

2 538 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) all the obs n the system whch ncludes the processng tme(s) at a node or nodes (processng delay), any queung delays, and any communcaton delays [22]... Load balancng for sngle-class obs Ths load balancng problem s formulated as a cooperatve game among the computers and the communcaton subsystem. The several decson makers (e.g., computers and the communcaton subsystem) cooperate n makng decsons such that each of them wll operate at ts optmum. The decson makers have complete freedom of pre-play communcaton to make ont agreements about ther operatng ponts. Based on the Nash Barganng Soluton (NBS) whch provdes a Pareto optmal and far soluton, we provde an algorthm (CCOOP) for computng the NBS for our cooperatve load balancng game. The obectve of ths cooperatve load balancng scheme s to provde farness to all the obs,.e. all the obs (of approxmately the same sze) should experence approxmately the same expected response tme ndependent of the computers allocated for ther executon..2. Load balancng for mult-user obs Ths problem s formulated, takng nto account the users mean node delays and the mean communcaton delays, as a non-cooperatve game among the users. Each user mnmzes her/hs own response tme ndependently of the others and they all eventually reach an equlbrum. We use the concept of Nash equlbrum as the soluton of our non-cooperatve game and derve a dstrbuted algorthm (NCOOPC) for computng t. The obectve of ths non-cooperatve load balancng scheme s to provde farness to all the users.e. all the users should have approxmately the same expected response tme ndependent of the computers allocated for the executon of ther obs (of approxmately the same sze). Remark.. In the above we do not mean that the computers or the users engage n games, but, the load balancng problems wll be solved usng game theory models and the soluton of the games wll be used for ob allocaton..3. Motvaton and contrbuton Most of the prevous studes on statc load balancng consdered the mnmzaton of the overall system expected response tme as ther man obectve. However, some obs or users may experence much longer expected response tme than others n such allocatons. Also, past load balancng algorthms whose obectve s to provde farness dd not take the communcaton costs nto account. In current dstrbuted systems, especally Grd computng systems [0], the computng resources are dstrbuted over the globe and so communcaton delays wll be ncurred because of ob transfers whch can play an mportant role n load balancng. Here, we consder the statc load balancng problem for both sngle-class obs and mult-user obs n a dstrbuted computer system wth the obectve of provdng farness to all the obs (n the sngle-class ob system) and the users of the obs (n the mult-user ob system) by takng the communcaton costs nto account. Farness of allocaton s an mportant factor n modern dstrbuted systems and our schemes wll be sutable for systems n whch the far treatment of the obs or users s as mportant as other performance characterstcs. Farness s a maor ssue n many modern utlty computng systems such as Amazon Elastc Compute Cloud [4] and Sun Grd Compute Utlty [38] where users pay the prce for the compute capacty they actually consume. Guaranteeng the farness of allocaton to the users n such fxed prce settngs s an mportant and dffcult problem. To provde farness to all the obs n the system.e. to fnd an allocaton of obs to computers that yelds an equal or approxmately equal expected response tme for all the obs (of approxmately the same sze), we use the framework provded by cooperatve game theory. To provde farness to all the users n the dstrbuted system.e. to fnd an allocaton of users obs to computers that yelds an equal or approxmately equal expected response tme for all the users (wth obs of approxmately the same sze), we use the framework provded by non-cooperatve game theory. We assume all obs are of the same sze n terms of the computaton tme requred to be executed by the slowest computer. In the case where there exst obs of unequal sze then we assume that they are dvsble. We thus assume that they are dvded nto obs of the same sze before they are scheduled for executon. We perform smulatons wth varous system loads and confguratons to evaluate the performance of the proposed load balancng schemes. For comparson, we also mplemented other representatve statc load balancng schemes. These statc schemes are: () schemes that yeld the system-wde optmal expected response tme whch are used as baselne schemes for our experments (OPTIM [26] (whch mnmzes the expected response tme of all the obs n a sngle-class ob system) and GOS [25] (whch mnmzes the expected response tme of all the obs n a mult-class ob system)); and (b) schemes whch allocate obs to computers n proporton to ther computng power and yeld the worst expected response tme of the statc schemes n the lterature (PROP [8] (whch allocates the obs to the computers n proporton to ther processng speeds n a sngle-class ob system) and PROP_M [8] (whch allocates the users obs to the computers n proporton to ther processng speeds n a mult-class ob system)). We show that the proposed load balancng schemes not only provde farness but also perform near the system-wde optmal load balancng schemes..4. Related work Extensve studes have been made on the statc load balancng problem n sngle-class and mult-class ob dstrbuted systems ([5,26,3,50,22,30,32,33,37,43,,6] and references there-n). Most of the above used the global approach, where the focus s on mnmzng the expected response tme of the entre system over all the obs. Dfferent network confguratons were consdered and the problem was formulated as a non-lnear optmzaton problem and as a polymatrod optmzaton problem. The schemes that mplement the global approach determne a load allocaton to obtan a system-wde optmal response tme and the farness of allocaton was not consdered. Load balancng for sngle-class obs based on cooperatve game theory has been studed n [20,9]. However, the communcaton costs were not taken nto account n the above studes and the effect of system sze and communcaton tme on the proposed scheme were not studed. In ths paper, we study game-theoretc load balancng schemes for both sngle-class and mult-class ob systems by takng the communcaton costs nto account and study the effect of system sze and communcaton tme on the proposed schemes. Prelmnary results based on cooperatve game theory for sngle-class ob systems by takng the communcaton subsystem nto account can be found n [40]. Here, we evaluate the performance of the proposed cooperatve scheme (CCOOP) usng 32 computers compared to 6 computers n [40]. Also, n ths paper, the effect of system sze and the communcaton tme on CCOOP are studed, the performance metrcs and other mplemented schemes are explaned n more detal, and a numercal example llustratng CCOOP s provded. In ths paper, we also study the NCOOPC load balancng scheme for mult-class ob systems based on

3 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) non-cooperatve game theory. The obectve of NCOOPC s to provde farness to all the users n the system n terms of ther expected response tme. The performance of NCOOPC s evaluated wth 32 computers and 20 users. The effect of system utlzaton, heterogenety, system sze, and communcaton tme on NCOOPC are studed and a numercal example llustratng NCOOPC s also provded. The proof for computng the best response of the users for Nash equlbrum s also presented. There exst only a few studes that use the non-cooperatve approach for load balancng n dstrbuted systems. Kameda et al. [22] derved load balancng algorthms for sngle and mult-class obs usng non-cooperatve games based on Wardrop equlbrum. In ths case each of the nfntely many obs optmzes ts own response tme ndependently of the others and they all eventually reach an equlbrum. However, under certan condtons [] ths equlbrum load allocaton provdes a sub-optmal systemwde response tme. Non-cooperatve load balancng for fntely many obs based on Nash equlbrum was studed n [8]. However, the communcaton subsystem was not taken nto account. The problem was formulated as a Stackelberg game n [44] and was shown that t s NP-hard to compute the optmal Stackelberg allocaton strategy. Extensve studes were made on the problem of routng traffc n networks usng non-cooperatve game models ([3,28,35,39,2,27] and references there-n) and game theory was also used to model grd systems ([29,23,45,49,48] and references there-n) and for prce-based ob allocaton n dstrbuted systems [6,7,54]. Prelmnary results for ob allocaton n grd systems based on non-cooperatve game theory by takng the communcaton subsystem nto account can be found n [4]. In [5], a workload allocaton polcy (MMP) for heterogeneous systems s studed whose obectve s to provde farness to the obs n the system. More specfcally, MMP mnmzes or elmnates the dfference n expected response tmes at the fastest and slowest computers. It was shown that farness s acheved at the expense of a tolerable ncrease n the overall system expected response tme. MMP assumes that all the obs are of the same type (or belong to the same class) and does not take nto account the communcaton costs for transferrng obs. CCOOP and NCOOPC (studed n ths paper) provde farness to the obs and the users respectvely and the performance of CCOOP and NCOOPC are very close to OPTIM and GOS respectvely for low and medum system loads (OPTIM and GOS provde the mnmum overall expected response tme for sngle and multclass ob dstrbuted systems respectvely). CCOOP and NCOOPC perform sgnfcantly better than PROP and PROP_M respectvely for hgh system loads (PROP and PROP_M are not optmal and n smulatons they yeld the worst overall expected response tme for sngle and mult-class ob dstrbuted systems respectvely). In [49], a load balancng scheme (GT) based on game theory for computatonal grds was proposed. GT s dynamc n nature as t responds to changes n system states durng runtme. Expermental results showed that GT provdes farness to the grd users but wth an ncrease n the overall system expected response tme. However, as the overheads for transferrng the system state nformaton between the nodes ncrease, the effcency and farness of GT decreases. In [49], GT was only compared wth PROP_M (denoted by PS n [49]) and not wth any optmal schemes (e.g. GOS). Sgnfcant results on workload heterogenety for task schedulng n dstrbuted systems have been publshed ([24,7,2] and references there-n). In ths paper, we consder a work model on ob schedulng n heterogeneous computer systems. A ob may consst of one or more tasks. A ob s ready to be executed, when all ts tasks are ready for executon. Jobs n ths paper may belong to dfferent users and dffer n ther arrval rates. node node 2 Fg...5. Organzaton φ x Communcaton Network Processor node Dstrbuted system model for sngle-class obs. β x node n The rest of the paper s organzed as follows. In Secton 2, we study the load balancng for sngle-class obs based on cooperatve game theory. In Secton 3, we study the load balancng for multuser obs based on non-cooperatve game theory. The performance of the proposed cooperatve and non-cooperatve load balancng schemes s evaluated n Sectons 4 and 5 respectvely. Conclusons are drawn and future research drectons are presented n Secton Cooperatve load balancng for sngle-class obs 2.. System model We consder a dstrbuted system model wth n nodes (computers) connected by a communcaton network as shown n Fg.. The termnology and assumptons used are as follows: The nodes and the communcaton network are modeled as M/M/ queung systems [2]. In these queung systems, the nterarrval tmes and the servce (processng) tmes are exponentally dstrbuted and obs arrve n a sngle queue (whch s assumed to have nfnte capacty) to a sngle computng resource wth a Frst Come, Frst Served servce dscplne. The followng enttes characterze M/M/ queung systems. We denote () the external ob arrval rate at node (.e. the number of external obs arrvng at node per unt tme) by φ, () the total external ob arrval rate of the system (.e. the total number of external obs arrvng nto the system per unt tme) by Φ (so, Φ = n φ ), () the maxmum processng rate of node (.e. the maxmum number of obs that can be processed at node per unt tme) by µ, (v) the ob processng rate (or load) allocated by the load balancng scheme for node (.e. the number of obs that are to be processed at node per unt tme) by β, and (v) the ob flow rate from node to node (.e. the number of obs sent from to per unt tme) by x. A ob arrvng at node may be ether processed at node or transferred to another node through the communcaton network for remote processng. The decson of transferrng a ob does not depend on the state of the system and hence s statc n nature. A ob transferred from node to node receves ts servce at node and s not transferred to other nodes. If a node sends (receves) obs to (from) node, node does not send (receve) obs to (from) node. The response tme of a ob n a system as above conssts of a node delay (queung delay + processng delay) at the processng node and also some possble communcaton delay ncurred due to a ob transfer. Let the mean node delay for a ob at node be denoted by D (β ). Modelng each node as an M/M/ queung system [2], D (β ) = µ β, =,..., n. ()

4 540 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) We assume that the expected communcaton delay from node to node s ndependent of the source-destnaton par (, ) but may depend on the total traffc through the network denoted by λ where λ = n n = x. Let the mean communcaton delay for a ob be denoted by G(λ). Modelng the communcaton network as an M/M/ queung system [2], G(λ) = t tλ, λ < t where t s the mean communcaton tme for sendng or recevng a ob. Note that D (β ) and G(λ) are ncreasng postve functons. We also assume that the communcaton delay ncurred as a result of sendng a ob drectly from node to node s less than or equal to the sum of the delays from node to node k and from node k to node. Based on ths, we classfy the nodes n the followng way smlar to [5]: Snk (S): Only receves obs from other nodes but does not send out any obs. Idle source (R d ): Does not process any obs (β = 0) and sends all the obs to other nodes. Does not receve any obs from other nodes. Actve source (R a ): Processes a part of the obs that arrve and sends the remanng obs to other nodes. But, t does not receve any obs. Neutral (N): Processes obs locally wthout sendng or recevng obs. The network traffc λ can be expressed n terms of the varable β as: λ = n φ 2 β. We defne the dfferental node delay (d ), dfferental communcaton delay (g), and nverse of dfferental node delay (d ) as follows: d (β ) = ln D (β ) = (3) β µ β g(λ) = t ln G(λ) = λ ( tλ) µ d x, f x > µ (x) = 0, f x. µ Remark 2.. d (β ) and g(λ) are ncreasng postve functons based on our assumptons on D (β ) and G(λ) Cooperatve load balancng In ths secton, we formulate the load balancng problem as a cooperatve game among the computers and the communcaton network. We consder an n+ player game where the n computers try to mnmze ther mean node delays D (β ) and the (n + )th player, the communcaton subsystem, tres to mnmze the expected communcaton delay G(λ). So, the obectve functon for each computer, =,..., n can be expressed as: f (X) = D (β ) (6) and the obectve functon for the communcaton subsystem can be expressed as: f n+ (X) = G(λ) (7) where X = [β,..., β n, λ] T s the set of strateges of the n + players. Defnton 2. (The Cooperatve Load Balancng Game). The cooperatve load balancng game conssts of: n computers and the communcaton subsystem as players. (2) (4) (5) The set of strateges, X, s defned by the followng constrants: Stablty : β < µ, =,..., n (8) Conservaton : β = φ = Φ, (9) Postvty : β 0, =,..., n. (0) For each computer, =,..., n, the obectve functon f (X) = D (β ); for the communcaton subsystem, the obectve functon f n+ (X) = G(λ); X = [β,..., β n, λ] T. The goal s to mnmze smultaneously all f (X), =,..., n +. For each player, =,..., n +, the ntal performance u 0 = f (X 0 ), where X 0 s a zero vector of length n +. Remark 2.2. In the above defnton, we can assume that β ˆµ to satsfy the compactness requrement for X where ˆµ = µ ϵ for a small ϵ > 0. We gnore ths condton for smplcty. We also assume that all the players n the above game are able to acheve performance strctly superor to ther ntal performance. Theorem 2.. For the cooperatve load balancng game defned above there s a unque barganng pont and the barganng solutons are determned by solvng the followng optmzaton problem: n mn G(λ) D (β ) () X subect to the constrants (8) (0). Proof. In Appendx A. Theorem 2.2. For the cooperatve load balancng game defned above the barganng soluton s determned by solvng the followng optmzaton problem: n mn X ln D (β ) + ln G(λ) subect to the constrants (8) (0). Proof. In Appendx A. (2) Theorem 2.3. The soluton to the optmzaton problem n Theorem 2.2 satsfes the relatons d (β ) α + g(λ), β = 0 ( R d ), d (β ) = α + g(λ), 0 < β < φ ( R a ), α + g(λ) d (β ) α, β = φ ( N), d (β ) = α, β > φ ( S), (3) subect to the total flow constrant, d (α) + φ + d (α + g(λ)) = Φ (4) S N R a where α s the Lagrange multpler. Proof. In Appendx A. The relatons n Theorem 2.3 can be nterpreted as follows: The dfferental node delays of all snks are the same (.e. α). The dfferental node delays of all actve sources are equal; they consst of the dfferental node delay at a snk and the dfferental communcaton delay due to sendng a ob through the network to a snk. The dfferental node delay for neutrals s not less than the dfferental node delay of snks but not greater than the dfferental node delay of actve sources. The dfferental node delay for dle sources s not less than the dfferental node delay of actve sources; ths makes dle sources send all ther obs to snks.

5 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Snce obtanng a closed form soluton for α from Eq. (4) s not possble, we use a smple method such as a bnary search to solve Eq. (4) teratvely for α as n [26]. Ths s descrbed n the CCOOP algorthm below. Usng the α n an teraton of the bnary search, a set of snk (S(α)) and source nodes (R d (α) and R a (α)) are determned and t s checked whether the traffc from the set of source nodes (λ R (α)) equals the traffc nto the set of snk nodes (λ S (α)), n whch case an optmal α s found. The set of neutral nodes are denoted by N(α). (S(α), R d (α), R a (α), N(α), λ S (α), and λ R (α) are defned n Defnton A.4 n Appendx A.) In the followng, we present an algorthm (CCOOP) for obtanng the Nash Barganng Soluton for our cooperatve load balancng game. CCOOP algorthm: Input: Node processng rates: µ, µ 2,... µ n ; Node ob arrval rates: φ, φ 2,... φ n ; Mean communcaton tme: t. Output: Load allocaton to the nodes: β, β 2,... β n.. Intalzaton: β φ ; N; =,..., n. 2. Sort the computers such that d (φ ) d 2 (φ 2 )... d n (φ n ). If d (φ ) + g(0) d n (φ n ), STOP. (No load balancng s requred) 3. Determne α (usng a bnary search): a d (φ ) b d n (φ n ) whle() do λ S (α) 0 λ R (α) 0 α a+b 2 Calculate: S(α), λ S (α), R d (α), R a (α), and λ R (α) (eqs. (72) (76)) n the order gven for =,..., n If ( λ S (α) λ R (α) < ϵ) EXIT If (λ S (α) > λ R (α)) b α else a α 4. Determne the loads on the computers: β 0, for R d (α) β d (α + g(λ)), for R a (α) β d (α), for S(α) β φ, for N(α) The followng remark descrbes the stoppng crtera and the tme complexty of CCOOP. Remark 2.3. () In step 2, we STOP when the total (node + communcaton) tme for a ob to be transferred from a more powerful to a less powerful node exceeds the node tme on the less powerful node, f the network traffc equals 0. Ths means that a ob wll run faster on the orgn node than f transferred to a dfferent node. () The runnng tme of ths algorthm s O(n log n + n log /ϵ), where ϵ denotes the acceptable tolerance used for computng α n step 3 of the algorthm. The followng remark descrbes the mplementaton of CCOOP n practce. Remark 2.4. The CCOOP algorthm must be run perodcally or when the system parameters (system load) change n order to recompute a new load allocaton. For example, the ob arrval rate at a node can be estmated by consderng the number of arrvals over a fxed nterval of tme. When the arrval rates change above some threshold, then the algorthm can be restarted to compute the new loads for each computer. node node 2 φ x r Processor node Communcaton Network β xr Fg. 2. Dstrbuted system model for mult-user obs. node n The followng example descrbes the CCOOP algorthm for a system of 3 nodes. Example 2.. In ths example, we apply CCOOP algorthm to a system of 3 nodes. Let the processng rates of the nodes be µ = 0, µ 2 = 20, and µ 3 = 40. Let the ob arrval rates to the nodes be φ = 8, φ 2 = 5, and φ 3 = 2. Let the mean communcaton tme be 0.00 s. Step ntalzes the loads on the nodes to β = 8, β 2 = 5, and β 3 = 2. After sortng the nodes n step 2 we have d 3 (2) d 2 (5) d (8) and d 3 (2)+g(0) = = 7. d (8) = 0.5. Snce, d 3 (2)+g(0) < d (8) the algorthm proceeds to step 3. In step 3, α s determned usng a bnary search. Intal value of α wll be and the fnal value of α after extng the whle loop s 0.04 and λ S = λ R = λ = 3 (ϵ s assumed to be 0 5 ). Step 4 determnes the fnal loads for the nodes as β = 0, β 2 = 0, and β 3 = 5. Thus, node 3 s a snk and node s and 2 are dle source nodes. 3. Non-cooperatve load balancng for mult-user obs 3.. System model We consder a dstrbuted system model as shown n Fg. 2. The system has n nodes (computers) connected by a communcaton network. The nodes and the communcaton network are modeled as M/M/ queung systems [2]. Jobs arrvng at each node may belong to m dfferent users. The termnology and notatons used are as follows: We denote () the external ob arrval rate of user to node (.e. the number of external obs of user arrvng at node per unt tme) by φ, () the total ob arrval rate of user by φ so, φ = n φ, () the total ob arrval rate of the system by Φ so, Φ = m φ =, (v) the maxmum processng rate of node (.e. the maxmum number of obs that can be processed at node per unt tme) by µ, (v) the ob processng rate (or load) of user allocated by the load balancng scheme for node (.e. the number of obs of user that are to be processed at node per unt tme) by β, (v) the vector of loads at node from user s,..., m by β = [β, β2,..., βm ] T, (v) the load vector of all nodes =,..., n (from all user s,..., m) by β = [β, β 2,..., β n ] T, (v) the vector of loads of user k allocated to nodes,..., n by β k = [β k, βk,..., 2 βk n ]T, (x) the ob flow rate of user from node r to node s (.e. the number of obs of user sent from to per unt tme) by x rs, (x) the ob traffc through the network of user by λ λ = n n r= s= x rs, λ = [λ, λ 2,..., λ m ] T, λ = m =, λ and (x) the mean communcaton tme for sendng or recevng a ob from one node to another for any user by t. A ob arrvng at node may be ether processed at node or transferred to a neghborng node for remote processng through

6 542 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) the communcaton network and s not transferred to any further nodes. The decson of transferrng a ob does not depend on the state of the system and hence s statc n nature. If a node sends (receves) obs to (from) node, node does not send (receve) obs to (from) node. For each user, nodes are classfed nto the followng as n [25]: Idle source (R d ): does not process any user obs (β = 0). Actve source (R a): processes some of the user obs that arrve and t sends the remanng user obs to other nodes. But, t does not receve any user obs from other nodes. Neutral (N ): processes user obs locally wthout sendng or recevng user obs. Snk (S ): only receves user obs from other nodes but t does not send out any user obs. Assumng that each node s modeled as an M/M/ queung system, the mean node delay for an user ob processed at node s gven by: F (β ) = µ m β k k=. (5) We assume that the expected communcaton delay of a ob from node r to node s s ndependent of the source-destnaton par (r, s) but may depend on the total traffc through the network denoted by λ where λ = m = λ. Modelng the communcaton network as an M/M/ queung system, the expected communcaton delay of an user ( =,..., m) ob s gven by: G t (λ) =, m t λ k k= m λ k < t. (6) k= Remark 3.. F (β ) and G (λ) are ncreasng postve functons. The network traffc of user can be expressed n terms of the varable β as: λ = n 2 φ β. Thus, the overall expected response tme of user s gven by: D (β) = β φ F (β ) + λ φ G (λ) = β φ µ m β k k= 3.2. Non-cooperatve game among the users λ t +. (7) m φ t λ k In ths secton, we formulate the load balancng problem as a non-cooperatve game among the users. We use the game theory termnology ntroduced n [8]. Each user ( =,..., m) must fnd the workload (β ) that s assgned to computer such that the expected response tme of her/hs own obs (D (β)) s mnmzed. The vector β = [β, β,..., 2 β n] s called the load balancng strategy of user ( =,..., m) and the vector β = [β, β 2,..., β m ] s called the strategy profle of the load balancng game. The strategy of user depends on the load balancng strateges of the other users. The assumptons for the exstence of a feasble strategy profle are as follows: k= () Postvty: β 0, =,..., n, =,..., m; () Conservaton: n β = φ, =,..., m; () Stablty: m = β < µ, =,..., n. A Non-cooperatve load balancng game conssts of a set of players, a set of strateges, and preferences over the set of strategy profles: () Players: The m users. () Strateges: Each user s set of feasble load balancng strateges. () Preferences: Each user s preferences are represented by her/hs expected response tme (D ). Each user prefers the strategy profle β to the strategy profle β f and only f D (β ) < D (β ). We need to solve the above game for our load balancng scheme. A soluton can be obtaned at the Nash equlbrum [4] whch s defned as follows. Defnton 3. (Nash Equlbrum). A Nash equlbrum of the load balancng game defned above s a strategy profle β such that for every user ( =,..., m): β arg mn D (β,..., β,..., β m ). (8) β At the Nash equlbrum, a user cannot further decrease her/hs expected response tme by choosng a dfferent load balancng strategy when the other users strateges are fxed. The equlbrum strategy profle can be found when each user s load balancng strategy s a best response to the other users strateges. The best response for user, s a soluton to the followng optmzaton problem (BR ): mn D (β) (9) β subect to the constrants: β 0, =,..., n (20) β = φ (2) m β < µ, =,..., n. (22) = Remark 3.2. In fndng the soluton to BR, the strateges of all the other users are kept fxed and so the varables n BR are the workloads of user,.e. β = (β, β 2,..., β n). In order to solve the optmzaton problem n Eq. (9), for each user, we defne the dfferental node delay (f ), the dfferental communcaton delay (g ), and the nverse of the dfferental node delay ((f ) ) as follows: f (β ) = β [β F (β µ )] = (µ (23) β )2 where µ = µ m k=,k βk. g (λ) = λ [λ G (λ)] = where g = (f ) (β β =x) = m t k=,k. λk tg (g tλ ) 2 (24) µ µ, x f x > µ 0, f x. µ (25)

7 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) The best response strategy of user, whch s the soluton of BR, s gven n the followng theorem. Theorem 3.. The soluton to the optmzaton problem BR satsfes the relatons f (β ) α + g (λ), β = 0 ( R d ), f (β ) = α + g (λ), 0 < β < φ ( R a ), α + g (λ) f (β ) α, β = φ ( N ), f (β ) = α, β > φ ( S ), subect to the total flow constrant, S (f ) (β β =α ) + N φ + R a where α s the Lagrange multpler. Proof. In Appendx B. (26) (f ) (β β =α +g (λ) ) = φ (27) The relatons n Theorem 3. can be nterpreted as follows: For each user : the dfferental node delays of all snks are the same; the dfferental node delays of all actve sources are equal; they consst of the dfferental node delay at a snk and the dfferental communcaton delay due to sendng a ob through the network to a snk. The dfferental node delay for neutrals s not less than the dfferental node delay of snks but not greater than the dfferental node delay of actve sources; and the dfferental node delay for dle sources s not less than the dfferental node delay of actve sources. Snce t s not possble to obtan a closed form soluton for α from Eq. (27), we use a bnary search to solve Eq. (27) teratvely for α smlar to [25]. Ths s descrbed n the BEST-RESPONSE algorthm below. Usng the α n an teraton of the bnary search, a set of snk (S (α )) and source nodes (R d (α ) and R a(α )) for a user are determned and t s checked whether the traffc from the set of source nodes (λ R (α )) equals the traffc nto the set of snk nodes (λ S (α )), n whch case an optmal α s found. The set of neutral nodes are denoted by N (α ). (S (α ), R d (α ), R a(α ), N (α ), λ S (α ), and λ R (α ) are defned n Defnton B. n Appendx B.) In the followng, we present an algorthm for determnng user s best response strategy. BEST-RESPONSE algorthm: Input: φ, β, λ, µ,..., µ n. Output: β.. Intalzaton: β φ ; N ; =,..., n. 2. Sort the computers such that f (β β ) =φ... fn(β n β ) + g (λ λ =0) fn(β n β ), ). If f (β β =φ n=φ n STOP. (No load balancng s requred) 3. Determne α (usng a bnary search): ) n=φ n a f (β β =φ b fn(β n β n=φ ) n whle() do λ S (α ) 0 λ R (α ) 0 α a+b 2 Calculate: S (α ), λ S (α ), R d (α ), R a(α ), and λ R (α ) (eqs. (03) (07)) n the order gven for =,..., n If ( λ S (α ) λ R (α ) < ϵ) EXIT If (λ S (α ) > λ R (α )) b α else a α 4. Determne user s loads on the computers: β 0, for R d (α ) β (f ) (β β =α +g (λ) ), for R a(α ) β (f ) (β β ), for S (α ) =α β φ, for N (α ) The followng remark proves the correctness of BEST-RESPONSE algorthm. Remark 3.3 (Correctness of BEST-RESPONSE Algorthm). In the above BEST-RESPONSE algorthm, the whle loop n step 3 computes an optmal α whch parttons the nodes nto Idle Sources, Actve Sources, Neutrals, and Snks. Once the partton s known, the loads for user on varous nodes are computed n step 4. These are n accordance wth Theorem 3.. Thus the load balancng strategy computed by the BEST-RESPONSE algorthm solves the optmzaton problem BR and ts soluton s the best response strategy of user. The followng remark descrbes the tme complexty of BEST- RESPONSE algorthm. Remark 3.4. The runnng tme of BEST-RESPONSE algorthm s O(n log n + n log /ϵ), where ϵ denotes the tolerance used for computng α n step 3 of the algorthm. The avalable processng rate at each computer ( ) as seen by a user ( ) (.e. µ n Eq. (23)) used n the above algorthm can be determned by statstcal estmaton of the run queue length of each node. In order to obtan the equlbrum allocaton, we need an teratve algorthm where each user updates her/hs strateges (by computng her/hs best response) perodcally by fxng the other users strateges. We can set a vrtual rng topology of the users to communcate and teratvely apply the BEST-RESPONSE algorthm to compute the Nash equlbrum. In the followng we present an teratve algorthm (NCOOPC) for computng the Nash equlbrum for our non-cooperatve load balancng game. One of the users can ntate the algorthm (ntatng user) who calculates her/hs ntal strateges by fxng the other users strateges to zero (or by requestng the other users for ther ntal strateges). An teraton s sad to be complete f ths ntatng user receves a message from her/hs left neghbor. She/he then checks f the error norm s less than a tolerance n whch case she/he sends a termnatng message to her/hs rght neghbor to be propagated around the rng. NCOOPC dstrbuted load balancng algorthm: Each user, =,..., m n the rng performs the followng steps n each teraton:. Receve the current strateges of all the other users from the left neghbor. 2. If the message s a termnaton message, then pass the termnaton message to the rght neghbor and EXIT. 3. Update the strateges (β ) by callng the BEST-RESPONSE. 4. Calculate D (Eq. (7)) and update the error norm. 5. Send the updated strateges and the error norm to the rght neghbor.

8 544 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) An mportant queston s f such best response-based algorthms converge to the Nash equlbrum. There exsts results about the convergence of such algorthms n the context of routng n parallel lnks [39,27]. For our load balancng game there exsts a unque Nash equlbrum because the obectve functons of the players are contnuous, convex, and ncreasng. Orda et al. [39,27] proved that f the obectve functons are contnuous, convex, and ncreasng there exsts a unque Nash equlbrum for the game. Our smulatons of NCOOPC algorthm wth dfferent settngs n Secton 5 confrm the theoretcal results. The followng remark descrbes the mplementaton of NCOOPC n practce. Remark 3.5. In practce, the NCOOPC algorthm could be mplemented by the schedulng agent (process) assocated wth each user. Users (or agents) wll use the strateges that are computed at the Nash equlbrum and the system remans n equlbrum. Ths equlbrum s mantaned untl a new executon of the algorthm s ntated. The schedulng agent of a user communcates wth the agents of other users and makes the allocaton decsons. The NCOOPC algorthm can be restarted perodcally by the schedulng agents when the system parameters (or system load) change above some threshold. When the ob arrval rate at a node changes, then the ob queue length at that node also changes. The schedulng agent can estmate the ob arrval rate of the user to a node by consderng the number of arrvals over a fxed nterval of tme. The followng example descrbes the NCOOPC and BEST- RESPONSE algorthms for a system of 3 nodes and 2 users. Example 3.. In ths example, we apply the NCOOPC dstrbuted load balancng algorthm to a system of 3 nodes and 2 users. Let the processng rates of the nodes be µ = 5, µ 2 = 0, and µ 3 = 5; the ob arrval rates of user to the nodes be φ = 3, φ = 2 2, and φ = 3 7; the ob arrval rates of user 2 to the nodes be φ2 =, φ 2 = 2 5, and φ2 3 = 4; and the mean communcaton tme be 0.00 s. In teraton of NCOOPC, user receves the ntal strateges of user 2 (.e. β 2 =, β 2 2 = 5, and β 2 3 = 4 (the ntal strateges of each user are ther own arrval rates)) and updates her/hs strateges by callng the BEST-RESPONSE (BR) algorthm as follows: BR step ntalzes user s loads on the computers to β = 3, β 2 = 2, and β 3 = 7. After sortng the nodes n BR step 2 we have f ([2, 5]) < 2 f ([7, 4]) < 3 f ([3, ]) and f ([2, 5]) + 2 g ([0, 0]) = < f ([3, ]) = 4.0. So, the BR algorthm proceeds to step 3. In BR step 3, α s determned usng a bnary search. Intal value of α s 2.27 and the fnal value of α after extng the whle loop s 0.89 and λ = S λ = R λ =. (ϵ s assumed to be 0 5 ). BR step 4 determnes the fnal loads for =.88. User checks user as β = 2.63, β = , and β 3 for the error norm and sends her/hs updated strateges (loads) to user 2. User 2 receves the current strateges of user and updates her/hs strateges by callng the BEST-RESPONSE (BR) algorthm as follows: BR step ntalzes user 2 s loads on the computers to β 2 =, β 2 2 = 5, and β 2 3 = 4. After sortng the nodes n BR step 2 we have f 2([7.48, 5]) < 2 f 2([.88, 4]) < 3 f 2 ([2.63, ]) and f 2([7.48, 5]) + 2 g 2 ([., 0]) = < f 2 ([2.63, ]) =.3. So, the BR algorthm proceeds to step 3. In BR step 3, α 2 s determned usng a bnary search. Intal value of α 2 s 0.96 and the fnal value of α 2 after extng the whle loop s 0.8 and λ 2 = S λ 2 = R λ2 = BR step 4 determnes the fnal loads for user 2 as β 2 = 4.47, β2 = 2.6, and β2 3 = User 2 updates the error norm and sends her/hs updated strateges (loads) and error norm to user. In teraton 2 of NCOOPC, user updates her/hs loads by callng the BR algorthm usng the loads of user 2 from teraton, checks for the error norm, and passes the updated loads to user 2. User 2 now updates her/hs loads usng the updated loads of user, updates the error norm, and passes the updated loads and error norm to user. Ths process contnues untl the desred error norm s reached (0 teratons n ths example). The fnal loads of user are β =.67, β = , and β 3 = 6.37, and the fnal loads of user 2 are β 2 =.30, β2 = , and β2 3 = Thus, for user, node s an actve source, node 2 s a snk, and node 3 s an actve source, and for user 2 node s a snk, node 2 s an actve source, and node 3 s a snk. 4. Performance evaluaton of CCOOP We perform smulatons to evaluate the performance of the CCOOP scheme. The system parameters that are used n the experments below are obtaned usng Sm++ [9] smulaton software package. The performance metrcs used n our smulatons are the expected response tme and the farness ndex. The farness ndex [2], I(D) = [ n ] 2 D n n D 2 (28) s used to quantfy the farness of load balancng schemes. Here the nput D s the vector D = (D, D 2,..., D n ) where D s the expected response tme of obs that are processed at computer. Ths ndex s a measure of the equalty of response tmes at dfferent computers. If all the computers have the same expected ob response tme, then I = and the system s 00% far to all obs and t s load-balanced. If the dfferences on D ncrease, I decreases and the load balancng scheme favors only some obs. We perform smulatons to study the mpact of system utlzaton, heterogenety, system sze, and communcaton tme on the performance of the proposed scheme. We descrbe the system confguraton and smulaton setup for each of the above factors n the subsectons correspondng to them. We also mplemented the followng statc load balancng schemes for comparson purposes. Overall Optmal Scheme (OPTIM) [26]: Ths scheme mnmzes the expected response tme over all the obs executed by the system. The loads (β ) at each computer are obtaned by solvng the followng non-lnear optmzaton problem: mn n β D (β ) + λg(λ), (29) Φ subect to the constrants (8) (0). The centralzed algorthm for obtanng the loads s gven n [26]. Ths scheme provdes a system optmal soluton but s unfar. Proportonal Scheme (PROP) [8]: Ths scheme allocates the obs to the computers n proporton to ther processng speeds as follows: β Φ µ. (30) n µ = The allocaton may not mnmze the overall expected response tme of the system and s unfar. In the followng we present and dscuss the smulaton results. 4.. Effect of system utlzaton System utlzaton (ρ) represents the amount of load on the system. It s defned as the rato of the total arrval rate to the

9 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Expected Response Tme (sec) CCOOP OPTIM PROP System Utlzaton(%) (a) Expected response tme. Farness Index I CCOOP OPTIM PROP (b) Farness ndex. System Utlzaton(%) Fg. 3. Effect of system utlzaton. Table System confguraton. Relatve processng rate Number of computers Processng rate (obs/s) aggregate processng rate of the system: ρ = Φ. (3) n µ We smulated a heterogeneous system consstng of 32 computers to study the effect of system utlzaton. The system has computers wth sx dfferent processng rates. The system confguraton s shown n Table. The relatve processng rate for a computer s defned as the rato of ts processng rate to the processng rate of the slowest computer n the system. For each experment the total ob arrval rate n the system Φ s determned by the system utlzaton ρ and the aggregate processng rate of the system. We chose fxed values for the system utlzaton and determned the total ob arrval rate Φ. For example, f we consder ρ = 0% and an aggregate processng rate of 3860 obs/s, then the total arrval rate n the system s Φ = 386 obs/s. The ob arrval rate for each computer φ, =,..., 32 s determned from the total arrval rate as φ = q Φ, where the fractons q are gven n Table 2. The mean communcaton tme t s assumed to be 0.00 s. Table 3 presents the ob arrval rates to each computer (φ ) and the ob processng rates (departures rates or loads) at each computer (β ) (usng the notaton φ /β ) for system utlzatons rangng from 0% to 90%. φ, =,..., 32 are calculated usng the fractons gven n Table 2 and Φ as descrbed n the above paragraph. β, =,..., 32 are obtaned based on the CCOOP algorthm. The queue length for M/M/ systems s nfnte [2] and n so we assumed that no obs are lost φ = n β. In Fg. 3(a), we present the expected response tme of the system for dfferent values of system utlzaton rangng from 0% to 90%. Ths corresponds to a total arrval rate rangng from 386 obs/s to 3474 obs/s (Table 3). It can be observed that CCOOP performs near the OPTIM for ρ rangng from 0% to 50% and s better than PROP for ρ rangng from 0% to 90%. For example, at 70% system utlzaton, the response tme of CCOOP s around 6% less than that of PROP and around 9% greater than OPTIM. The poor performance of PROP s due to the fact that the less powerful computers are sgnfcantly overloaded. CCOOP does not provde a system optmal soluton lke OPTIM but provdes a Pareto-optmal soluton. In Fg. 3(b), we present the farness ndex for dfferent values of system utlzaton. The CCOOP scheme has a farness ndex of almost for any system utlzaton. The farness ndex of OPTIM drops from 0.98 at low load to 0.69 at hgh load and PROP mantans a farness ndex of 0.35 over the whole range of system loads. The farness acheved by CCOOP comes at the cost of ncreasng the response tme of the system. Ths ncreased response tme s stll close to that of OPTIM as seen n Fg. 3(a) except for very hgh system loads. Fg. 4 shows the expected response tme at each computer for all the schemes at hgh system load (ρ = 90%). CCOOP guarantees almost equal expected response tmes for all the computers. Ths means that the obs wll have the same expected response tme Table 2 Job arrval fractons q for each computer. Computer q Table 3 Job arrval rates/ob processng rates (loads) for each computer (C) for varous system utlzatons. System utlzaton (%) C C6 C7 C2 C3 C4 C8 C9 C22 C23 C26 C27 C /0 3.86/0 7.72/0 9.65/0 5.44/0 9.3/ / /0 7.72/0 5.44/0 9.3/ / / / /0.58/0 23.6/ / / / / /0 5.44/ / / / / / /0 9.3/0 38.6/ / / / / /0 23.6/0 46.3/ / / / / / / / / / / / / / / / / / / / / / / / / /237.5

10 546 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Expected Response Tme (sec) CCOOP OPTIM PROP 0 C- C6 C7 - C2 C3 - C8 C9 - C22 C23 - C26 C27 - C32 Computers Fg. 4. Expected response tme at each computer (ρ = 90%). Table 4 System parameters. Speed skewness µ of C C µ of C9 C µ of C25 C Φ (obs/s) Table 5 Job arrval rates/ob processng rates (loads) for each computer (C) for varous skewness levels. Speed skewness C C8 C9 C24 C25 C / / / / /3.4 0/ / /.28 8/ / /0 6/ / /0 4/0 ndependent of the allocated computers. In the case of OPTIM the expected response tmes are less balanced than CCOOP but the overall expected response tme s lower than CCOOP as can be seen from Fg. 3(a). PROP overloads the slowest computers and the overall expected response tme s ncreased. The dfference n the expected response tme at Computer s through 6 (slowest) and Computer s 27 through 32 (fastest) s sgnfcant. In the case of OPTIM and PROP, obs are treated unfarly n the sense that a ob allocated to the fast computer wll have a low expected response tme and a ob allocated to a slow computer wll have a hgh expected response tme. CCOOP provdes a far and load balanced allocaton whch s desrable n many current dstrbuted systems Effect of heterogenety In ths secton, we study the effect of heterogenety on the performance of load balancng schemes. A smple way to characterze system heterogenety s to use the processor speed. One of the common measures of heterogenety s the speed skewness [50] whch s defned as the rato of maxmum processng rate to mnmum processng rate of the computers n the system. We study the effectveness of load balancng schemes by varyng the speed skewness. We smulated a heterogeneous system of 32 computers (8 fast, 6 medum-fast, and 8 slow) to study the effect of heterogenety. The slow computers have a relatve processng rate of and the relatve processng rate of the fast and medum-fast computers s vared from (homogeneous system) to 00 and 0 respectvely Table 6 Total arrval rates. No. of computers Φ (obs/s) (hghly heterogeneous system). The system utlzaton was kept constant (ρ = 60%) and the mean communcaton tme t s assumed to be 0.00 s. In Table 4, we present the processng rates (µ obs/s) of the computers n the dfferent heterogeneous systems and the total arrval rates (Φ) of the systems. C through C8 represent the fast computers, C9 through C24 represent the medum-fast computers, and C25 through C32 represent the slow computers. The total arrval rates (Φ) are calculated usng Eq. (3). Table 5 presents the ob arrval rates to each computer (φ ) and the ob processng rates (departures rates or loads) at each computer (β ) (usng the notaton φ /β ) for varous skewness levels. φ, =,..., 32 are calculated as φ = q Φ, where q are fractons smlar to that n Table 2. β, =,..., 32 are obtaned based on the CCOOP algorthm. Fg. 5(a) shows the effect of speed skewness on the expected response tme. It can be observed that as the skewness ncreases, the performance of CCOOP approaches to that of OPTIM whch means that n hghly heterogeneous systems CCOOP s very effectve. PROP performs poorly because t overloads the slow computers. The performance of CCOOP s ncreased wth an ncrease n speed skewness because CCOOP dles some of the slowest computers when the power of the fastest computers n the system s ncreased. Fg. 5(b) shows the effect of speed skewness on the farness ndex. It can be observed that CCOOP has a farness ndex of almost over all range of speed skewness. Ths shows that CCOOP guarantees almost equal expected response tmes for all the computers. The farness ndex of OPTIM and PROP drops below at low skewness to 0.83 and 0.35 respectvely at hgh skewness Effect of system sze An mportant ssue s to study the nfluence of system sze on the performance of load balancng schemes. To study ths ssue we smulated a heterogeneous dstrbuted system consstng of four types of computers: slow computers (relatve processng rate =, 2 and 5) and fast computers (relatve processng rate = 0). For a system sze of 2, we used two fast computers. To ncrease the system sze, we kept constant the number of fast computers and ncreased the number of slow computers. In Table 6, we present the total arrval rates for some of the experments. The system

11 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Expected Response Tme (sec) (a) Expected response tme. CCOOP OPTIM PROP Speed Skewness Farness Index I (b) Farness ndex. Speed Skewness CCOOP OPTIM PROP Fg. 5. Effect of heterogenety Expected Response Tme (sec) CCOOP 0.03 OPTIM PROP (a) Expected response tme. Number of Computers Farness Index I CCOOP OPTIM PROP (b) Farness ndex. Number of Computers Fg. 6. Effect of system sze. Table 7 Job arrval rates/ob processng rates (loads) for computers for varous system szes. System sze No. of computers φ /β 2 80/59., 40/ /72.4, 40/23., 5/0, 9/ /78.7, 2 40/29.2, 2 5/0, 2 9/ /82.9, 4 40/33.2, 4 5/3.2, 4 9/ /84.6, 6 40/34.8, 6 5/4.8, 6 9/ /85.6, 8 30/35.6, 8 5/5.9, 8 9/ /86.2, 0 30/36.2, 0 5/6.5, 0 9/0 utlzaton was kept constant (ρ = 60%) and the mean communcaton tme t s assumed to be 0.00 s. The total arrval rates (Φ) are calculated usng Eq. (3). Table 7 presents φ and β (usng the notaton φ /β ) (the computers are ordered n decreasng order of ther speeds (actual processng rates)) for varous system szes presented n Table 6. φ, =,..., 32 are calculated as φ = q Φ, where q are fractons smlar to that n Table 2 and β, =,..., 32 are obtaned based on the CCOOP algorthm. Fg. 6(a) shows the expected response tme where the number of computers ncreases from 2 to 32. It can be observed that the performance of CCOOP les n between OPTIM and PROP. The sublnear ncrease n the expected response tme of CCOOP also shows that t s scalable. Fg. 6(b) shows the farness ndex for all the schemes as the system sze ncreases. It can be observed that the CCOOP mantans a farness ndex approxmately equal to wth an ncrease n the system sze. Ths means that the expected response tmes for all the computers s almost equal n the case of CCOOP. Thus, CCOOP provdes an allocaton whch s far to all the obs ndependent of the allocated computers Effect of communcaton tme From the above, t can be observed that CCOOP scheme s not only far, but also performs near the OPTIM for a low value of mean communcaton tme t (low value of t represents a faster communcatons network). The effect of ncreasng t on the expected response tme at medum system utlzaton (ρ = 60%) for the confguraton gven n Table s shown n Fg. 7(a). From Fg. 7(a), t can be observed that wthout load balancng (each computer processes ts own local stream of obs) the expected response tme s 0.09 s. As t ncreases, the expected response tme of CCOOP ncreases and reaches 0.09 s at t = 0.2 s, whch s the lmtng case of load balancng when the communcaton tme s hgh (t 0.2). Ths s because, as t ncreases, the load transfer from slow computers to fast computers decreases and the CCOOP scheme wll not be able to load balance to ts optmum. Smlarly, the Farness Index of CCOOP decreases as the communcaton tme ncreases. We also mplemented the COOP scheme [9] for comparson. COOP s a cooperatve game based load balancng scheme. However, t does not take the communcaton costs nto account n fndng the optmal soluton that provdes farness. Fg. 7(b) shows the effect of communcaton tme on the performance of COOP and CCOOP at 50% and 70% system utlzatons for the confguraton gven n Table. It can be observed that as t ncreases (from 0.00 to 0.0 s), CCOOP outperforms COOP. As t ncreases, the performance degradaton of COOP s hgher compared to that of CCOOP. For example, as

12 548 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Expected Response Tme (sec) CCOOP OPTIM NO LOAD BALANCING (a) CCOOP. Communcaton Tme (sec) Expected Response Tme (sec) COOP, 50% system utlzaton COOP, 70% system utlzaton CCOOP, 50% system utlzaton CCOOP, 70% system utlzaton Communcaton Tme (sec) (b) COOP vs. CCOOP. Fg. 7. Effect of communcaton tme. t ncreases from 0.00 to 0.0 s, the performance degradaton of COOP s around 20% whereas the performance of CCOOP degrades by only around 3%. Although the fgure shows plots for only a few values, the performance degradaton of COOP compared to CCOOP s hgher for any value of t > 0. Ths s because CCOOP takes the communcaton delays nto account whereas COOP does not. Thus, CCOOP can show a sgnfcant performance mprovement over COOP. 5. Performance evaluaton of NCOOPC We perform smulatons to study the mpact of system utlzaton, heterogenety, system sze, and communcaton tme on the performance of NCOOPC scheme. The system parameters that are used n the experments below are obtaned usng Sm++ [9] smulaton software package. The performance metrcs used are the expected response tme and the farness ndex [2]. The farness ndex (defned from the users perspectve) s defned as, 2 m C I(C) = =. (32) m m C 2 = The nput C s the vector C = (C, C 2,..., C m ) where C s the expected response tme of user s obs. If all the users have the same total expected response tme, then I = and the system s 00% far to all users and t s load-balanced. If I decreases, then the load balancng scheme favors only some users. We also mplemented the followng load balancng schemes for comparson purposes: Global Optmal Scheme (GOS) [25]: Ths scheme mnmzes the expected response tme over all the obs executed by the system to provde a system optmal soluton. The loads (β ) for each user are obtaned by solvng the followng non-lnear optmzaton problem: mn D(β) = Φ m n = β F (β ) + λ G (λ) (33) subect to the constrants (20) (22). Proportonal Scheme (PROP_M) [8]: Accordng to ths scheme each user allocates her/hs obs to computers n proporton to ther processng rate as follows: β φ µ. (34) n µ k k= Table 8 Job arrval fractons (q ) for each user. User q Ths allocaton may not mnmze the users expected response tmes or the overall expected response tme. The farness ndex for ths scheme s always as can be easly checked from Eq. (32). In the followng we present and dscuss the smulaton results. 5.. Effect of system utlzaton (ρ) We smulated a heterogeneous system consstng of 32 computers to study the effect of system utlzaton. The system has computers wth sx dfferent processng rates (Table ) and s shared by 20 users. For each experment the total ob arrval rate n the system Φ s determned by the system utlzaton ρ and the aggregate processng rate of the system. The total ob arrval rate Φ s chosen by fxng the system utlzaton. The ob arrval rate for each user φ, =,..., 20 s determned from the total arrval rate as φ = q Φ, where the fractons q are gven n Table 8. The ob arrval rates of each user, =,..., 20 to each computer, =,..., 32,.e. the φ s are obtaned smlar to Table 3. t s assumed to be 0.00 s. In Fg. 8(a), we present the expected response tme of the system for dfferent values of system utlzaton rangng from 0% to 90%. It can be observed that the performance of NCOOPC and GOS are smlar for ρ rangng from 0% to 50%. NCOOPC performs sgnfcantly better than PROP_M for ρ rangng from 60% to 90%. For example, at 70% system utlzaton, the response tme of NCOOPC s around 20% less than that of PROP_M and around 8% greater than that of GOS. The poor performance of PROP_M s due to the fact that t sgnfcantly overloads the less powerful computers where as NCOOPC does not. The decentralzaton and stablty of allocaton under non-cooperatve behavor are the man advantages of NCOOPC scheme. Fg. 8(b) shows the farness ndex for dfferent values of system utlzaton. The PROP_M scheme mantans a farness ndex of over the whole range of system loads. It can be shown that the PROP_M has a farness ndex of whch s a constant ndependent of the system load. The farness ndex of GOS vares from at low load to 0.94 at hgh load. The farness ndex of NCOOPC s approxmately equal to and each user obtans the mnmum possble expected response tme for her/hs own obs (.e. t s useroptmal). Fg. 9 shows the expected response tme for each user consderng all the schemes at medum system load (ρ = 60%).

13 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Expected Response Tme (sec) NCOOPC GOS PROP_M (a) Expected response tme. System Utlzaton(%) 90 Farness Index I NCOOPC GOS PROP_M (b) Farness ndex. System Utlzaton(%) Fg. 8. Effect of system utlzaton. Expected Response Tme (sec) NCOOPC GOS PROP_M Users Fg. 9. Expected Response Tme for each User (ρ = 60%) Expected Response Tme (sec) NCOOPC 0.05 GOS PROP_M Speed Skewness (a) Expected response tme. Farness Index I (b) Farness ndex. NCOOPC GOS PROP_M Speed Skewness Fg. 0. Effect of heterogenety. The PROP_M scheme guarantees equal expected response tmes for all the users. The expected response tmes of the users n case of NCOOPC are almost the same. However, the dsadvantage of PROP_M s that the users have a hgher expected response tme for ther obs where as NCOOPC provdes the mnmum possble expected executon tme for each user accordng to the propertes of the Nash equlbrum. It can be observed that n the case of GOS, there are large dfferences n the users expected response tmes. Thus, the performance of NCOOPC s not only close to that of GOS but also makes an allocaton that provdes farness to the users Effect of heterogenety To study the effect of heterogenety on the performance of load balancng schemes, we smulated heterogeneous systems of 32 computers wth confguratons gven n Table 4. The systems have obs from 20 users. The system utlzaton was kept constant (ρ = 60%) and the mean communcaton tme t s assumed to be 0.00 s. The total arrval rates (Φ) are calculated usng Eq. (3) and the arrval rates of each user (φ s) are calculated usng the fractons gven n Table 8. The ob arrval rates of each user to each computer (φ s) are obtaned smlar to Table 5. Fg. 0(a) shows the effect of speed skewness on the expected response tme of all the schemes. It can be observed that as the skewness ncreases, the performance of NCOOPC approaches to that of GOS whch means that n hghly heterogeneous systems NCOOPC s very effectve. PROP_M performs poorly because t overloads the slowest computers. From Fg. 0(b) t can be observed that NCOOPC mantans a farness ndex of almost wth ncreasng speed skewness. The farness ndex of GOS drops

14 550 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Expected Response Tme (sec) NCOOPC 0.03 GOS PROP_M (a) Expected response tme. Number of Computers Farness Index I NCOOPC GOS PROP_M (b) Farness ndex. Number of Computers Fg.. Effect of system sze. Expected Response Tme (sec) NCOOPC GOS NO LOAD BALANCING (a) NCOOPC. Communcaton Tme (sec) Expected Response Tme (sec) NASH, 50% system utlzaton NASH, 70% system utlzaton NCOOPC, 50% system utlzaton NCOOPC, 70% system utlzaton (b) NASH vs. NCOOPC. Communcaton Tme (sec) Fg. 2. Effect of communcaton tme. from at low skewness to 0.8 at hgh skewness. Ths means that the GOS produces an allocaton whch does not guarantee equal expected response tme for all the users n the system. PROP_M has a farness ndex of for any speed skewness. NCOOPC and PROP_M guarantees almost equal expected response tmes for all the users. However, the expected response tmes of the users n the case of NCOOPC are consderably less than that of PROP_M. The dstrbuted nature, user-optmalty, and near-optmal performance are the advantages of NCOOPC whch are very mportant n dstrbuted computer systems Effect of system sze In ths secton, we study the effect of system sze on the performance of load balancng schemes. We ncreased the sze of the heterogeneous system from 2 to 32 wth confguratons smlar to those dscussed n Secton 4.3. The system utlzaton was kept constant (ρ = 60%) and t s assumed to be 0.00 s. The total arrval rates (Φ) are calculated usng Eq. (3) and the arrval rates of each user (φ s) are calculated usng the fractons gven n Table 8. The ob arrval rates of each user to each computer (φ s) are obtaned smlar to Table 7. Fg. (a) shows the expected response tme where the number of computers ncreases from 2 to 32. It can be observed that the performance of NCOOPC les n between GOS and PROP_M and the sub-lnear ncrease n the expected response tme of NCOOPC also shows that t s scalable. From Fg. (b) t can be observed that the farness ndex of PROP_M s and the farness ndex of NCOOPC s very close to wth an ncrease n the system sze. The farness ndex of GOS drops consderably wth an ncrease n the system sze. The expected response tmes of all the users are the same n the case of PROP_M and are almost equal n the case of NCOOPC. However, the expected response tmes of the users are consderably less n the case of NCOOPC compared to PROP_M. Thus, NCOOPC provdes an allocaton whch s far to all the users ndependent of the allocated computers and also performs near the optmal scheme Effect of communcaton tme Fg. 2(a) shows the effect of ncreasng communcaton tme on the performance of NCOOPC and GOS. It can be observed that, as t ncreases, the expected response tme of NCOOPC and GOS ncreases and reaches 0.09 s at around t = 0.7 s, whch s the lmtng case of load balancng when the communcaton tme s hgh. Ths s because, as t ncreases, the load transfer from slow computers to fast computers decreases and NCOOPC wll not be able to load balance to ts optmum. Smlarly, the Farness Index of NCOOPC decreases as the communcaton tme ncreases. We also mplemented NASH scheme [8] for comparson. NASH s a non-cooperatve game based load balancng scheme that provdes farness to the users. However, t does not take the communcaton costs nto account n fndng the optmal soluton. Fg. 2(b) shows the effect of communcaton tme on the performance of NASH and NCOOPC at 50% and 70% system utlzatons for the confguraton gven n Table. It can be observed that, as t ncreases, the performance degradaton of NASH s hgher compared to that of NCOOPC. For example, as t ncreases from 0.00 to 0.0 s, the performance degradaton of NASH s around 25% whereas the performance of NCOOPC degrades by only around 5%. Ths s because NCOOPC takes the communcaton delays nto account whereas NASH does not. Thus, NCOOPC can show a sgnfcant performance mprovement over NASH.

15 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Conclusons and future work In ths paper, we proposed two far load balancng schemes for dstrbuted systems by takng the communcaton costs nto account. Usng cooperatve game theory we proposed the CCOOP algorthm that provdes farness to all the obs n a sngle-class ob dstrbuted system. Usng non-cooperatve game theory we proposed the NCOOPC dstrbuted algorthm that provdes farness to all users n a mult-user ob dstrbuted system. The dervaton of CCOOP and NCOOPC s valdated theoretcally usng Game Theory results and ther performance s evaluated usng smulatons. The smulatons were performed on a varety of system confguratons that allowed us to compare the schemes n a far manner. The expermental results showed that CCOOP and NCOOPC not only perform near the system optmal schemes OPTIM and GOS respectvely but also provde farness to the users and ther obs. CCOOP and NCOOPC wll be sutable for systems n whch the far treatment of the users or ther obs s as mportant as other performance characterstcs. In future work, we plan to mplement the proposed schemes on real dstrbuted systems consstng of heterogeneous computers n order to valdate our results and develop mechansms that take nto account the selfsh behavor of the enttes n the system. We also plan to develop dynamc load balancng schemes based on dynamc game theory that provde farness by takng the current system load nto account and also consder other aspects of heterogenety. Acknowledgments We wsh to express our sncere thanks to the revewers for ther helpful and constructve suggestons whch consderably mproved the qualty of the manuscrpt. Appendx A For the cooperatve load balancng game defned n Defnton 2., we are nterested n fndng the NBS whch provdes a Pareto optmal soluton. We use the exstng theory on cooperatve games [34,36,47,52,4] (and references there-n) n the proofs of Theorems 2. and 2.2. Defnton A. (Pareto Optmalty). Let U R N be the set of achevable performances. The pont u U s sad to be Pareto optmal f for each v U, v u, then v = u. Defnton A.2 (The Nash Barganng Soluton (NBS)). A mappng S : G R N (where G denotes the set of achevable performances wth respect to the ntal agreement pont) s sad to be a NBS f: (a) S(U, u 0 ) U 0 ; (b) S(U, u 0 ) s Pareto optmal and satsfes the farness axoms. Defnton A.3 (Barganng Pont). u s a (Nash) barganng pont f t s gven by S(U, u 0 ) and f (u ) s called the set of (Nash) barganng solutons. Theorem A. (Nash Barganng Pont Characterzaton [47,52]). Consder the assumptons from Defntons A. A.3 above. Let J denote the set of players who are able to acheve a performance strctly superor to ther ntal performance and let X 0 denote the set of strateges that enable the players to acheve at least ther ntal performances. Let the vector functon {f }, J be one-to-one on X 0. Then, there exsts a unque barganng pont u and the set of the barganng solutons f (u ) s determned by solvng the followng optmzaton problems: (P J ) : mn x (P J ) : mn x (f (x) u 0 ) x X 0 (35) J ln(f (x) u 0 ) x X 0. (36) J Then, (P J ) and (P J ) are equvalent. (P J ) s a convex optmzaton problem and has a unque soluton. The unque soluton of (P J ) s the barganng soluton. Proof of Theorem 2.. The obectve functon for each player f (X) (Defnton 2.) s convex and bounded below. The set of constrants s convex and compact. Thus, the condtons n Theorem A. are satsfed and the result follows. Proof of Theorem 2.2. The obectve vector functon {f },,..., n+ (Defnton 2.) of the players s a one-to-one functon of X. Thus, applyng Theorem A. the result follows. Proof of Theorem 2.3. Let u and v denote the network traffc nto node and the network traffc out of node respectvely. From the balance of the total traffc n the network, we have u = v. (37) The load β on node can then be wrtten as β = φ + u v (38) and the network traffc λ can be wrtten as λ = v = u. (39) Hence, the problem becomes: n n mn E(u, v) = ln D (φ + u v ) + ln G v subect to (40) β = φ + u v 0, =,..., n (4) u + v = 0 (42) β = φ + u v < µ, =,..., n (43) u 0, =,..., n (44) v 0, =,..., n. (45) The obectve functon n Eq. (40) s convex and the constrants are all lnear and defne a convex polyhedron. Ths mply that the frst-order Kuhn Tucker condtons are necessary and suffcent for optmalty [42]. Let α, δ 0, η 0, ψ 0 denote the Lagrange multplers [42]. The Lagrangan s L(u, v, α, δ, η, ψ) = E(u, v) + α u + v + δ (φ + u v ) + η u + ψ v. (46) We gnore the constrant n Eq. (43) snce all the assocated multplers wll be zero f we ntroduce t n the Lagrangan. The optmal soluton satsfes the followng Kuhn Tucker condtons: L = d (φ + u v ) α + δ + η = 0, =,..., n (47) u L n = d (φ + u v ) + g v v + α δ + ψ = 0, =,..., n (48)

16 552 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) L α = u + v = 0 (49) φ + u v 0, δ (φ + u v ) = 0, δ 0, =,..., n (50) u 0, η u = 0, η 0, =,..., n (5) v 0, ψ v = 0, ψ 0, =,..., n. (52) In the followng, we fnd an equvalent form of Eqs. (47) (52) n terms of β. Addng Eqs. (47) and (48) we have, g( v ) = η + ψ, =,..., n. Snce g > 0, ether η < 0 or ψ < 0 (or both). Hence, from Eqs. (5) and (52), for each, ether u = 0 or v = 0 (or both). We consder each case separately. Case I: u = 0, v = 0: Then, we have β = φ. It follows from Eq. (50) that δ = 0. Substtutng ths nto Eqs. (47) and (48) gves d (β ) = α η α (53) d (β ) = α + g(λ) + ψ α + g(λ). (54) From the above, we have α d (β ) α + g(λ). (55) Ths case corresponds to neutral nodes. Case II: u = 0, v > 0: Then, from Eq. (52), we have ψ = 0. We consder the followng subcases. Case II. v < φ : Then, we have 0 < β < φ. It follows from Eq. (50) that δ = 0. Substtutng ths nto Eqs. (47) and (48) gves d (β ) = α η α (56) d (β ) = g(λ) + α. (57) Ths case corresponds to actve sources. Case II.2 v = φ : Then, we have β = 0 and Eqs. (47) and (48) gves d (β ) = α δ η α (58) d (β ) = α + g(λ) δ α + g(λ). (59) Thus, we have d (β ) α + g(λ). (60) Ths case corresponds to dle sources. Case III: u > 0, v = 0: Then, we have β > φ. It follows from Eqs. (50) and (5) that δ = 0 and η = 0. Substtutng ths nto Eq. (47), we have d (β ) = α. (6) Ths case corresponds to snks. Eq. (49) may be wrtten n terms of β as β = Φ. (62) Usng Eqs. (57) and (6), the above equaton becomes d (α) + φ + d (α + g(λ)) = Φ (63) S N R a whch s the total flow constrant. Defnton A.4. From Theorem 2.3, the followng propertes whch are true n the optmal soluton can be derved and ther proofs are smlar to those n [26]. The condtons n Property A. help to partton the nodes nto one of the four categores. Once the node partton s known, the optmal loads for each node can be calculated based on Property A.2. Property A.3 states that the ob flow out of the sources equals the ob flow nto the snks. Property A.. d (0) α + g(λ), ff β = 0 (64) d (φ ) > α + g(λ) > d (0), ff 0 < β < φ (65) α d (φ ) α + g(λ), ff β = φ (66) α > d (φ ), ff β > φ. (67) Remark A.. Property A. states that at an optmal soluton (α): the dfferental node delay of a snk node (α) s greater than ts ntal (pror to load balancng) dfferental node delay (Eq. (67)); the dfferental node delay of an actve source node les between ts ntal dfferental node delay and the dfferental node delay when the node has no load (ths delay conssts of the dfferental node delay at a snk and the dfferental communcaton delay due to sendng a ob through the network to a snk) (Eq. (65)); the dfferental node delay of a neutral node s the same as ts ntal dfferental node delay and les between that of a snk and an actve source (Eq. (66)); and the dfferental node delay of an dle source node s not less than the dfferental node delay of an actve source (Eq. (64)). Property A.2. If β s an optmal soluton to the problem n Theorem 2.2, then we have: β = 0, R d (68) β = d (α + g(λ)), R a (69) β = φ, N (70) β = d (α), S. (7) Remark A.2. Property A.2 states that the optmal load of an dle source node s 0 (Eq. (68)); the optmal load of a neutral node s the same as ts ntal load (ob arrval rate) (Eq. (70)); and the optmal loads of an actve source node and a snk node are gven by ther nverse (functons) dfferental node delays (Eqs. (69) and (7)). Property A.3. If β s an optmal soluton to the problem n Theorem 2.2, then we have λ = λ S = λ R, where λ S = S [d (α) φ ] and λ R = R d φ + R a [φ d (α + g(λ S ))]. Remark A.3. Property A.3 states that, at an optmal soluton, the total ob traffc through the network s the same as the traffc from all the source nodes whch s the same as the traffc to all the snk nodes. Based on the above propertes, we have the followng defntons for an arbtrary α ( 0): S(α) = { d (φ ) < α} (72) λ S (α) = [d (α) φ ] (73) S(α) R d (α) = { d (0) α + g(λ S (α))} (74) R a (α) = { d (φ ) > α + g(λ S (α)) > d (0)} (75) λ R (α) = φ + [φ d (α + g(λ S (α)))] (76) R d (α) R a (α) N(α) = { α d (φ ) α + g(λ S (α))}. (77) Eq. (72) denotes the set of snk nodes, Eq. (73) denotes the ob traffc nto the snks, Eq. (74) denotes the set of dle source nodes, Eq. (75) denotes the set of actve source nodes, Eq. (76) denotes the ob traffc from the sources, and Eq. (77) denotes the set of neutral nodes. Thus, f an optmal α s gven, the node parttons n the optmal soluton are characterzed as R d = R d (α), R a = R a (α), N = N(α), S = S(α) and λ = λ S = λ R = λ S (α) = λ R (α).

17 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Appendx B Proof of Theorem 3.. We restate the problem ntroducng the varables u and v whch denote the user s network traffc nto node and network traffc out of node respectvely. From the balance of the total traffc of user n the network, we have λ = u = v. (78) The load β of user on node can then be wrtten as β = φ + u v, =,..., n. (79) Usng the above equatons, the problem n Eq. (9) becomes mn D (u, v) u,v = (φ + u v) φ (µ (φ + u v)) + subect to the followng constrants: u + t n v φ g t n v (80) v = 0 (8) u v + φ 0, =,..., n (82) u 0, =,..., n (83) v 0, =,..., n. (84) We gnore the stablty constrant (Eq. (22)) because at Nash equlbrum t s always satsfed and the total arrval rate (Φ) does not exceed the total processng rate of the system. The obectve functon n Eq. (80) s convex and the constrants are all lnear. Ths mples that the frst-order Kuhn Tucker condtons are necessary and suffcent for optmalty [42]. The total arrval rate of user (φ ) s constant. Let α, δ 0, ψ 0, η 0, denote the Lagrange multplers [42]. The Lagrangan s L(u, v, α, δ, ψ, η) = φ D (u, v) + n α v + δ (u v + φ) + ψ u + u η v. (85) The optmal soluton satsfes the followng Kuhn Tucker condtons: L u L v = f (φ + u v ) α + δ + ψ = 0, =,..., n. (86) = f (φ n + u v ) + g v l + α δ u + l= + η = 0, =,..., n. (87) v = 0 (88) φ + u v 0, δ (φ + u v ) = 0, δ 0, =,..., n. (89) u 0, ψ u = 0, ψ 0, =,..., n. (90) v 0, η v = 0, η 0, =,..., n. (9) In the followng, we fnd an equvalent form of Eqs. (86) (9) n terms of β. We consder two cases: Case I: u v +φ = 0: From Eq. (79), we have β = 0 and snce φ > 0, t follows that v > 0. Eq. (9) mples η = 0. Then from Eqs. (87) and (89), we get f (β ) = α + g (λ) δ α + g (λ). Ths case corresponds to dle sources. Case II: u v + φ > 0: From Eq. (89), we have δ = 0. Case II.: v > 0: Then, 0 < β < φ and from Eq. (9) we have η = 0. Eq. (87) mples, f (β ) = α + g (λ). (92) Ths case corresponds to actve sources. Case II.2: v = 0: Case II.2.: u = 0: Then, β = φ. From Eqs. (86) and (90), we have f (β ) = α ψ α. From Eqs. (87) and (9), we have f (β ) = α + g (λ) + η α + g (λ). Ths case corresponds to neutral nodes. Case II.2.2: u > 0: Then, β > φ. From Eq. (90), we have ψ = 0. Substtutng ths n Eq. (86), we have f (β ) = α. (93) Ths case corresponds to snk nodes. Eq. (88) may be wrtten n terms of β as n β = φ and usng the Eqs. (92) and (93), ths can be wrtten as (f ) (β β ) + φ + S =α (f ) (β β N R =α +g (λ) ) = φ (94) a whch s the total flow constrant for user. Defnton B.. From Theorem 3., the followng propertes whch are true n the optmal soluton can be derved and ther proofs are smlar to those n [25]. The condtons n Property B. help to partton the nodes nto one of the four categores for user. Once the node partton for user s known, her/hs optmal loads can be calculated based on Property B.2. Property B.3 states that the ob flow out of the sources equals the ob flow nto the snks for each user. Property B.. f (β β =0) α + g (λ), ff β = 0 (95) f (β β ) > α + g (λ) > f (β =φ β =0), ff 0 < β < φ (96) α f (β β ) α + g (λ), ff β =φ = φ (97) α > f (β β ), ff β =φ > φ. (98) Remark B.. Property B. states that at an optmal soluton (α ), for each user: the dfferental node delay of a snk node (α) s greater than ts ntal (pror to load balancng) dfferental node delay (Eq. (98)); the dfferental node delay of an actve source node les between ts ntal dfferental node delay and the dfferental node delay when the node has no load (ths delay conssts of the dfferental node delay at a snk and the dfferental communcaton delay due to sendng a ob through the network to a snk) (Eq. (96)); the dfferental node delay of a neutral node s the same as ts ntal dfferental node delay and les between that of a snk and an actve source (Eq. (97)); and the dfferental node delay of an dle source node s not less than the dfferental node delay of an actve source (Eq. (95)).

18 554 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) Property B.2. If β s an optmal soluton to the problem n Eq. (9), then we have: β = 0, R d (99) β = (f ) (β β =α +g (λ) ), R a (00) β = φ, N (0) β = (f ) (β β =α ), S. (02) Remark B.2. Property B.2 states that for each user, the optmal load of an dle source node s 0 (Eq. (99)); the optmal load of a neutral node s the same as ts ntal load (ob arrval rate) (Eq. (0)); and the optmal loads of an actve source node and a snk node are gven by ther nverse (functons) dfferental node delays (Eqs. (00) and (02)). Property B.3. If β s an optmal soluton to the problem n Eq. (9), then we have λ = λ = S λ R, where λ = S S [(f ) (β β ) φ ] =α and λ = R R φ + R [φ (f ) (β d a β =α +g (λ S ) )]. Remark B.3. Property B.3 states that, at an optmal soluton, the total ob traffc through the network of user s the same as the traffc from all the source nodes of user whch s the same as the traffc to all the snk nodes of user. Based on the above propertes, we have the followng defntons for an arbtrary α ( 0): S (α ) = { f (β β ) < α } (03) =φ λ S (α ) = [(f ) (β β ) φ ] =α (04) S (α ) R d (α ) = { f (β β =0) α + g (λ λ =λ )} (05) S (α ) R a (α ) = { f (β β ) > α + g (λ =φ λ =λ S (α ) ) > f (β β =0)} (06) λ R (α ) = φ + [φ (f ) R d (α ) R a(α ) (β β =α +g (λ λ ))] (07) =λ S (α ) N (α ) = { α f (β β ) α + g (λ =φ λ =λ )}. (08) S (α ) Eq. (03) denotes the set of snk nodes for user, Eq. (04) denotes the ob traffc nto the snks for user, Eq. (05) denotes the set of dle source nodes for user, Eq. (06) denotes the set of actve source nodes for user, Eq. (07) denotes the ob traffc from the sources for user, and Eq. (08) denotes the set of neutral nodes for user. Thus, f an optmal α s gven, the node parttons n the optmal soluton are characterzed as R d = R d (α ), R a = R a(α ), N = N (α ), S = S (α ) and λ = λ S = λ R = λ S (α ) = λ R (α ). References [] I. Ahmad, A. Ghafoor, Semdstrbuted load balancng for massvely parallel multcomputer systems, IEEE Trans. Softw. Eng. 7 (0) (99) [2] S. Al, T.D. Braun, H.J. Segel, A.A. Maceewsk, N. Beck, L. Bolon, M. Maheswaran, A.I. Reuther, J.P. Robertson, M.D. Theys, B. Yao, Characterzng resource allocaton heurstcs for heterogeneous computng systems, Adv. Comput. 63 (2005) [3] E. Altman, T. Basar, T. Jmenez, N. Shmkn, Routng n two parallel lnks: gametheoretc dstrbuted algorthms, J. Parallel Dstrb. Comput. 6 (9) (200) [4] Amazon elastc compute cloud. [5] T.E. Anderson, D.E. Culler, D.A. Patterson, The NOW team, a case for now (networks of workstatons), IEEE Mcro. 5 () (995) [6] X. Ba, D.C. Marnescu, L. Bolon, H.J. Segel, R.A. Daley, I.-J. Wang, A macroeconomc model for resource allocaton n large-scale dstrbuted systems, J. Parallel Dstrb. Comput. 68 (2) (2008) [7] T.D. Braun, H.J. Segel, A.A. Maceewsk, Y. Hong, Statc resource allocaton for heterogeneous computng envronments wth tasks havng dependences, prortes, deadlnes, and multple versons, J. Parallel Dstrb. Comput. 68 () (2008) [8] Y.C. Chow, W.H. Kohler, Models for dynamc load balancng n a heterogeneous multple processor system, IEEE Trans. Comput. C-28 (5) (979) [9] R. Cubert, P. Fshwck, Sm++ reference manual, n: CISE, Unversty of Florda, July 995. [0] J. Dongarra, A. Lastovetsky, An overvew of heterogeneous hgh performance and grd computng, n: Engneerng the Grd: Status and Perspectve, Amercan Scentfc Publshers, [] P. Dubey, Ineffcency of nash equlbra, Math. Oper. Res. () (986) 8. [2] A.A. Economdes, J. Slvester, A game theory approach to cooperatve and non-cooperatve routng problems, n: Proc. of the Telecommuncaton Symp., 990, pp [3] R. Freund, H.J. Segel, Heterogeneous processng, IEEE Comput. Mag. 26 (6) (993) 3 7. [4] D. Fudenberg, J. Trole, Game Theory, The MIT Press, 994. [5] L. Georgads, C. Nkolaou, A. Thomasan, A far workload allocaton polcy for heterogeneous systems, J. Parallel Dstrb. Comput. 64 (4) (2004) [6] P. Ghosh, K. Basu, S.K. Das, A game theory-based prcng strategy to support sngle/multclass ob allocaton schemes for bandwdth-constraned dstrbuted computng systems, IEEE Trans. Parallel Dstrb. Syst. 8 (3) (2007) [7] P. Ghosh, N. Roy, S.K. Das, K. Basu, A prcng strategy for ob allocaton n moble grds usng a non-cooperatve barganng theory framework, J. Parallel Dstrb. Comput. 65 () (2005) [8] D. Grosu, A.T. Chronopoulos, Noncooperatve load balancng n dstrbuted systems, J. Parallel Dstrb. Comput. 65 (9) (2005) [9] D. Grosu, A.T. Chronopoulos, M.Y. Leung, Load balancng n dstrbuted systems: an approach usng cooperatve games, n: Proc. of the 6th IEEE Intl. Parallel and Dstrbuted Processng Symp., Ft Lauderdale, Florda, USA, Aprl [20] D. Grosu, A.T. Chronopoulos, M.Y. Leung, Cooperatve load balancng n dstrbuted systems, Concurr. Comput. Pract. Exp. 20 (6) (2008) [2] R. Jan, The Art of Computer Systems Performance Analyss: Technques for Expermental Desgn, Measurement, Smulaton, and Modelng, Wley- Interscence, 99. [22] H. Kameda, J. L, C. Km, Y. Zhang, Optmal Load Balancng n Dstrbuted Computer Systems, Sprnger Verlag, London, 997. [23] S.U. Khan, I. Ahmad, Non-cooperatve, sem-cooperatve, and cooperatve games-based grd resource allocaton, n: Proc. of the Intl. Parallel and Dstrbuted Proc. Symp., Rhodes, Greece, Aprl [24] J.-K. Km, D.A. Hensgen, T. Kdd, H.J. Segel, D.S. John, C. Irvne, T. Levn, N.W. Porter, V.K. Prasanna, R.F. Freund, A flexble mult-dmensonal QoS performance measure framework for dstrbuted heterogeneous systems, Cluster Comput. 9 (3) (2006) (specal ssue on Cluster Computng n Scence and Engneerng). [25] C. Km, H. Kameda, Optmal statc load balancng of mult-class obs n a dstrbuted computer system, n: Proc. of the 0th Intl. Conf. on Dstrbuted Computng Systems, May 990, pp [26] C. Km, H. Kameda, An algorthm for optmal statc load balancng n dstrbuted computer systems, IEEE Trans. Comput. 4 (3) (992) [27] Y.A. Korls, A.A. Lazar, A. Orda, Capacty allocaton under noncooperatve routng, IEEE Trans. Automat. Control 42 (3) (997) [28] E. Koutsoupas, C. Papadmtrou, Worst-case equlbra, n: Proc. of the 6th Annual Symp. on Theoretcal Aspects of Computer Scence, 999, pp [29] Y.K. Kwok, K. Hwang, S. Song, Selfsh grds: game-theoretc modelng and NAS/PSA benchmark evaluaton, IEEE Trans. Parallel Dstrb. Syst. 8 (5) (2007) [30] H. Lee, Optmal statc dstrbuton of prortzed customers to heterogeneous parallel servers, Comput. Oper. Res. 22 (0) (995) [3] J. L, H. Kameda, A decomposton algorthm for optmal statc load balancng n tree herarchy network confguraton, IEEE Trans. Parallel Dstrb. Syst. 5 (5) (994) [32] J. L, H. Kameda, Optmal statc load balancng n star network confguratons wth two-way traffc, J. Parallel Dstrb. Comput. 23 (3) (994) [33] J. L, H. Kameda, Load balancng problems for multclass obs n dstrbuted/parallel computer systems, IEEE Trans. Comput. 47 (3) (998) [34] A. Mas-Collel, M.D. Whnston, J.R. Green, Mcroeconomc Theory, Oxford Unv. Press, New York, 995. [35] M. Mavroncolas, P. Spraks, The prce of selfsh routng, n: Proc. of the 33rd Annual ACM Symp. on Theory of Computng, July 200, pp [36] A. Muthoo, Barganng Theory wth Applcatons, Cambrdge Unv. Press, Cambrdge, UK, 999.

19 S. Penmatsa, A.T. Chronopoulos / J. Parallel Dstrb. Comput. 7 (20) [37] L.M. N, K. Hwang, Adaptve load balancng n a multprocessor system wth many ob classes, IEEE Trans. Softw. Eng. SE- (5) (985) [38] M. Koh, On-demand computng usng network.com, n: Proceedngs of the Internatonal Symposum on Grd computng, Aprl 7, 2008, Tape, Tawan. [39] A. Orda, R. Rom, N. Shmkn, Compettve routng n multuser communcaton networks, IEEE/ACM Trans. Netw. (5) (993) [40] S. Penmatsa, A.T. Chronopoulos, Cooperatve load balancng for a network of heterogeneous computers, n: Proc. of the 20th IEEE Intl. Parallel and Dstrbuted Processng Symposum, 5th Heterogeneous Computng Workshop, Rhodes Island, Greece, Aprl [4] S. Penmatsa, A.T. Chronopoulos, Prce-based user-optmal ob allocaton scheme for grd systems, n: Proc. of the 20th IEEE Intl. Parallel and Dstrbuted Processng Symposum, 3rd Hgh Performance Grd Computng Workshop, Rhodes Island, Greece, Aprl [42] G.V. Reklats, A. Ravndran, K.M. Ragsdell, Engneerng Optmzaton: Methods and Applcatons, Wley-Interscence, 983. [43] K.W. Ross, D.D. Yao, Optmal load balancng and schedulng n a dstrbuted computer system, J. ACM 38 (3) (99) [44] T. Roughgarden, Stackelberg schedulng strateges, n: Proc. of the 33rd Annual ACM Symp. on Theory of Computng, July 200, pp [45] K. Rzadca, D. Trystram, A. Werzbck, Far game-theoretc resource management n dedcated grds, n: Proc. of the 7th IEEE Intl. Symp. on Cluster Computng and the Grd, Brazl, May 2007, pp [46] N.G. Shvaratr, P. Krueger, M. Snghal, Load dstrbutng for locally dstrbuted systems, Comput. 25 (2) (992) [47] A. Stefanescu, M.V. Stefanescu, The arbtrated soluton for mult-obectve convex programmng, Rev. Roum. Math. Pure Appl. 29 (984) [48] R. Subrata, A. Zomaya, B. Landfeldt, A cooperatve game framework for QoS guded ob allocaton schemes n grds, IEEE Trans. Comput. 57 (0) (2008) [49] R. Subrata, A. Zomaya, B. Landfeldt, Game theoretc approach for load balancng n computatonal grds, IEEE Trans. Parallel Dstrb. Syst. 9 () (2008) [50] X. Tang, S.T. Chanson, Optmzng statc ob schedulng n a network of heterogeneous computers, n: Proc. of the Intl. Conf. on Parallel Processng, August 2000, pp [5] A.N. Tantaw, D. Towsley, Optmal statc load balancng n dstrbuted computer systems, J. ACM 32 (2) (985) [52] H. Yache, R.R. Mazumdar, C. Rosenberg, A game theoretc framework for bandwdth allocaton and prcng n broadband networks, IEEE/ACM Trans. Netw. 8 (5) (2000) [53] Y. Zhang, H. Kameda, S.L. Hung, Comparson of dynamc and statc loadbalancng strateges n heterogeneous dstrbuted systems, IEE Proc. Comput. Dgt. Tech. 44 (2) (997) [54] Q. Zheng, C.-K. Tham, B. Veeravall, Dynamc load balancng and prcng n grd computng wth communcaton delay, J. Grd Comput. 6 (3) (2008) Satsh Penmatsa receved hs M.S. and Ph.D. n Computer Scence from the Unversty of Texas at San Antono n 2003 and 2007 respectvely. He s currently wth the Department of Mathematcs and Computer Scence at the Unversty of Maryland Eastern Shore. Hs research nterests are n the areas of parallel and dstrbuted systems, hgh performance computng, grd computng, wreless networks, game theory, scence and engneerng applcatons. He s a member of IEEE, IEEE Computer Socety, and the ACM. Anthony T. Chronopoulos receved hs Ph.D. n Computer Scence from the Unversty of Illnos at Urbana- Champagn n 987. He s currently a Professor n Computer Scence at the Unversty of Texas at San Antono. He has publshed 39 ournal and 5 refereed conference proceedngs publcatons n the areas of Dstrbuted Systems, Hgh Performance Computng and Applcatons. He has been awarded 3 federal/state government research grants. Hs work s cted n over 220 non-coauthors research artcles. He s a senor member of IEEE (snce 997).