Energy-Efficient Task Scheduling on Multiple Heterogeneous Computers: Algorithms, Analysis, and Performance Evaluation

Transcription

1 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL., NO., JANUARY-JUNE Energy-Efficient Tas Scheduing on Mutipe Heterogeneous Computers: Agorithms, Anaysis, and Performance Evauation Keqin Li, Feow, IEEE Abstract The probems of energy-constrained and time-constrained tas scheduing on mutipe heterogeneous computers are investigated as combinatoria optimization probems. For a given set of independent tass, our strategy is to find a schedue of the tass first, and then find a power aocation to the tass, where the power aocation is performed in such a way that the tota tas execution time or the tota energy consumption is minimized. We are abe to find an optima partition of a given woroad and use a modified ist scheduing (MLS) agorithm to generate a partition of the set of tass that is an approximation of the optima woroad partition. Our simuation resuts demonstrate that when compared with optima soutions, the MLS agorithm has exceent expected performance in soving the probems of energy-constrained and time-constrained scheduing of independent tass on mutipe heterogeneous computers. For precedence constrained tass represented by a directed acycic graph (dag), our eve-by-eve modified ist scheduing (LL-MLS) agorithm schedues tass eve by eve (LL) and schedues tass in the same eve by using the MLS agorithm. We are abe to sove the probems of optima energy/time aocation to the eves and optima woroad partition for a eves in a given dag. Our simuation resuts demonstrate that when compared with optima soutions, the LL-MLS agorithm has exceent expected performance in soving the probems of energy-constrained and time-constrained scheduing of precedence constrained tass on mutipe heterogeneous computers. Index Terms Energy-constrained scheduing, expected performance bound, modified ist scheduing, mutipe heterogeneous computers, time-constrained scheduing Ç INTRODUCTION. Motivation HETEROGENEOUS system architectures (HSA) [] have attracted growing interest in modern high-performance computing to achieve higher performance/power ratio. In a recent (November 205) Top500 isting, there are 04 supercomputing systems which adopt HSA. The Sugon Custer W780I system equipped with Xeon E5-2640v3 8C 2.6 GHz and NVIDIA Tesa K80 (instaed in the Institute of Modern Physics of Chinese Academy of Sciences) can achieve the power efficiency of Gfops/Watt. In the area of coud computing, there is a consensus that heterogeneous server architectures wi dominate the future data centers [33]. A typica future heterogeneous data center contains speciaized servers and acceerators incuding graphica processing units (GPUs), fied-programmabe gate arrays (FPGAs), and digita signa processors (DSPs) various storage systems such as networ fie system (NSF) and Hadoop distributed fie system (HDFS) and fexibe interconnects such as Gigabit Ethernet and InfiniBand [9]. A heterogeneous architecture is aso abe to provide ow-power and high-throughput cores, aong with appication-specific acceerators for arge-scae appications [3]. Notice that The author is with the Department of Computer Science, State University of New Yor, New Patz, NY 256. E-mai: i@newpatz.edu. Manuscript received 3 Feb. 206 revised 3 Oct. 206 accepted 30 Oct Date of pubication Nov. 206 date of current version 5 Jan Recommended for acceptance by R.G. Mehem. For information on obtaining reprints of this artice, pease send e-mai to: reprints@ieee.org, and reference the Digita Obect Identifier beow. Digita Obect Identifier no. 0.09/TSUSC heterogeneity exists not ony within a server [8], but aso between the servers [28]. It is a chaenge on how to effectivey and efficienty manage heterogeneous processors in current and future supercomputing systems and coud computing patforms. The probem becomes more difficut when processors are equipped with the capabiity of dynamic votage and frequency scaing. Virtuay a existing studies on energy-efficient tas scheduing on mutiprocessors or mutipe computers assume that a the processors or computers have the same power consumption mode, i.e., the power consumption is proportiona to the execution speed raised to the power a, where a is the same for a processors or computers. However, it is we nown that processors from different manufacturers and vendors have very different characteristics of performance (i.e., CPU/core speed) and power consumption [2]. Unfortunatey, there has been itte study on energy-efficient tas scheduing on mutipe heterogeneous computers which are characterized by different vaues of a. The motivation of this paper is to mae some initia effort in this direction. Such investigation is certainy of theoretica interest and practica importance, since a arge-scae data center may empoy different servers manufactured by different vendors with different technoogies, which resut in different power consumption modes..2 Our Contributions In this paper, we investigate the probems of energy-constrained and time-constrained tas scheduing on mutipe heterogeneous computers. We define these probems as combinatoria optimization probems. Given a set of tass ß 206 IEEE. Persona use is permitted, but repubication/redistribution requires IEEE permission. See for more information.

2 8 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL., NO., JANUARY-JUNE 206 with an energy budget, the energy-constrained scheduing probem is to find a nonpreemptive schedue of the tass, such that the tota energy consumption does not exceed the given energy budget and that the tota tas execution time is minimized. Given a set of tass with a time deadine, the time-constrained scheduing probem is to find a nonpreemptive schedue of the tass, such that the tota tas execution time does not exceed the given time deadine and that the tota energy consumption is minimized. These probems have been investigated for independent/precedence-constrained and sequentia/parae tass on homogeneous computers [4], [5], [6], [8], [9], [20]. We find that for independent tass, both probems contain two subprobems, i.e., power aocation and tas scheduing. For a given set of tass, our strategy is to find a schedue of the tass first, and then find a power aocation to the tass, where the power aocation is performed in such a way that the tota tas execution time or the tota energy consumption is minimized (Theorems and 2). Our investigation reveas the fact that both scheduing probems can be reduced to the probem of finding an optima partition of the given set of tass into disoint subsets. Fortunatey, we are abe to find an optima partition of a given woroad (Theorems 3 and 4), and use a modified ist scheduing (MLS) agorithm to generate a partition of the set of tass that is an approximation of the optima woroad partition. Our simuation resuts demonstrate that when compared with optima soutions, the MLS agorithm has exceent expected performance in soving the probems of energy-constrained and time-constrained scheduing of independent tass on mutipe heterogeneous computers. For precedence constrained tass represented by a directed acycic graph (dag), there is an additiona subprobem of deaing with tas inter-dependency. Our strategy to hande precedence constraints is to schedue tass eve by eve (LL) and to schedue tass in the same eve by using the MLS agorithm. Hence, our agorithm is caed eve-byeve modified ist scheduing (LL-MLS) agorithm. The ey issue in using the LL scheduing method is to find an optima energy or time aocation to the eves. Fortunatey, we are abe to sove the probems of optima energy/time aocation to the eves and optima woroad partition for a eves in a given dag (Theorems 5 and 6). Our simuation resuts demonstrate that when compared with optima soutions, the LL-MLS agorithm has exceent expected performance in soving the probems of energy-constrained and time-constrained scheduing of precedence constrained tass on mutipe heterogeneous computers. To the best of our nowedge, this is the first paper that investigates energy-efficient tas scheduing on mutipe heterogeneous computers in a systematic and anaytic way..3 Paper Outine Due to space imitation, this section is moved to the suppementary materia, which can be found on the Computer Society Digita Library at org/0.09/tsusc RELATED RESEARCH Reducing processor energy consumption has been an important and pressing research issue in recent years. There has been increasing interest and importance in deveoping highperformance and energy-efficient computing systems. There exists an exposive body of iterature on energy-efficient computing (see [3], [4], [32], [39] for comprehensive surveys). 2. Heterogeneous High-Performance Computing Energy-efficient high-performance computing on heterogeneous systems has been studied extensivey by numerous researchers. In [22], the authors were concerned with the probem of scheduing a bag-of-tass appication, made of a coection of independent stochastic tass with norma distributions of tas execution times, on a heterogeneous patform with deadine and energy consumption budget constraints. In [25], the authors proposed a new energy-aware scheduing agorithm with reduced tas dupication, which taes the energy consumption as we as the maespan of an appication into consideration. In [29], the authors deveoped a reatime and energy-efficient resource scheduing and optimization framewor to achieve high energy efficiency and ow response time in big data stream computing environments. In [30], the authors proposed an energy-efficient worfow tas scheduing agorithm in order to obtain more energy reduction as we as maintain the quaity of service by meeting the deadines. In [35], the authors addressed the probem of energy-aware data aocation and tas scheduing on a heterogeneous distributed shared-memory mutiprocessor system for rea-time appications. In [37], the authors devised a nove reiabiity maximization with energy constraint agorithm to effectivey baance the tradeoff between high reiabiity and ow energy consumption. In [38], the authors deveoped new green tas scheduing agorithms for heterogeneous computers with changeabe continuous and discrete speeds to reduce energy consumption as much as possibe and finish a tass before a deadine. 2.2 Heterogeneous Coud Computing Many researchers have investigated various issues in heterogeneous coud systems, incuding resource aocation, tas scheduing, performance guarantee, and energy saving. In [], the authors outined a principed approach to designing energy-efficient and heterogeneous data centers that are robust against data center woroad variations. In [23], the authors emphasized that the operationa cost of data centers is dominated by the cost on energy consumption, and modeed a data center as a cyber physica system to capture its therma properties. In [24], the authors presented a green strategy mode for heterogeneous coud systems, and provided a soution for heterogeneous ob-communicating tass and heterogeneous virtua machines that mae up the nodes of the coud to guarantee the service-eve agreement and to optimize energy savings. In [26], the authors had the obective to satisfy performance expectations of customers by heterogeneous dynamic dedicated server scheduing whie considering heterogeneous servers and different priority casses of customers. In [27], the authors proposed an optimization strategy based on a mixed integer programming mode for achieving improvement on power-efficiency whie providing performance guarantee in the virtuaized custer. In [34], the authors studied the muti-resource aocation probem in coud computing systems where the resource poo is

3 LI: ENERGY-EFFICIENT TASK SCHEDULING ON MULTIPLE HETEROGENEOUS COMPUTERS: ALGORITHMS, ANALYSIS, AND... 9 constructed from a arge number of heterogeneous servers, representing different points in the configuration space of resources such as processing, memory, and storage. However, the above research did not incude dynamic votage and frequency scaing into consideration. Due to space imitation, some part of this section is moved to the suppementary materia, avaiabe onine. 3 POWER ALLOCATION The power consumption mode adopted in this paper is the foowing, which is a standard mode used by virtuay a researchers in the fied [3], [4], [5], [6], [8], [9], [20], [22], [25], [29], [30], [35], [37], [38], [39]. Power dissipation and circuit deay in digita CMOS circuits can be accuratey modeed by simpe equations, even for compex microprocessor circuits. CMOS circuits have dynamic, static, and short-circuit power dissipation however, the dominant component in a we designed circuit is dynamic power consumption p (i.e., the switching component of power), which is approximatey p ¼ acv 2 f, where a is an activity factor, C is the oading capacitance, V is the suppy votage, and f is the coc frequency [7]. Since s / f, where s is the processor speed, and f / V f with 0 < f, which impies that V / f =f, we now that power consumption is p / f a and p / s a, where a ¼ þ 2=f 3. Furthermore, the vaue of a can be as high as for high suppy votage [36]. (Note: Athough it has been nown that power consumption can be highy dependent on types of appications [0], we assume that p is primariy determined by s, as in the existing iterature.) The main parameter which characterizes the power efficiency of a processor is a. In this paper, we consider tas scheduing on m heterogeneous computers specified by a a 2... a m. Assume that we are given n independent or precedence constrained sequentia tass to be executed on m heterogeneous processors. Let r i represent the execution requirement (i.e., the number of CPU cyces or the number of instructions) of tas i, where i n. We use p i to represent the power aocated to execute tas i which is executed on processor. For ease of discussion, we wi assume that p i is simpy s i, where s i ¼ p = i is the execution speed of tas i. The execution time of tas i is t i ¼ r i =s i ¼ r i =p = i. The energy consumed to execute tas i is e i ¼ p i t i ¼ r i p = i ¼ r i s i. 3. Energy-Constrained Scheduing In this section, we define and examine the energy-constrained scheduing probem. Given a set of n tass with execution requirements r r 2...r n, a group of m heterogeneous computers characterized by a a 2... a m, and an energy constraint E, the energy-constrained scheduing probem is to find a power aocation p p 2...p n to the n tass and a nonpreemptive schedue of the n tass on the m computers, such that the tota energy consumption of the n tass does not exceed E and that the tota execution time of the n tass (i.e., the schedue ength) is minimized. A schedue of a set of n tass on m computers is actuay a partition of the set of n tass into m disoint subsets or groups, such that tass in the th group are executed on the th computer. Let R denote the th group as we as the tota execution requirement of the tass in the th group, where m. We use R ¼ r þ r 2 þþr n ¼ R þ R 2 þþr m to represent the tota execution requirement of a the n tass. Assume that the given energy budget E is divided into m parts, i.e., E E 2...E m, such that E is aocated to computer, where m. The foowing theorem gives the optima power aocation and the minimized schedue ength for a given schedue. Theorem. For a given energy constraint E and a given schedue of a set of n tass on m computers R R 2...R m, the minimized schedue ength T satisfies E ¼ Ra T þ Ra2 2 a T a 2 Ram þþ m T : am The above minimized schedue ength is achieved when E ¼ R =T a and a tass in R are aocated with the same power p ¼ðR =TÞ and executed with the same speed s ¼ R =T, for a m. Proof. It is aready nown in [4] that given energy constraint E, the tota execution time of the tass in R on computer is minimized when a tass in R are suppied with the same power p ¼ðE =R Þ =ð Þ and executed with the same speed s ¼ðE =R Þ =ða Þ, and the minimized execution time is T ¼ R =ð Þ =E =ð Þ : It is cear that in order to minimize the schedue ength T of the n tass on the m computers, we need to have T ¼ T 2 ¼¼T m ¼ T, i.e., a the m computers compete their assigned tass at the same time. Hence, we get T ¼ R =ð Þ =E =ð Þ which gives E ¼ R =T a for a m. Since E ¼ E þ E 2 þþe m, we have E given in the theorem. It is easy to verify that p ¼ ðr =T Þ and s ¼ R =T, for a m. This proves the theorem. tu The significance of Theorem is that we have reduced the energy-constrained scheduing probem on mutipe heterogeneous computers to the probem of finding a schedue R R 2...R m, such that T is minimized. This suggests an effective approach to soving the energy-constrained scheduing probem. The method consists of three steps. In the first step, we find R R 2...R m, such that T is minimized (see Section 4. and Theorem 3). In the second step, we find a partition R R 2...R m of the n tass into m disoint groups, such that R R 2...R m are cose to R R 2...R m (see Section 5.). In the third step, we use Theorem to find the optima power aocation p p 2...p m and speed setting s s 2...s m for the given schedue R R 2...R m. 3.2 Time-Constrained Scheduing In this section, we define and examine the time-constrained scheduing probem. Given a set of n tass with execution requirements r r 2...r n, a group of m heterogeneous computers characterized by a a 2... a m, and a time constraint T, the timeconstrained scheduing probem is to find a power aocation p p 2...p n to the n tass and a nonpreemptive schedue of the n tass on the m computers, such that the tota execution

4 0 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL., NO., JANUARY-JUNE 206 time of the n tass does not exceed T and that the tota energy consumption of the n tass is minimized. The foowing theorem gives the optima power aocation and the minimized energy consumption for a given schedue. Theorem 2. For a given a time constraint T and a given schedue of a set of n tass on m computers R R 2...R m, the minimized energy consumption E is E ¼ Ra T a þ Ra2 2 T a 2 Ram þþ m T am : The above minimized energy consumption is achieved when a tass in R are aocated with the same power p ¼ðR =T Þ and executed with the same speed s ¼ R =T, for a m. Proof. It is aready nown in [4] that given time constraint T, the tota energy consumption of the tass in R on computer is minimized when a tass in R are suppied with the same power p ¼ðR =TÞ and executed with the same speed s ¼ R =T, and the minimized energy consumption is E ¼ R =T a where m. This impies that given a time constraint T and a partition R R 2...R m, the minimized energy consumption of the n tass is E ¼ E þ E 2 þþe m, i.e., the E given in the theorem. This proves the theorem. tu The significance of Theorem 2 is that we have reduced the time-constrained scheduing probem on mutipe heterogeneous computers to the probem of finding a schedue R R 2...R m, such that E is minimized. This suggests an effective approach to soving the time-constrained scheduing probem. The method consists of three steps. In the first step, we find R R 2...R m, such that E is minimized (see Section 4.2 and Theorem 4). In the second step, we find a partition R R 2...R m of the n tass into m disoint groups, such that R R 2...R m are cose to R R 2...R m (see Section 5.). In the third step, we use Theorem 2 to find the optima power aocation p p 2...p m and speed setting s s 2...s m for the given schedue R R 2...R m. 4 LOWER BOUNDS 4. A Lower Bound for Optima Schedue Length The main purpose of this section is to deveop a numerica method to find R R 2...R m, i.e., an optima partition of a given woroad R, assuming that R R 2...R m are continuous variabes, such that T in Theorem is minimized. From Theorem, we now that the minimized schedue ength T can be viewed as a function of R R 2...R m.we are interested in the foowing optimization probem (i.e., optima woroad partition), namey, given m, a a 2... a m, R, and E, finding a partition R R 2...R m of the woroad R, such that T is minimized, subect to the constraint that F ¼ R þ R 2 þþr m ¼ R: Unfortunatey, there is no cosed-form soution. Our obective is to deveop a numerica method to find R R 2...R m and the minimized T. The significance of the study is two fod. First, the vaues of R R 2...R m can be used to guide us in finding an optima schedue to sove the energy-constrained scheduing probem. Second, the minimized T can be used as a ower bound for the optima schedue ength T OPT, such that our soutions can be compared with the optima soutions. Our main resut of this section is the foowing theorem. Theorem 3. The optima schedue ength has a ower bound T OPT T, where the minimized T and the partition R R 2...R m that resuts in T can be obtained by soving the m þ equations, i.e., the constraint and m equations T ¼ R þ R 2 þþr m ¼ R R R E =ða Þ for a m. Note: The ower bound is impicity given by equations in the theorem. We wi deveop a numerica agorithm to sove these equations after the proof. The main idea of the proof is essentiay to treat T as a function of R R 2...R m. The optima woroad partition probem can be soved by minimizing T using a standard Lagrange mutipier system. The proof can be sipped if the reader is ess interested in the detais. Proof. It is cear that the optima schedue ength may not reach the minimized T, since the partition R R 2...R m of R may not be reaized by a partition of the n tass, due to the imited vaues of r r 2...r n. Hence, we have T OPT T. To sove the optimization probem, we use a Lagrange mutipier system, i.e., rt ¼ frf where f is a Lagrange mutipier. The above equation impies ¼ f@f=@r for a m. To et us reca the equation of T in Theorem, namey, E ¼ Ra T a þ Ra2 2 T a 2 Ram þþ m T am : We tae a partia derivative of R on both sides of the above equation and get R T Xm which gives rise ¼ R T ð ÞR T a ¼ ð ÞR a T a for a m. ¼ we obtain R T for a m. ð ÞR a T a ¼ f

5 LI: ENERGY-EFFICIENT TASK SCHEDULING ON MULTIPLE HETEROGENEOUS COMPUTERS: ALGORITHMS, ANALYSIS, AND... The ast equation can be rewritten as R =T a ¼ fs where S ¼ Xm ð ÞR T a : Hence, by Theorem, we have E ¼ R T ¼ fs R for a m. The above equation impies that and and E ¼ E þ E 2 þþe m ¼ fs Xm fs ¼ E Xm R R E ¼ R X m R E a R = ¼ E R =a þ R 2 =a 2 þþr m =a m for a m. By Theorem, we have T ¼ R =E =ða Þ : Substituting E, we get T ¼ R R E =ða Þ for a m. From the ast m equations, pus the condition R þ R 2 þþr m ¼ R we can find R R 2...R m and the resuting minimized T. tu It is cear that the sophistication of the system of noninear equations in Theorem 3 does not accommodate a cosed-form soution. In the foowing, we deveop a numerica method to find a soution to the m þ equations in Theorem 3. First, we need a method to find R R 2...R m for a fixed T. To this end, we et W ¼ P m ¼ R = : Therefore, we obtain E =ða Þ R ¼ T W for a m. It is cear that W can be found by the cassic bisection method [5]. The reason is that R R 2...R m a decrease with W. Hence, our numerica method can be described as foows. First, et W ¼ W 0 be some arbitrary vaue. We eep reducing W (e.g., having W) and increasing R R 2...R m, unti W ¼ W and W < P m ¼ R = : Next, we et W ¼ W 0 and eep increasing W (e.g., doubing W) and decreasing R R 2...R m, unti W ¼ W 2 and W 2 > P m ¼ R = : Finay, we can search W in ½W W 2 Š, such that W ¼ P m ¼ R = : Once W is determined, R R 2...R m can be cacuated in a straightforward manner. The detaied procedure of the above method is given in Agorithm of the suppementary materia, avaiabe onine. Second, we can aso find the minimized T by using the bisection method. It is observed that the vaues in R R 2...R m a increase with T. Hence, our numerica method can be described as foows. First, et T ¼ T 0 be some arbitrary vaue. We eep reducing T (e.g., having T) and decreasing R R 2...R m, unti R þ R 2 þþ R m <R: Next, we et T ¼ T 0 and eep increasing T (e.g., doubing T) and increasing R R 2...R m, unti R þ R 2 þþr m > R: Finay, we can search T in ½T T 2 Š, such that R þ R 2 þþr m ¼ R: The fina vaues of R R 2...R m are considered as the numerica soution of R R 2...R m. The detaied procedure of the above method is given in Agorithm 2 of the suppementary materia, avaiabe onine. 4.2 A Lower Bound for Optima Energy Consumption The main purpose of this section is to deveop a numerica method to find R R 2...R m, i.e., an optima partition of a given woroad R, assuming that R R 2...R m are continuous variabes, such that E in Theorem 2 is minimized. From Theorem 2, we now that the minimized energy consumption E can be viewed as a function of R R 2...R m. We are interested in the foowing optimization probem (i.e., optima woroad partition), namey, given m, a a 2... a m, R, and T, finding a partition R R 2...R m of the woroad R, such that E is minimized, subect to the constraint that F ¼ R þ R 2 þþr m ¼ R: Again, there is no cosed-form soution. Our obective is to deveop a numerica method to find R R 2...R m and the minimized E. The significance of the study is two fod. First, the vaues of R R 2...R m can be used to guide us in finding an optima schedue to sove the time-constrained scheduing probem. Second, the minimized E can be used as a ower bound for the optima energy consumption E OPT, such that our soutions can be compared with the optima soutions. Our main resut of this section is the foowing theorem. Theorem 4. The optima energy consumption has a ower bound E OPT E, where the minimized E and the partition R R 2...R m that resuts in E are E ¼ T Xm f a =ð Þ and R ¼ ¼ f =ða Þ T for a m, and f satisfies ¼ f =ða Þ ¼ R T : Note: The ower bound is impicity given by equations in the theorem. We wi deveop a numerica agorithm to sove these equations after the proof. The main idea of the proof is simiar to that of Theorem 3 and can be sipped.

6 2 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL., NO., JANUARY-JUNE 206 Proof. It is cear that the optima energy consumption may not reach the minimized E, since the partition R R 2...R m of R may not be reaized by a partition of the n tass. Hence, we have E OPT E. To sove the optimization probem, we use a Lagrange mutipier system, i.e., re ¼ frf where f is a Lagrange mutipier. The above equation impies ¼ f@f=@r for a m. From Theorem 2, we get R =T a ¼ f and consequenty, R ¼ f =ða Þ T for a m. Since R þ R 2 þþr m ¼ R, we get R ¼ T Xm f =ða Þ : By Theorem 2, we have E ¼ Xm ¼ ¼ R a T ¼ T Xm f ¼ a =ð Þ The theorem is proven. tu The equation for f in Theorem 4 can be soved by using the bisection method, where we observe that the eft-hand side of the equation is an increasing function of f. Hence, our numerica method can be described as foows. Assume that S ¼ Xm f =ða Þ : ¼ First, et f ¼ f 0 be some arbitrary vaue. We eep reducing f (e.g., having f), unti S < R=T. Next, we et f ¼ f 0 and eep increasing f (e.g., doubing f), unti S > R=T. Finay, we can search f in ½f f 2 Š, such that S ¼ R=T. Once f is avaiabe, we can cacuate R R 2...R m and the minimized energy consumption E. The detaied procedure of the above method is given in Agorithm 3 of the suppementary materia, avaiabe onine. : 5 SCHEDULING INDEPENDENT TASKS 5. The MLS Agorithm We now describe our heuristic agorithms. Let R R 2...R m denote the partition obtained in Theorem 3 which resuts in the minimized T. Our strategy to sove the energy-constrained scheduing probem is to find a partition R R 2...R m of the n tass into m disoint groups, such that R R 2...R m are cose to the optima woroad partition R R 2...R m. Let R R 2...R m denote the partition obtained in Theorem 4 which resuts in the minimized E. Our strategy to sove the time-constrained scheduing probem is to find a partition R R 2...R m of the n tass into m disoint groups, such that R R 2...R m are cose to the optima woroad partition R R 2...R m. For both energy-constrained scheduing and time-constrained scheduing, we are facing the same probem, namey, given R R 2...R m, finding a partition of the n tass into m disoint groups, such that R R 2...R m are cose to R R 2...R m. Once a schedue R R 2...R m is determined, we can use Theorem to find an optima power aocation for energy-constrained scheduing, or Theorem 2 to find an optima power aocation for time-constrained scheduing. To generate a partition of the n tass into m disoint groups, we treat the tas execution requirements r r 2... r n as tas execution times. (Note: They are not the actua tas execution times, but ony the input to the MLS agorithm for the purpose of generating a partition.) The n tass are assigned to the m computers by using the cassic ist scheduing agorithm [2] with some modifications. Our modified ist scheduing agorithm wors as foows. Initiay, the n tass are arranged into a ist. The first m tass are assigned to the m computers, one tas per computer. Whenever a computer competes a tas, the next tas in the ist is assigned to the computer. As soon as the tota tas execution requirement of a computer exceeds R, the computer is boced and does not accept more tass. The resut of the above procedure is a tas assignment to the m computers, i.e., a schedue of the n tass on the m computers. The resuted partition R R 2...R m is considered as an approximation of R R 2...R m. 5.2 Numerica Exampes In this section, we give exampes of our numerica cacuations. In Tabe, we demonstrate numerica data of energyconstrained scheduing of independent tass. We consider m ¼ 7 heterogeneous computers with ða a 2 a 3 a 4 a 5 a 6 a 7 Þ¼ð3:0 3:5 4:0 4:5 5:0 5:5 6:0Þ: (Note: As commony accepted, the vaue of a is at east 3 [7]. As mentioned earier, the vaue of a can be as high as for high suppy votage [36].) Let us assume that the tota amount of tas execution requirement is set as R ¼ 00. For energy constraint E ¼ , we show R, E, p, and s for a m. (Note: The vaues of R and E are set in such a way that the execution speeds are in a reasonabe range.) These data are cacuated by using Theorem 3. The minimized T is aso given. We observe that due to the heterogeneity of the s, the woroad R and the energy E are uneveny distributed among the computers, such that a computer with a smaer is assigned more woroad and aocated more energy. Furthermore, tass assigned to a computer with a smaer are suppied with more power and executed with faster speed. In fact, as E increases, the ratios R =R m, E =E m, p =p m, and s =s m a increase. In Tabe 2, we demonstrate numerica data of time-constrained scheduing of independent tass. The vaues of m, R, and the s are the same as Tabe. For time constraint T ¼ 2:0 0:5 9:0 7:5 6:0, weshowr, E, p,ands for a m. These data are cacuated by using Theorem 4. The minimized E is aso given. It is observed that as T decreases, the ratios R =R m, E =E m, p =p m,ands =s m a increase. 5.3 Performance Data We now present simuation data to demonstrate the performance of our heuristic agorithms on m ¼ 7 heterogeneous computers with the same s in Section 5.2.

7 LI: ENERGY-EFFICIENT TASK SCHEDULING ON MULTIPLE HETEROGENEOUS COMPUTERS: ALGORITHMS, ANALYSIS, AND... 3 TABLE Numerica Data of Energy-Constrained Scheduing of Independent Tass E R E p s T , TABLE 2 Numerica Data of Time-Constrained Scheduing of Independent Tass T R E p s E , The soutions produced by our agorithms wi be compared with optima soutions (actuay, the ower bounds in Theorems 3 and 4). Our performance measures are defined as foows. Let A be a heuristic agorithm for the energy-constrained scheduing probem. We use T A to denote the ength of the schedue produced by agorithm A. Then, the ratio T A =T OPT is caed the performance ratio of agorithm A. If T A =T OPT B, then B is caed a performance bound. Let T denote the minimized T in Theorem 3. It is cear that B ¼ T A =T is a performance bound. Furthermore, if the tas execution requirements are random variabes, T A, T, and B are a random variabes. The expectation B is caed the expected performance bound. Let A be a heuristic agorithm for the time-constrained scheduing probem. We use E A to denote the amount of energy consumed by agorithm A. Then, the ratio E A =E OPT is caed the performance ratio of agorithm A. If E A =E OPT C, then C is caed a performance bound. Let E denote the minimized E in Theorem 4. It is cear that C ¼ E A =E is a performance bound. Furthermore, if the tas execution requirements are random variabes, E A, E, and C are a random variabes. The expectation C is caed the expected performance bound. In Tabe 3, we show simuation data for the expected performance bound B of the modified ist scheduing agorithm for the energy-constrained scheduing probem. For each combination of n ¼ and E ¼ , we generate 2,000 sets of n tass, whose tas execution requirements are random numbers uniformy TABLE 3 Simuation Data of B for Independent Tass (CI¼ 0:34%) n E ¼ 00 E ¼ 200 E ¼ 300 E ¼ 400 E ¼

8 4 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL., NO., JANUARY-JUNE 206 TABLE 4 Simuation Data of C for Independent Tass (CI¼ 0:44%) n T ¼ 4:0t T ¼ 3:5t T ¼ 3:0t T ¼ 2:5t T ¼ 2:0t distributed in ½0 Š. For each set of tass, we cacuate the ower bound T of T OPT by using Theorem 3. We aso simuate the MLS agorithm and find its schedue ength T MLS. The performance bound is obtained as B ¼ T MLS =T. The average vaue of the 2,000 vaues of B is then reported in Tabe 3. The 99 percent confidence interva of a the data in Tabe 3 is no more than 0:34 percent. In Tabe 4, we show simuation data for the expected performance bound C of the modified ist scheduing agorithm for the time-constrained scheduing probem. For each combination of n ¼ and T ¼ 4:0t 3:5t 3:0t 2:5t 2:0t, where t ¼ n=50, we generate 2,000 sets of n tass, whose tas execution requirements are random numbers uniformy distributed in ½0 Š. For each set of tass, we cacuate the ower bound E of E OPT by using Theorem 4. We aso simuate the MLS agorithm and find its energy consumption E MLS. The performance bound is obtained as C ¼ E MLS =E. The average vaue of the 2,000 vaues of C is then reported in Tabe 4. The 99 percent confidence interva of a the data in Tabe 4 is no more than 0:44 percent. In the foowing, we point out some observations. From Tabe 3, we observe that B is very cose to. For a given n, the expected performance bound B sighty increases as E increases, and then decreases as E further increases, primariy due to the ower bounds. For a given E, the expected performance bound B decreases as n increases. From Tabe 4, we observe that C is very cose to. For a given n, the expected performance bound C sighty increases as T decreases, and then decreases as T further decreases, primariy due to the ower bounds. For a given T, the expected performance bound C decreases as n increases. Notice that a the data in Tabes 3 and 4 are for expected performance bounds. The actua vaues of expected performance ratios are smaer and coser to the optima. Our simuation resuts demonstrate that the MLS agorithm has exceent expected performance in soving the probems of energy-constrained and time-constrained scheduing of independent tass on mutipe heterogeneous computers. 6 SCHEDULING PRECEDENCE CONSTRAINED TASKS 6. The LL-MLS Agorithm A set of precedence constrained tass can be represented by a directed acycic graph. The nodes in a dag are tass and arcs in the dag are precedence constraints. A dag can be divided into v eves. Tass with no any predecessor are in eve. In genera, a tas is in eve if the ongest path from a node in eve to the tas contains nodes. Let R denote the tota execution requirement of tass in eve,where v. Our strategy to schedue precedence constrained tass is to schedue tass in a dag eve by eve,thatis,tassineve cannot be executed unti a tass in eve are finished. Since tass in each eve are independent of each other, they can be schedued by using the MLS agorithm. Therefore, our agorithm is caed eve-by-eve modified ist scheduing agorithm. Let E denote the amount of energy aocated to tass in eve (or, the amount of energy consumed by tass in eve ), where v, such that E þ E 2 þþe v ¼ E. Let T denote the execution time aocated to tass in eve (or, the tota execution time of tass in eve ), where v, such that T þ T 2 þþt v ¼ T. Our main concern for energy-constrained scheduing is the determination of E E 2...E v, such that the tota execution time of a tass in a dag, i.e., T ¼ T þ T 2 þþt v,isminimized (see Section 6.2 and Theorem 5). Once an optima energy aocation E E 2...E v is avaiabe, we can use the MLS agorithm in Section 5. to schedue the v eves separatey and sequentiay. Our main concern for timeconstrained scheduing is the determination of T T 2...T v, such that the tota energy consumption of a tass in a dag, i.e., E ¼ E þ E 2 þþe v, is minimized (see Section 6.3 and Theorem 6). Once an optima time aocation T T 2...T v is avaiabe, we can use the MLS agorithm in Section 5. to schedue the v eves separatey and sequentiay. We wi address these two issues in the next two sections. 6.2 Optima Energy Aocation According to the eve-by-eve scheduing method, the tota execution time T of a tass in a dag is simpy the summation of the T s, i.e., T ¼ T þ T 2 þþt v where T is viewed as a function of E, and T is viewed as a function of E E 2...E v. Hence, we can define the foowing optimization probem (i.e., optima energy aocation and woroad partition), namey, given m, a a 2... a m, R R 2...R v, and E, finding E E 2...E v, such that T is minimized, subect to the constraint that F ¼ E þ E 2 þþe v ¼ E: Notice that each R shoud be further divided into R R 2...R m by using Theorem 3, such that T is minimized for a given E, where v. Note: The main purpose of the foowing mathematica derivation is to deveop a numerica procedure to sove the above optimization probem. The reader can sip this part and go to Tabe 5. To sove the above optimization probem, we use a Lagrange mutipier system, i.e., rt ¼ frf where f is a Lagrange mutipier. The above equation impies @E or, =@E ¼ f for a v. In the foowing, we deveop a numerica procedure to sove the optimization probem.

9 LI: ENERGY-EFFICIENT TASK SCHEDULING ON MULTIPLE HETEROGENEOUS COMPUTERS: ALGORITHMS, ANALYSIS, AND... 5 TABLE 5 Numerica Data of Energy-Constrained Scheduing of Precedence Constrained Tass E R E p s T To =@E, we reca from Theorem 3 that for a given E, there is an optima partition R R 2...R m which resuts in the minimized T. The vaues of R R 2...R m and T can be obtained by soving the m þ equations, i.e., the constraint and m equations T ¼ R þ R 2 þþr m ¼ R R R E =ða Þ for a m. From Tabe, we now that T as we as R R 2...R m are a functions of E. We tae a partia derivative of E on both sides of the above equation and T 2 þr ð ÞR 2 R 0 R 0 R E R E R where R 0 =@E, for a m. The ast equation can be rewritten as E 2 ð ÞR 0 ¼ ft 2 R E þ R R 2 þ R R 0 R E for a m. For a given f, we need to find E, where v. For a given E, the vaues of R R 2...R m and T can be obtained from Theorem 3. The vaues of R 0 R0 2...R0 m can be obtained by soving the above m equations, which form a system of inear equations, X 6¼ R ¼ ft 2 R 0 þ ð Þ E R 2 þ R R þ R R E R 0 for a m. The detaied procedure of the above method is given in Agorithm 4 of the suppementary materia, avaiabe onine. Hence, we ony need to determine E, such that R 0 þ R0 2 þþr0 m ¼ 0. It is noticed that R 0 þ R0 2 þþr0 m is a decreasing function of E, and the vaue of E can be found by using the bisection method. The detaied procedure of the above method is given in Agorithm 5 of the suppementary materia, avaiabe onine. Finay, to find E E 2...E v, we notice that a the E s as we as E þ E 2 þþe v are increasing functions of f. Therefore, we ony need to determine f, such that E þ E 2 þþe v ¼ E, by using the bisection method. Once E E 2...E v are avaiabe, we can cacuate R R 2...R m and T, for a v, by using Theorem 3, and T as a direct consequence. The detaied procedure of the above method is given in Agorithm 6 of the suppementary materia, avaiabe onine. In Tabe 5, we demonstrate numerica data of energy-constrained scheduing of precedence constrained tass. We consider m ¼ 7 heterogeneous computers with the same s in Section 5.2. Assume that a dag has v ¼ 4 eves, with R ¼ 0, where v, and R ¼ 00. The energy constraint is E ¼ 200. For a v, weshowe and T,asweasR, E, p,ands, for a m. The foowing facts are observed, which are described by surprisingy simpe reations, and are not obvious at a from the above derivation. Theorem 5. Assume that () R R 2...R m is an optima partition of a woroad R of independent tass obtained from Theorem 3 (2) E E 2...E m is the corresponding optima energy aocation of a given energy constraint E (3) T is the resuting minimized schedue ength (4) p p 2...p m are the power settings of the m computers (5) s s 2...s m are the speed settings of the m computers. For the optima energy aocation and woroad partition probem with R R 2...R v and E, where R þ R 2 þþr v ¼ R, we have the foowing facts. ) E, R, E, and T are ineary proportiona to R, i.e., they can be obtained from E, R, E, and T by scaing a factor of R =ðr þ R 2 þþr v Þ, for a v. 2) p ¼ p 2 ¼¼p v ¼ p. 3) s ¼ s 2 ¼¼s v ¼ s.

10 6 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL., NO., JANUARY-JUNE 206 Proof. Notice that the minimized T 0 of the optima energy aocation and woroad partition probem cannot be shorter than the minimized T of the woroad partition probem of the same set of tass with a precedence constraints removed (i.e., tass become independent of each other). The minimized T of the set of independent tassisgivenbytheorem3.ifwecanshowthatt 0 is the same as T, thent 0 wi be obviousy optima, i.e., we get an optima energy aocation and woroad partition. Fortunatey, this is indeed true, as stated in the theorem. Theeyobservationisthefactthattheequationsin Theorem 3 sti hod if E, ther s, and T are scaed by a common factor. This expains Fact ), i.e., if we set E, the R s, the E s, and T according to Fact ), then they wi satisfy the equations in Theorem 3, and become the input and output of the woroad partition probem for eve, fora v. Thisimpiesthat T 0 ¼ T þ T 2 þþt v ¼ T, andwedogetasoutionto the optima energy aocation and woroad partition probem. Furthermore, since p ¼ðR =T Þ and s ¼ R =T, for a v and m, Facts 2)-3) are direct consequences, since R and T are a scaed by a common factor. tu 6.3 Optima Time Aocation According to the eve-by-eve scheduing method, the tota energy consumption E of a tass in a dag is simpy the summation of the E s, i.e., E ¼ E þ E 2 þþe v where E is viewed as a function of T, and E is viewed as a function of T T 2...T v. Hence, we can define the foowing optimization probem (i.e., optima time aocation and woroad partition), namey, given m, a a 2... a m, R R 2... R v, and T, finding T T 2...T v, such that E is minimized, subect to the constraint that F ¼ T þ T 2 þþt v ¼ T: Notice that each R shoud be further divided into R R 2...R m by using Theorem 4, such that E is minimized for a given T, where v. Note: The main purpose of the foowing mathematica derivation is to deveop a numerica procedure to sove the above optimization probem. The reader can sip this part and go to Tabe 6. To sove the above optimization probem, we use a Lagrange mutipier system, i.e., re ¼ frf where f is a Lagrange mutipier. The above equation impies ¼ f@f=@t or, =@T ¼ f for a v. In the foowing, we deveop a numerica procedure to sove the optimization probem. To =@T, we reca from Theorem 4 that for a given T, there is an optima partition R R 2...R m which resuts in the minimized E given by E ¼ T ¼ f a =ð Þ TABLE 6 Numerica Data of Time-Constrained Scheduing of Precedence Constrained Tass T R E p s E where f is aso a function of T. We tae a partia derivative of T on both sides of the above equation ¼ Xm ¼ f a =ð Þ þ T f 0 ¼ =ða Þ f where f 0 =T. It remains to find f 0. Since f satisfies ¼ f =ða Þ ¼ R T we can tae a partia derivative of T on both sides of the above equation and get f 0 ¼ which impies that f 0 ¼ R T 2 ð Þ ¼ a ða 2Þ=ð Þ f ¼ R T 2 a ða 2Þ=ð Þ : ð Þ f Substituting f 0 into the equation =@T and noticing =@T ¼ f, we get

11 LI: ENERGY-EFFICIENT TASK SCHEDULING ON MULTIPLE HETEROGENEOUS COMPUTERS: ALGORITHMS, ANALYSIS, AND... 7 TABLE 7 Simuation Data of B for Precedence Constrained Tass (CI¼ 0:459%) v E ¼ 200 E ¼ 400 E ¼ 600 E ¼ 800 E ¼ TABLE 8 Simuation Data of C for Precedence Constrained Tass (CI¼ :809%) v T ¼ 0:45t T ¼ 0:40t T ¼ 0:35t T ¼ 0:30t T ¼ 0:25t f ¼ Xm ¼ R T f a =ð Þ ¼ ¼ a ða 2Þ=ð Þ ð Þ f f =ða Þ : It is noticed that the right-hand side of the above equation is negative and an increasing function of T. Thus, for any given f < 0, we can find T by using the bisection method. The detaied procedure of the above method is given in Agorithm 7 of the suppementary materia, avaiabe onine. To determine T T 2...T v, we notice that a the T s as we as T þ T 2 þþt v are increasing functions of f. Therefore, we ony need to determine f, such that T þ T 2 þþt v ¼ T. Once T T 2...T v are avaiabe, we can cacuate R R 2...R m and E, for a v, by using Theorem 4, and E as a direct consequence. The detaied procedure of the above method is given in Agorithm 8 of the suppementary materia, avaiabe onine. In Tabe 6, we demonstrate numerica data of time-constrained scheduing of precedence constrained tass. The vaues of m, the s, v, and the R s are the same as Tabe 5. The time constraint is T ¼ 2. For a v, we show T and E, as we as R, E, p, and s, for a m. We observe the same facts as those in Tabe 5. Theorem 6. Assume that () R R 2...R m is an optima partition of a woroad R of independent tass obtained from Theorem 4, where the time constraint is T (2) E E 2...E m is the corresponding energy consumption of the m computers (3) E is the resuting minimized energy consumption (4) p p 2...p m are the power settings of the m computers (5) s s 2...s m are the speed settings of the m computers. For the optima time aocation and woroad partition probem with R R 2...R v and T, where R þ R 2 þþr v ¼ R, we have the foowing facts. ) T, R, E, and E are ineary proportiona to R, i.e., they can be obtained from T, R, E, and E by scaing a factor of R =ðr þ R 2 þþr v Þ, for a v. 2) p ¼ p 2 ¼¼p v ¼ p. 3) s ¼ s 2 ¼¼s v ¼ s. Proof. The proof foows the same argument as that of Theorem 5. The ey observation is the fact that the equations in Theorem 4 sti hod if E, the R s, and T are scaed by a common factor. tu 6.4 Performance Data In this section, we present simuation resuts to demonstrate the expected performance of the LL-MLS agorithm on m ¼ 7 heterogeneous computers with the same s in Section 5.2. Again, the soutions produced by the LL-MLS agorithm are compared with optima soutions. Notice that the optima schedue ength of tass in a dag with a given energy budget cannot be shorter than the optima schedue ength of the same set of tass with the same energy budget and with a precedence constraints removed. Simiary, the minimum energy consumption of tass in a dag with a given time deadine cannot be ess than the minimum energy consumption of the same set of tass with the same time deadine and with a precedence constraints removed. Hence, the ower bounds in Section 4 are aso appicabe to precedence constrained tass. In Tabe 7, we show simuation data for the expected performance bound B of the LL-MLS agorithm for the energyconstrained scheduing probem. We consider a dag with v eves. The number of tass in eve is 0, for a v. For each combination of v ¼ and E ¼ , we generate 2,000 sets of n ¼ 5vðv þ Þ tass, whose tas execution requirements are random numbers uniformy distributed in ½0 Š. For each set of tass, we cacuate the ower bound T of T OPT by using Theorem 3. We aso simuate the LL-MLS agorithm and find its schedue ength T LL-MLS. The performance bound is obtained as B ¼ T LL-MLS=T. The average vaue of the 2,000 vaues of B is then reported in Tabe 7. The 99 percent confidence interva of a the data in Tabe 7 is no more than 0:459 percent. In Tabe 8, we show simuation data for the expected performance bound C of the LL-MLS agorithm for the timeconstrained scheduing probem. We consider the same dag as that in Tabe 7. For each combination of v ¼ and T ¼ 0:45t 0:40t 0:35t 0:30t 0:25t, where t ¼ v 2, we generate 2,000 sets of n tass, whose tas execution requirements are random numbers uniformy distributed in ½0 Š. For each set of tass, we cacuate the ower bound E of E OPT by using Theorem 4. We aso simuate the LL-MLS agorithm and find its energy consumption E LL-MLS. The performance bound is obtained as C ¼ E LL-MLS=E. The average vaue of the 2,000 vaues of C is then reported in

12 8 IEEE TRANSACTIONS ON SUSTAINABLE COMPUTING, VOL., NO., JANUARY-JUNE 206 Tabe 8. The 99 percent confidence interva of a the data in Tabe 8 is no more than :809 percent. From Tabe 7, we observe that B is very cose to. For a given v, the expected performance bound B sighty increases as E increases, primariy due to the ower bounds. For a given E, the expected performance bound B decreases as v increases. From Tabe 8, we observe that C is reasonaby cose to, except for sma vaues of v. For a given v, the expected performance bound C increases sighty (except for sma v) as T decreases, primariy due to the ower bounds. For a given T, the expected performance bound C decreases as v increases. Notice that a the data in Tabes 7 and 8 are for expected performance bounds. The actua vaues of expected performance ratios are smaer and coser to the optima. Our simuation resuts demonstrate that the LL-MLS agorithm has exceent expected performance in soving the probems of energy-constrained and time-constrained scheduing of precedence constrained tass on mutipe heterogeneous computers. 7 SUMMARY AND FURTHER RESEARCH We have addressed the probems of energy-constrained and time-constrained scheduing of independent or precedence constrained tass on mutipe heterogeneous computers as combinatoria optimization probems. Four ey techniques have been deveoped in this paper. First, we show how to find an optima energy or time aocation to eves of a dag, such that the performance of the eve-by-eve scheduing method can be optimized (Theorems 5 and 6). Second, for independent tass in the same eve, we show how to find an optima woroad partition, such that the modified ist scheduing agorithm can be empoyed to find an approximate tas schedue (Theorems 3 and 4). Third, for a given schedue, we show how to find an optima power aocation, such that the tota execution time or the tota energy consumption is minimized (Theorems and 2). Fourth, we derive ower bounds for the optima schedue ength and the optima energy consumption, so that soutions produced by our heuristic agorithms can be compared with optima soutions (Theorems 3 and 4). Our extensive simuation resuts demonstrate that by using the above techniques, the MLS agorithm has exceent and cose-to-optima expected performance in soving the probems of energy-constrained and time-constrained scheduing of independent tass on mutipe heterogeneous computers, and that the LL-MLS agorithm has exceent and cose-to-optima expected performance in soving the probems of energy-constrained and time-constrained scheduing of precedence constrained tass on mutipe heterogeneous computers. Our research in this paper reveas that energy-efficient tas scheduing on heterogeneous computers can be studied anayticay, ust as what we did for homogeneous computers. To the best of our nowedge, there has been no such method and resut in the existing iterature. Our investigation in this paper has made significant initia effort and our advanced methodsanddeepresutsinthispaperarethecurrentstate-ofthe-art. Furthermore, this paper shoud inspire further research interest. To concude the paper, we woud ie to mention severa further research directions. First, it has been observed that power consumption is dependent on specific types of appications [0]. It is a chaenge to incorporate such dependency into energy-efficient tas scheduing. Second, our study in this paper can be extended to more sophisticated power consumption modes, such as discrete and bounded coc frequency and suppy votage and execution speed and power suppy eves, as what we have done for homogeneous computers [20]. Third, in a way simiar to that of most existing studies, our research in this paper is primariy anaytica and experimenta. It needs further effort to appy such anaytica and simuation resuts to actua appications in reaistic systems. Due to space imitation, further investigation in these directions is beyond the scope of the paper and deserves separate papers. ACKNOWLEDGMENTS The author deepy appreciates four anonymous reviewers whose criticism and comments hep to improve the presentation of the manuscript. REFERENCES [] [Onine]. Avaiabe: heterogeneous-computing/what-is-heterogeneous-systemarchitecture-hsa, Accessed on 9 Nov [2] [Onine]. Avaiabe: power_ss2008.htm, Accessed on 9 Nov [3] S. Abers, Energy-efficient agorithms, Commun. ACM, vo. 53, no. 5, pp , 200. [4] A. Beogazov, R. Buyya, Y. C. Lee, and A. Zomaya, A taxonomy and survey of energy-efficient data centers and coud computing systems, Advances Comput., vo. 82, pp. 47, 20. [5] R. L. Burden, J. D. Faires, and A. C. Reynods, Numerica Anaysis, 2nd ed. Boston, MA, USA: Prinde, Weber & Schmidt, 98. [6] J. Cao, K. Li, and I. Stomenovic, Optima power aocation and oad distribution for mutipe heterogeneous muticore server processors across couds and data centers, IEEE Trans. Comput., vo. 63, no., pp , Jan [7] A. P. Chandraasan, S. Sheng, and R. W. Brodersen, Low-power CMOS digita design, IEEE J. Soid-State Circuits, vo. 27, no. 4, pp , Apr [8] S. Crago, et a., Heterogeneous coud computing, in Proc. IEEE Int. Conf. Custer Comput., 20, pp [9] S. P. Crago and J. P. Waters, Heterogeneous coud computing: The way forward, IEEE Comput.,vo.48,no.,pp.59 6,Jan.205. [0] H. Esmaeizadeh, T. Cao, X. Yang, S. M. Bacburn, and K. S. McKiney, Looing bac and ooing forward: Power, performance, and upheava, Commun. ACM, vo. 55, no. 7, pp. 05 4, 202. [] S. Garg, S. Sundaram, and H. D. Pate, Robust heterogeneous data center design: A principed approach, ACM SIGMETRICS Perform. Eva. Rev., vo. 39, no. 3, pp , Dec. 20. [2] R. L. Graham, Bounds on mutiprocessing timing anomaies, SIAM J. App. Math., vo. 2, pp , 969. [3] R. Iyer, et a., CogniServe: Heterogeneous server architecture for arge-scae recognition, IEEE Micro, vo. 3, no. 3, pp. 20 3, May/Jun. 20. [4] K. Li, Performance anaysis of power-aware tas scheduing agorithms on mutiprocessor computers with dynamic votage and speed, IEEE Trans. Parae Distrib. Syst., vo. 9, no., pp , Nov

13 LI: ENERGY-EFFICIENT TASK SCHEDULING ON MULTIPLE HETEROGENEOUS COMPUTERS: ALGORITHMS, ANALYSIS, AND... 9 [5] K. Li, Energy efficient scheduing of parae tass on mutiprocessor computers, J. Supercomputing, vo. 60, no. 2, pp , 202. [6] K. Li, Scheduing precedence constrained tass with reduced processor energy on mutiprocessor computers, IEEE Trans. Comput., vo. 6, no. 2, pp , Dec [7] K. Li, Optima power aocation among mutipe heterogeneous servers in a data center, Sustain. Comput.: Informat. Syst., vo. 2, no., pp. 3 22, 202. [8] K. Li, Energy-efficient and high-performance processing of argescae parae appications in data centers, in Data Centers, S. U. Khan and A. Y. Zomaya, Eds. Berin, Germany: Springer, 205, pp. 33. [9] K. Li, Power and performance management for parae computations in couds and data centers, J. Comput. Syst. Sci., vo. 82, pp , 206. [20] K. Li, Energy and time constrained tas scheduing on mutiprocessor computers with discrete speed eves, J. Parae Distrib. Comput., vo. 95, pp. 5 28, 206. [2] K. Li, Improving muticore server performance and reducing energy consumption by woroad dependent dynamic power management, IEEE Trans. Coud Comput., vo. 4, no. 2, pp , Apr./Jun [22] K. Li, X. Tang, and K. Li, Energy-efficient stochastic tas scheduing on heterogeneous computing systems, IEEE Trans. Parae Distrib. Syst., vo. 25, no., pp , Nov [23] S. U. R. Mai, K. Bia, S. U. Khan, B. Veeravai, K. Li, and A. Y. Zomaya, Modeing and anaysis of the therma properties exhibited by cyber physica data centers, IEEE Syst. J., 206, doi: 0.09/JSYST , in press, 207. [24] J. Mateo, J. Viapana, L. M. Pa, J. L. Lerida, and F. Sosona, A green strategy for federated and heterogeneous couds with communicating woroads, Sci. Word J., Nov., 204. [Onine]. Avaiabe: artices/pmc / [25] J. Mei, K. Li, and K. Li, Energy-aware tas scheduing in heterogeneous computing environments, Custer Comput., vo. 7, no. 2, pp , 204. [26] H. S. Narman, M. S. Hossain, and M. Atiquzzaman, h-ddss: Heterogeneous dynamic dedicated servers scheduing in coud computing, in Proc. IEEE Int. Conf. Commun., 204, pp [27] V. Petrucci, E. V. Carrera, O. Loques, J. C. B. Leite, and D. Mosse, Optimized management of power and performance for virtuaized heterogeneous server custers, in Proc. th IEEE/ACM Int. Symp. Custer Coud Grid Comput., 20, pp [28] C. Reiss, A. Tumanov, G. R. Ganger, R. H. Katz, and M. A. Kozuch, Heterogeneity and dynamicity of couds at scae: Googe trace anaysis, in Proc. 3rd ACM Symp. Coud Comput., 202, Art. no. 7. [29] D. Sun, G. Zhang, S. Yang, W. Zheng, S. U. Khan, and K. Li, Re- Stream: Rea-time and energy-efficient resource scheduing in big data stream computing environments, Inf. Sci., vo. 39, pp. 92 2, 205. [30] Z. Tang, L. Qi, Z. Cheng, K. Li, S. U. Khan, and K. Li, An energyefficient tas scheduing agorithm in DVFS-enabed coud environment, J. Grid Comput., vo. 4, no., pp , 206. [3] Y. Tian, C. Lin, and K. Li, Managing performance and power consumption tradeoff for mutipe heterogeneous servers in coud computing, Custer Comput., vo. 7, no. 3, pp , 204. [32] G. L. Vaentini, et a., An overview of energy efficiency techniques in custer computing systems, Custer Comput., vo. 6, no., pp. 3 5, 203. [33] L. Van Doorn, Heterogeneous server architectures wi dominate the future data center, Open Server Summit, Santa Cara, CA, USA, Nov. 3, 204. [Onine]. Avaiabe: [34] W. Wang, B. Liang, and B. Li, Muti-Resource fair aocation in heterogeneous coud computing systems, IEEE Trans. Parae Distrib. Syst., vo. 26, no. 0, pp , Oct [35] Y. Wang, K. Li, H. Chen, L. He, and K. Li, Energy-aware data aocation and tas scheduing on heterogeneous mutiprocessor systems with time constraints, IEEE Trans. Emerg. Topics Comput., vo. 2, no. 2, pp , Jun [36] B. Zhai, D. Baauw, D. Syvester, and K. Fautner, Theoretica and practica imits of dynamic votage scaing, in Proc. 4st Des. Autom. Conf., 2004, pp [37] L. Zhang, K. Li, Y. Xu, J. Mei, F. Zhang, and K. Li, Maximizing reiabiity with energy conservation for parae tas scheduing in a heterogeneous custer, Inf. Sci., vo. 39, pp. 3 3, 205. [38] L. M. Zhang, K. Li, D. C.-T. Lo, and Y. Zhang, Energy-efficient tas scheduing agorithms on heterogeneous computers with continuous and discrete speeds, Sustain. Comput.: Informat. Syst., vo. 3, no. 2, pp. 09 8, 203. [39] S. Zhuravev, J. C. Saez, S. Bagodurov, A. Fedorova, and M. Prieto, Survey of energy-cognizant scheduing techniques, IEEE Trans. Parae Distrib. Syst., vo. 24, no. 7, pp , Ju Keqin Li is a SUNY distinguished professor of computer science. His current research interests incude parae computing and high-performance computing, distributed computing, energy-efficient computing and communication, heterogeneous computing systems, coud computing, big data computing, CPU-GPU hybrid and cooperative computing, muticore computing, storage and fie systems, wireess communication networs, sensor networs, and peer-to-peer fie sharing systems. He has more than 460 pubications. He is currenty or has served on the editoria boards of the IEEE Transactions on Parae and Distributed Systems, the IEEE Transactions on Computers, the IEEE Transactions on Coud Computing, and the IEEE Transactions on Services Computing. He is a feow of the IEEE. " For more information on this or any other computing topic, pease visit our Digita Library at