End-to-End Delay Minimization in Thermally Constrained Distributed Systems

Transcription

1 End-to-End Delay Minimization in Thermally Constrained Distributed Systems Pratyush Kumar, Lothar Thiele Comuter Engineering and Networks Laboratory (TIK) ETH Zürich, Switzerland {ratyush.kumar, Abstract With ever-increasing ower densities, managing on-chi temeratures by otimizing maing and scheduling of tasks is becoming increasingly necessary. We study the minimization of end-to-end delay for thermally constrained scheduling of an alication that is secified as a task grah and is executing on arallel rocessors without seed scaling. We show that task grah scheduling on thermally constrained systems is monotonic, i.e., delaying the execution of a task longer than necessary cannot lead to the early comletion of any other task. Using this monotonicity rincile, we design the rovably otimal schedule for a given maing, called the JUST schedule. The JUST schedule can be easily imlemented using temerature sensors. We then resent different thermalaware modifications to standard maing heuristics and evaluate them on a large set of roblem instances. The exerimental results illustrate that with simle thermal-aware modifications, maings with much smaller end-to-end delay can be identified. I. INTRODUCTION With the inevitable, though slower, transistor scaling, the ower consumtion of today s electronic systems continues to rise. Higher ower densities directly translate to higher on-chi temeratures, which can imact the long-term reliability and also the functional correctness of the system [1]. Furthermore, with transistors becoming increasingly leaky, higher temeratures in turn lead to higher ower [2]. Imrovements in hardware-based cooling using better ackaging materials, more efficient active cooling techniques and novel solutions like liquid cooling, are not able to kee ace with the rise in temeratures [3]. As a result, softwarebased methods broadly classified as Dynamic Thermal Management (DTM) are becoming increasingly mainstream [4]. In modern rocessors there are two major sources of ower consumtion: switching activity in the caacitors leading to dynamic ower consumtion and ower dissiation between voltage rails leading to leakage ower consumtion. Dynamic ower consumtion can be reduced by scaling the voltage and/or frequency of the system to match erformance requirements: a technique referred to as Dynamic Voltage Scaling (DVS). Leakage ower consumtion can be reduced by utting the system into a low ower state, whenever the system is not exected to erform any comutation: a technique referred to as Dynamic Power Management (DPM) or execution throttling. The objective of thermally constrained scheduling, is to use DVS and DPM techniques to otimize the system erformance and/or to meet real-time deadlines while satisfying certain secified thermal constraints. Several works have formulated and studied such roblems. In [5], the authors derived the maximum amount of workload that a rocessor can serve using on-line voltage scaling while satisfying thermal constraints. In [6], reactive seed control to schedule framebased real-time tasks was roosed and schedulability tests under temerature constraints were derived. Proactive seed control to minimize resonse time of set of indeendent tasks under temerature constraints has been studied in [7]. In [8], comletion time of tasks is minimized while each task is assigned a fixed seed. There are two limitations with the above and similar works. Firstly, they only consider DVS for thermal-aware scheduling. While DVS is an attractive roosition in theory, there are ractical requirements it assumes: availability of different system-wide voltage rails, and ossibility of switching between these voltage rails at any given oint of time. Further, as VLSI technology continues to scale, and as thermal issues become more imortant, the headroom for voltage scaling is unfortunately shrinking. Finally, eriheral devices such as I/O controllers, memory modules and network devices such as radio, intrinsically do not suort seed scaling. For such devices, alying DPM techniques by utting the system in low-ower modes at times of low activity is more effective. The second limitation of these works is that they only consider indeendent tasks with individual erformance constraints. In reality, alications comrise of several deendent tasks, modeled usually as task grahs, which need to satisfy a cumulative erformance constraint such as the endto-end delay. In such alications, one task can be slowed down or delayed to seed u or run earlier another task, with no net effect on the end-to-end delay. The scheduling freedom in such roblems is thus otentially larger and this broadens the scoe of thermally constrained scheduling. Some existing works have studied the roblem of thermalaware scheduling of task grahs. In [9], authors study the roblem of binding and scheduling tasks of a task grah

2 on multi-core systems with secified thermal-constraints. In [10], the authors roose heuristic aroaches to distribute slack time across different tasks of a task grah, again for a multi-core architecture. These works consider heuristics based only on the steady-state behavior of the system. This simlifying assumtion is not valid when considering distributed comuting systems with non-negligible communication overhead, or for alications where execution times of tasks is much shorter than the thermal time-constants. Interconnecting heterogeneous comuting resources over communication networks is increasingly becoming a common way of comuting: examles range from networked embedded systems erforming co-ordinated tasks to higherformance grid comuters. Reresenting alications as task grahs not only exresses the data deendencies of the tasks, but also exoses the inherent concurrency in the alications, that can be exloited in such arallel systems. The roblem of thermally constrained scheduling of task grahs in such systems, is to minimize the end-to-end delay of the alication while satisfying secified thermal constraints on each of the rocessors. We consider rocessors with no seed scaling feature and hence the only way to manage temeratures is to force the rocessors in to low-ower states. In addition to such forced oeration, the rocessor would remain in the low-ower state when it does not have any tasks to run, either due to low concurrency in the alication or because inut data is yet to arrive. Thus, both the arallel and distributed natures of such alications and architectures have to be carefully considered in thermally constrained scheduling. Our contributions in this work are on three fronts. Firstly, we show that task grah scheduling on thermally constraint systems is monotonic, i.e., delaying the execution of a task more than is necessary to meet the thermal constraint, cannot lead to early comletion of any other task. Using this desirable roerty, for a given maing of tasks to the rocessors, we derive the rovably otimal JUST scheduling strategy, that minimizes the end-to-end delay of the alication while satisfying the thermal constraints on each of the rocessors. We resent how the otimal scheduling can be easily imlemented using temerature sensors on the rocessors. The second contribution is to study heuristics to otimize the maing of the tasks. The nature of the otimal JUST schedule motivates us to encode the thermal constraints as art of the task grah and then aly the well-studied maing heuristics, to decide the maing of the tasks. We comare the end-to-end delays of JUST schedules for maings generated with the original task grah and the modified thermal-aware task grah with the HEFT heuristic roosed in [11]. We resent results on realistic as well as a large number of randomly generated task grahs. Our results illustrate that thermal-aware maing and scheduling of tasks can substantially reduce the end-to-end delay of alications, in comarison with a thermal-unaware aroach, even for the otimal scheduling olicy. We also characterize the nature of the task grahs and system architectures where thermal-awareness is more critical. Finally, as the third contribution, we design more sohisticated thermal-aware maing strategies by exlicitly modeling the thermal behavior of the system in the maing hase. and comare them with the simle maing strategy based on the modification of the task grah. From exerimental results we conclude that the sohisticated methods that we roose rovide only a small imrovement at a substantial run-time overhead. The rest of the aer is organized thus. In Section 2, we formally resent the model of the system and the roblem definition. As the main contribution of this work, we resent the otimal scheduling strategy, for a given maing, in Section 3. In Section 4, we resent methods to otimize the maing of tasks. We resent results of the exeriments in Section 5 and conclude in Section 6. II. MODELS AND PROBLEM DEFINITION A. Alication and system models An alication is reresented by its task grah, also called a directed acyclic grah (DAG), G = (V, E), where V is the set of tasks and E is the set of directed edges between the tasks. Each edge (u, v) E reresents the recedence constraint that before task v begins to execute, required inut from task u must be available. The system that we consider consists of a set, R, of ossibly heterogeneous rocessors. The frequency of oeration of rocessor R is denoted as f. We consider a fully connected interconnect where communication on links can haen arallel to the comutation on rocessors. The task v has a worst-case execution time (WCET) on rocessor given by c v,. The communication delay between tasks u and v, when maed on rocessors and q, resectively, is given by q (u,v),,q. Thus, the alication and the system are secified by the tule A = (G, R, c, q). B. Thermal model We consider a distributed network of heterogeneous rocessors. Each rocessor can have a different thermal model based on the ower consumtion of the rocessor, the cooling solution emloyed and the ambient temerature around that rocessor. We do not consider any heat exchange between the rocessors, i.e. the rocessors are thermally insulated from each other. In all the notation used below, we consider a secific rocessor R and denote this by using as a subscrit in all arameters. The hysical rocess of heat transfer is a comlex one. However, for architectural level thermal management, the Fourier heat transfer model is tyically used [5], [6], [8]. Under this model, the evolution over time

3 of temerature of a rocessor is given by the following differential equation: C dt dt = P G (T T amb ), (1) where T amb is the ambient temerature around the rocessor, G reresents the thermal conductance of the cooling solution, P reresents the ower being consumed in the system, C reresents the thermal heat caacity of the rocessor and T (t) is the temerature of the rocessor. The leakage current on a VLSI system is itself a function of temerature: at higher temerature the leakage current is higher. In [12], a linear model is shown to work well in ractice. In this model, the ower can be exressed as the following function of rocessor-secific arameters α, β P = α T + β. (2) Later, in Section 3.2, we consider a more general ower model where tasks executing on the same rocessor can consume different amounts of ower. Including (2), the differential equation in (1) becomes C dt dt = (G α )T + (β + G T amb ). (3) To simlify notation we shorten the above equation as dt dt = a T + b, with a = G α C, b = β + G T amb. (4) C The closed form solution to this differential equation is T (t) = T + (T (t 0 ) T ) e a(t t0), (5) where T = b /a is the steady-state temerature of the rocessor. From (5), the thermal behavior of the rocessor is characterized by the time constant a and the steady-state temerature T. When a rocessor is executing tasks it is said to be in the active mode, otherwise it is said to be in the idle mode. Intuitively, the ower consumtion of the rocessor is different in either mode. We reresent the corresonding α and β terms, as exressed in (2), as α act, β act for the active mode and α idl and β idl for the idle mode. Similarly, we reresent the derived time constant a for the active and idle modes as a act and a idl, resectively. While reresenting T for the two modes, to aid readability, we dro the symbol and denote them simly as T act and T idl. For a valid thermal mode, we should have T act T idl. Using these derived quantities, the thermal behavior of the system is secified by the tule T = (a act, T act, a idl, T idl ). C. Maing and Scheduling Model We consider a artitioned static-ordered non-reemtible scheduling of a given task grah. For artitioned scheduling there exists a binding function π where π v denotes the rocessor on which task v is always executed on. The schedule is static-ordered: the tasks bound to a rocessor are executed according to a fixed ordering σ, where σ,i denotes the ith task to execute on rocessor. Partitioned static-ordered scheduling is a standard aroach to schedule task grahs [13]. We denote by σ the total number of tasks maed on rocessor. The scheduling is nonreemtible in nature, i.e., once a task begins to execute, it cannot be reemted. Preemtion by other tasks is clearly not allowed under static-ordered scheduling. Preemtion by utting the rocessor in the idle mode is also disallowed as such reemtion would require an additional feature: the rocessor must be able to reserve its context in the idle mode and be able to resume its execution after the transition from idle to active mode. We do not make this assumtion, and hence the rocessor can be in idle mode only in-between the execution of tasks. To control the temerature of the rocessors we use dynamic ower management (DPM), whereby, we allow insertion of idle times before execution of each task. The value γ,i denotes the minimum amount of time rocessor remains in idle mode before the execution of the task σ,i. The maing and scheduling of the task grah is thus fully secified by the tule S = (π, σ, γ). We also refer to the binding and ordering of the tasks as given by the tule M = (π, σ) as the maing of tasks. D. Problem Definition For each rocessor R, a thermal constraint secifies the maximum allowed temerature of that rocessor, denoted as T max. A thermal constraint is valid if T max T act, i.e., the maximum allowed temerature should not be above the maximum temerature that can be reached by the rocessor. We assume that tasks can begin execution in each of the rocessors at time t = 0. The initial temerature, at t = 0, of each rocessor R is given as T start. The end-to-end delay, d, is defined as the largest finish times among all tasks of the alication. The thermally constrained task grah scheduling roblem can then be defined as follows: Given an alication and system secification A and a thermal model T. Design maing and scheduling S such that for all rocessors R the temerature on rocessor at all times does not exceed T max, and the end-to-end delay, d, is minimized. III. OPTIMAL END-TO-END DELAY FOR A GIVEN MAPPING OF TASKS We solve the roblem of identifying the otimal maing and scheduling in two stes. Firstly, in this section, we

4 assume a given maing, M, with binding π and staticordering σ. For this maing, we comute the idle times γ to otimally minimize the end-to-end delay, while satisfying the thermal constraints. We then, in the next section, consider the roblem of otimizing the maing of the tasks. We resent some notation to characterize a schedule. Let t s,i and T,i s denote the time and temerature of rocessor, resectively, when the task σ,i starts to execute. Similarly, let t f,i and T f,i denote the time and temerature of rocessor, resectively, when task σ,i finishes. When discussing multile schedules, we label the above variables with a suerscrit denoting the schedule. For each rocessor R, tasks can start at time t = 0. To enable ease of notation later, we reresent this as the finish time of an imaginary 0th task, t f,0 := 0, for all R. Similarly, we set the initial temerature of each rocessor T f start,0 := T. The insertion of idle times, γ, before the execution of tasks gives the following relations on the start times t s,i t f,i 1 + γ,i, R, 1 i σ. (6) The γ terms only define a lower bound on the amount of time the rocessor is in the idle mode between tasks; a rocessor may continue to remain in the idle mode for longer, if the inut data required by the next task is not yet available. For a given maing of tasks, the end-to-end delay, d is given in terms of the finish times as d = max R,i= σ tf,i. (7) Given (6), the value of d is a function of the inserted idle times γ. The otimal end-to-end delay, denoted as d ot, is defined as the smallest end-to-end delay over all choices of idle times γ. The goal of this section is to comute this otimal assignment of idle times. A. Worst-case Execution Pattern The otimal end-to-end delay, d ot, is a function of the execution times of tasks. In the alication secification, tasks are characterized only by their WCETs, leading to an uncertainty in the execution attern. In order to rovide real-time guarantees which hold irresective of the actual execution times of tasks, we must ascertain the combination of execution times of tasks that leads to the largest d ot. For systems with no thermal considerations, clearly, the longest end-to-end delay is obtained when the all task executions take as long as ossible. But does this hold for thermally-constrained systems as well? The following theorem answers this question in the affirmative. Theorem 1: For a given maing of tasks, d ot is maximum when all tasks run for their WCETs. Proof: For the given maing of tasks, let S be the otimal schedule with resect to end-to-end delay, satisfying thermal constraints, when all tasks exactly execute for their WCETs. For the same maing of tasks, let S be a schedule, where tasks run for secified times, no more than their WCETs. From the monotonicity of dataflow, we know that an earlier execution of a task cannot lead to a later finish of any other task. This imlies that in schedule S, all tasks can be started at the same time as they are started in schedule S. Let S indeed be such that (t s,i )S = (t s,i )S. Since all tasks run for their WCETs in S, we have (t f,i )S (t f,i )S, and thus d S d S. We have to now establish that S satisfies thermal constraints. For each task of the task grah, in schedule S the corresonding rocessor satisfies two roerties: (a) it remains in the idle mode, rior to the start of the task, for at least as long as in S, and (b) it runs in the active mode for at most as long in S. Using T act T idl, and by reeatedly alying (5) with the above two roerties, we can show that (T f,i )S (T f,i )S, for all R and 1 i σ. Hence, like in S, thermal constraints are not violated in S, and S is a valid schedule. Thus, we have d S = d ot (tasks run for WCETs) d S d ot (tasks run for arbitrary amounts of time uer-bounded by WCETs). (8) From this theorem, we have the following conclusion: irresective of the actual execution times of tasks, under the otimal scheduling olicy, we can always guarantee a makesan constraint of d ot (tasks run for WCETs). In the remainder of this aer, we will assume all tasks run for their WCETs, so as to be able to comute the makesan guarantee and use it to comare different maings. B. JUST Schedule Define T target,i as the value of T,i s for which, given that task σ,i runs for its WCET, the corresonding value of T f max,i is T. In other words, T target,i denotes the maximum allowed temerature of rocessor at the start of execution of task σ,i, without violating the temerature constraint. To simlify notation, let c denote the WCET of the considered task σ,i on rocessor, i.e. c = c σ,i,. By alying (5), we can statically derive the following relation = T act ( T act T max ) e a act c. (9) T target,i Again alying (5), given the value of T f,i 1, we can comute a lower bound on the idle time γ,i such that T,i s T target.i as ( ) γ,i min T f,i 1 = 0, if T f,i 1 T target,i ( = 1 T f,i 1 a idl log T ) idl, else. (10) T idl T target,i

5 The interretation of γ min is as follows: if the idle time before the execution of task σ,i is less than γ,i min, then the thermal constraint of rocessor would be violated by the end of the execution of task σ,i. Thus, any schedule that satisfies the secified thermal constraints must kee the rocessor in the idle mode for at least as long as given by the above γ min values. We now define the schedule which we shall, later in this section, rove to be the otimal schedule. Definition 1 (JUST(π, σ)): For a given binding π and for a given ordering σ, the JUst Sufficient Throttling JUST(π, σ) schedule is one where all idle times, γ, are set to γ min. In a JUST schedule the rocessors are ut in the idle mode for just sufficient amount of time to avoid violating thermal constraints, and hence the name. Note that the JUST schedule is not statically defined as the values of γ min deend on the temeratures T f, which are revealed only at run-time. However, the comutation of the idle times is causal in nature: once a task comletes, the length of the idle time to be inserted before the next task begins can be comuted. From the definition of the JUST schedule, we have the following result. Lemma 2: For a given maing of tasks, in any schedule that does not violate thermal constraints, if executing a single task earlier does not violate thermal constraints at the end of its execution, then it will not violate thermal constraints during the entire schedule. Proof: For given maing, let S be a given valid schedule. For the same maing, let S be another given schedule that varies from S only with an earlier start time for some task σ,i, i.e. (ts,i < (t s )S,i )S and for all other tasks σ,i, (, i) (, i ), (t s,i = (t s )S,i )S. The given value of (t s,i is such that task σ )S,i satisfies all recedence constraints and the thermal constraint on rocessor at the end of its execution. Again from the monotonicity rincile, we know that S would satisfy recedence constraints on all tasks. We need to show that S also satisfies thermal constraints. The start and finish of all tasks on all rocessors, is the same as in S, and hence the temerature constraints are satisfied on these rocessors. Similarly, on rocessor, the thermal constraints are satisfied until the start of task σ,i. It is also given that the thermal constraints are satisfied at the end of the execution of task σ,i. We thus need to show that S satisfies thermal constraints on rocessor from the beginning of the execution of task σ,i +1. During this time, the tasks start and run for exactly the same time in both S and S. Notice that (5) is monotonic with resect to the starting temerature of the rocessor. Therefore, the comosition of (5) for the different tasks and idle times from the beginning of execution of task σ,i +1 is also monotonic in the starting temerature, T s,i +1. Thus, it is sufficient to show that (T s,i +1 )S (T s,i +1 )S. To show the above, we first derive an analytical result. Consider a rocessor that is at temerature T 1. It is ut in the idle mode for x units of time, then in the active mode for y units of time and finally in the idle mode again for z units of time. Let the temerature of the rocessor at the end of the three hases be T 2. Then using (5) reeatedly we derive the following relation: T 2 = T idl + (T 1 T idl )e aidl (x+z) e aact y + (T act T idl )(1 e aacty )e aidlz. (11) We have droed the subscrit denoting the rocessor. If we changed x to x δ and z to z + δ, for some δ > 0, the value of T 2 decreases, for a given value of T 1. This can be interreted as follows: For a given total idle time, (x + z), the final temerature, T 2, can be reduced by shifting some of the earlier idle time (x) to a the later idle time (z). We can aly (11) searately to the two schedules S and S, on the rocessor, where T 1 = T f,i 1, T 2 = T s,i +1, x = ts,i tf,i 1, y = c σ,i,, and z = t s,i +1 tf,i. In S, the value of x is reduced than in S as σ,i is started earlier than in S, whereas the value of z is increased by the same amount. Hence, using the above result, we have (T s,i +1 )S (T s,i +1 )S. The above lemma imlies that modifying any valid schedule by executing a task earlier entails only local checks: whether the inut data is available at the earlier starting time and whether the thermal constraints are satisfied after the execution of the task. If these local checks hold true, the entire modified schedule is a valid one. We use this result to derive the central theorem of this work. Theorem 3: For a given binding π and ordering σ of tasks on rocessors, the JUST(π, σ) schedule otimally minimizes the end-to-end delay while satisfying the thermal constraints. Proof: For binding π and ordering σ, let J denote the JUST(π, σ) schedule. For the same maing, let S denote any other schedule, that satisfies the thermal constraints. We are to show that d J d S. Let µ be a sorting of the tasks of the task grah in nondecreasing order of their start times in the JUST schedule, (t s ) J, with ties broken randomly. The ith task, for 1 i V, in the order µ is denoted as µ i. Let i be the smallest number such that task µ i has different start times in schedules J and S. The start time of task µ i in J cannot be later than in S as it would contradict the definition of the JUST schedule. Hence, the start time of task µ i is greater in S than in J. We modify schedule S such that task µ i starts at the same time as in J, while the starting times of other tasks are ket the same. Lemma 2 shows that this modification leads to a valid schedule.

6 We erform the above rocess iteratively, modifying the start time of one task in S in each iteration. The number of tasks for which the start times differ in S and J reduces by 1 in each iteration. Hence, after a finite number of iterations, S would be transformed to J. Since in each iteration we are executing tasks in S earlier, the end-to-end delay cannot increase. Thus, any given schedule can be transformed to the JUST schedule iteratively without increasing the end-to-end delay in any iteration. As discussed earlier, dataflow grahs are monotonic: executing any task later than the time of arrival of all its inuts, cannot lead to an earlier finish time of any task. With the above theorem, this monotonicity can be extended for thermally constrained dataflow scheduling as follows. Definition 2: (MONOTONICITY IN THERMALLY CON- STRAINED DATAFLOW SCHEDULING) Executing any task later than the time of arrival of all its inuts and the time of the rocessor cooling to corresonding T target as given in (9), cannot lead to an earlier finish time of any task. This motivates modeling the idling of the rocessor by dummy tasks in the task grah: dummy task ρ,i is inserted between σ,i 1 and σ,i. The task ρ,i is said to comlete execution (or fire in dataflow vocabulary) when the temerature of the rocessor falls below T target,i. These dummy tasks can be easily imlemented in a real system by using temerature sensors on each of the rocessors. Once the temerature of the system falls below the T target, as given in (9), the temerature sensor can rovide an enable signal to start the execution of the next task; in some sense roviding a re-rogrammed temerature interrut to enable the next task. The JUST schedule then is simly the as-soonas-ossible (asa) scheduling of this modified task grah. Indeed, the desirable monotonicity rincile allows for this simle imlementation. We have modeled our system such that the ower consumtion of a rocessor, as given in (2), is indeendent of the running task. In ractice, different tasks use different amounts of ower. The roosed JUST scheduling can model such task-deendent ower consumtion. To this end, we can modify the comutation of T target in (9) by obtaining the arameters T act and a act secific to the next task that is to be executed. The thermal sensor, as before, can wait for the temerature to fall below this T target and then enable the next task. This also means that the monotonicity roerty of Definition 2 holds for systems with task-deendent ower, if T target is aroriately comuted. IV. OPTIMIZING MAPPING OF TASKS In the revious section we showed how to construct the otimal schedule with resect to end-to-end delay given the maing of tasks. We now turn our attention to otimizing the maing of tasks, with resect to the end-to-end delay of the corresonding JUST schedules (a) Task grah Parameter Value f MHz f 2 50 MHz c, 5000/f ms d (, ),, 50ms T start 327K α idl 0.1 W/K β idl -11 W α act 0.1 (f /100) 2 W/K β act -25 (f /100) 2 W T amb 300 K T1 max 363 K T2 max 352 K G 0.3 W/K 0.03 JK C (b) Problem secification T1 max 360 P1 T2 max 345 P P P (c) Temerature (in K) vs Time (in ms) for maing M P P (d) Temerature (in K) vs Time (in ms) for maing M 2 Figure 1. A. Motivating examles FFT examle We start by illustrating, with two ractical examles of task grahs, the key considerations for maing of tasks in thermally constrained systems. In either examle, we consider two maings: a) maing M 1 generated by the HEFT algorithm [11] on the original task grah G, and b) maing M 2 that is generated by the same HEFT algorithm on a modified version of the task grah. Details of how these maings are generated are resented later in the aer. FFT: The first examle is an FFT alication characterized by high concurrency as shown in Figure 1(a). It is executing on a system of two different rocessors, with frequency in the ratio 2:1, and with a small communication delay between them. The system arameters are shown in Figure 1(b).

7 The maing M 1 mas two-thirds of the tasks on to the faster rocessor 1. For a schedule with no thermal constraint, M 1 leads to an otimal schedule with full utilization of the both rocessors for the duration of the schedule. On the other hand, M 2, in site of the difference in rocessor frequency, mas equal number of tasks to either rocessor. The JUST schedules corresonding to both maings are shown in Figures 1(c) and 1(d). The JUST schedule corresonding to M 2 has a much smaller end-to-end delay than that of M 1. As is clear from the figure, this haens because the tasks maed on to rocessor 1 require large idle times before they can begin, due to the higher ower consumtion of the fast rocessor. On the other hand, even for a smaller temerature bound, tasks running on the slower rocessor 2 do not need any idling time. This examle illustrates how a faster rocessor is not necessarily a good choice in a thermally constrained schedule. The enforced idle time changes the effective execution time of task on different rocessors to different extents. LU Decomosition: The second examle is an LU decomosition alication characterized by low concurrency as shown in Figure 2(a). Its executing on a system of two identical rocessors with a large communication delay between them with system arameters as shown in Figure 2(b). The individual rocessors considered in this examle have the same arameters as the faster rocessor in the last examle. In M 1 all tasks are maed on to rocessor 1, as the communication costs are found to be too large for maing tasks on rocessor 2. In M 2, desite the large communication costs, one of the tasks, τ 4 1, is maed to rocessor 2. As shown in Figures 2(c) and 2(d), JUST schedule corresonding to M 2 has a smaller end-to-end delay than that of M 1. This difference arises because communication along channels is allowed to haen arallel to a rocessor being in the idle mode. Thereby, the effective cost of communication can decrease. In the examle, the enforced idling of rocessor 1, was sufficient to amortize the communication delay incurred in rocessing task τ 4 1 on the far-away rocessor 2. More than an analysis challenge, this examle illustrates the otential to exloit this arallelism: allowing rocessors to idle while inut data is being transferred. B. Modification of execution and communication times The discussed examles show that thermal constraints translate to changes in both the effective execution times and communication latencies. Motivated by the monotonicity roerty of thermally constrained scheduling (Theorem 3), we would like to model the effective execution times and communication latencies as art of the alication task grah. To this end, we make two sets of modifications. Firstly, we change the WCET of tasks to model the idle time before the execution of that task. As shown in (10), the idle time inserted in the JUST schedule before a task, deends on the temerature of the rocessor at the end τ 1 1 τ 2 1 τ 3 1 τ 4 1 τ 2 2 τ 3 2 τ 4 2 τ 3 3 τ 4 4 (a) Task grah Parameter c,τ x y d (, ),, T start α β idl β act T amb T max G C Value (5 y) 10 if x = y (4 y) 10 if x y 100ms 327 K 0.1 W/K -11 W -25 W 300 K 363 K 0.3 W/K 0.03 JK (b) Problem secification T1 max =T2 max =363K 360 P1 P P1 τ 1 1 τ 2 1τ 3 1 τ 4 1 τ 2 2 τ 3 2 τ 4 2 τ 3 3 P2 (c) Temerature (in K) vs Time (in ms) for maing M τ P1 τ 1 1 τ 2 1τ 3 1 τ 3 3 τ 3 2 τ 3 3 τ 4 2 τ 4 3 P2 τ 4 1 (d) Temerature (in K) vs Time (in ms) for maing M 2 Figure 2. LU decomosition examle of the execution of the revious task. This temerature can be conservatively uer-bounded by assuming it to be the maximum allowed temerature on that rocessor,. Crucially, this conservative bound is a valid bound indeendent of the maing or the state of other rocessors. For a rocessor R and for a task v V, using (9) and (10), we can comute this conservative bound as T max γ,v max = 1 a idl log ( (T act T idl T max ) (T act T idl T max )e aact c,v ). (12)

8 We can then modify the execution times of the task v on each rocessor to include this conservative bound as c T,v := c,v + γ max,v. (13) The second change relates to the communication delays. As indicated in the LU decomosition examle, communication can haen in arallel to the idling of the rocessors. In light of the modification of the WCETs of tasks according to (13), we reduce the communication latencies as follows: d T (u,v),,q := max(0, d (u,v),,q γ max q,v ). (14) In the above equation, we reduce the communication latency along each channel by u to the inserted idle time of the destination task, but subject to no delay becoming negative. While the above modifications to the timing roerties of the task grah conservatively model thermally constrained scheduling, they are only aroximate. On the other hand, the modifications allow us to incororate the thermal constraints in the convenient task grah reresentation of the alication, whereby we can aly well-studied task-grahbased maing techniques. C. Maing using HEFT algorithm Minimizing the end-to-end delay of a task grah on a set of arallel rocessors has been an active area of research. Aart from two very simle cases, all such maing roblems are known to be NP-hard [13]. Heuristics are, thus, the common aroach in such roblems. For the roblem of task grah scheduling on a given set of heterogeneous rocessors, the maing generated by the Heterogeneous- Earliest-Finish-Time (HEFT) algorithm resented in [11] has been shown to erform well and with small comutation requirements. For comleteness, we resent a brief overview of the HEFT algorithm. Like other similar algorithms, there are two hases in the HEFT algorithm: a) a ranking hase wherein the tasks of a task grah are ordered based on a ranking function, and b) a maing hase, wherein tasks, in the derived ranking order, are assigned to be executed at secific times on chosen rocessors. The uward rank, as used in the HEFT algorithm, is defined recursively for every task v V as follows rank u (v) = c v + max (q v,w + rank u (w)), where w succ(v) c v = 1 c v,, and q v,w = 1 R R 2 q (v,w),,q.. R,q R (15) Here succ(v) denotes the set of tasks which are direct successors of the task v in the task grah G. In the maing hase, tasks ordered by non-increasing rank are maed sequentially. At all oints of time, a artial schedule is maintained of already maed tasks. A task is maed to that rocessor which leads to the smallest earliest finish time (EFT) based on the current artial schedule. Once the rocessor with the smallest EFT is chosen, the task is maed on to it, a secific time window for its execution is assigned, and the artial schedule is modified to include this window. This rocess is continued till all tasks are maed. A more detailed descrition is available in [11]. We generate two sets of maings with the HEFT algorithm. Firstly, we aly the HEFT algorithm directly to the task grah, G, with its original roerties. While the outut of the HEFT algorithm is an entire schedule, we only retain the maing of the tasks, i.e. the binding and the ordering of tasks. The maing thus obtained is denoted as M 1. Then, we modify the timing roerties (execution times of tasks and communication delays along channels) of the task grah as discussed in (13) and (14). We then again aly the HEFT algorithm to this modified thermalaware task grah. The maing so obtained is denoted as M 2. We then construct the otimal JUST schedules for the obtained maings considering the thermal constraints on each rocessor. We comare the maings, M 1 and M 2, by comaring the makesan of the resective JUST schedules. D. Thermal-Aware modifications to HEFT algorithm M 1 rovides a good baseline to comare M 2 to ascertain the need for thermal-aware maing otimization. However, it does not suffice to evaluate M 2 with resect to the otential of thermal-aware maing otimization. It is necessary to comare M 2 against more sohisticated thermal-aware maing strategies. To the best of our knowledge, there is no existing work in this direction. Thus, we roose techniques to generate two other maings denoted as M 3 and M 4. To generate these maings we exlicitly and more accurately model the thermal behavior of the rocessors during the maing otimization at the cost of added comlexity. In both M 3 and M 4, the ranking hase is the same as in the HEFT algorithm. However, the maing hases differ. While modifying the WCETs of the tasks in (13), we assumed that the temerature at the end of the execution of each task is the highest allowed temerature on that rocessor. This can be inaccurate for systems with large communication latencies or for task grahs with low concurrency. We change this in M 3 and M 4 by exlicitly comuting the temeratures of the rocessors for the artial schedules during the maing hase of HEFT algorithm using (5). We then derive the earliest finish time (EFT) of tasks on different rocessors by comuting the idle times accurately given the temerature of the rocessors using (10). Thus, at each ste of the maing, for the current artial schedules temerature behavior of each rocessor is comuted, and then the exact EFTs of the tasks are comuted. The second change is with regards to what is called insertion scheduling. In the HEFT algorithm, maing of tasks is erformed in the order of their ranks, and thus later tasks must not delay already scheduled, erhas, more

9 critical tasks. For systems with no thermal constraints, this amounts to checking that the scheduling windows of different tasks are non-overlaing. However, for thermally constrained systems, even for non-overlaing scheduling windows, earlier scheduled tasks can be delayed. This would haen if scheduling a new task v 1 before an already scheduled task v 2 raises the temerature sufficiently to require more idle time before execution of v 2. Thus, checking that already scheduled tasks are not delayed is comutationally more demanding for thermally constrained systems. The two techniques M 3 and M 4 differ from each other in their aroach to deal with this check. In M 3 we allow tasks to be scheduled only at the end of existing artial schedules, thereby trivially not delaying any of the earlier scheduled tasks, but erhas at the loss of otimality. In M 4 on the other hand, we allow tasks to be scheduled within an existing artial schedule, but subject to the fact that no earlier scheduled task is delayed. To accomlish this, we attemt every ossible ordering of a new task into a artial order and re-comute the thermal behavior. Clearly, the modeling accuracy and the comutation requirement of the maing hase increases from M 2 to M 3 to M 4. We exerimentally evaluate how this additional comlexity translates into imrovement in the end-to-end delays of the JUST schedules. V. EXPERIMENTAL RESULTS In Section 4.1, we resented two examles for FFT and LU decomosition alications, on different architectures. Indeed, the maings M 1 and M 2 as used in those examles corresond to the resective maings described in the revious section. As evidenced in the figures, the thermalaware maing M 2 clearly leads to JUST schedules of smaller end-to-end delay. Heuristics such as HEFT algorithms, are often comared to each other using several randomly generated task grahs. We emloy a similar aroach to comare the four maing techniques. There are three asects of a roblem which have to be randomly generated: the toology of the task grah, the timing roerties of the task grah and the system architecture. We briefly discuss each of them. We use three arameters to model the toology of the task grah: the number of nodes V, the density of the grah in terms of average degree of vertices, and the width of the grah in terms of the average width of a toological sort of the grah. We use two arameters to model the timing roerties of the task grah: the average execution time of the tasks and the Comutation to Communication Ratio (CCR), which is the ratio of average execution time to average communication time. We set the number of rocessors in the system to V /5, where V is the number of tasks in the generated task grah. We use two arameters to model the system architecture: the Parameter Value End-to-end M 1 comared to M 2 delay ratio Better Equal Worse #Nodes Density Width Nominal execution time CCR Processor frequency variability Link bandwidth variability Total Figure 3. Comarison of end-to-end delays of JUST schedule for maings M 1 and M 2 variability in the frequency of the rocessors f var about a nominal frequency and the variability in the bandwidth of the links B var about a nominal bandwidth. The values of f var and B var denote the standard deviation of the Gaussian distribution about the nominal values. The thermal model used in the exeriments is for an ARM-like rocessor sourced from [14] as shown in Figure 1(b), for a nominal frequency say f 0. For other frequency settings, we quadratically scale the ower arameters in the active mode, α act and β act, with frequency. The values in the idle mode, α idl and β idl are assumed to be constant: indeendent of the frequency. The WCET of a task v on a rocessor running at frequency f is given by (f 0 /f ) c v, where c v is the randomly generated execution time of task v with mean value equal to the average execution time. Similarly, we derive the delay along links as being inversely roortional to the bandwidth of the links. Each of the above arameters are varied through a set of values. The different values of the arameters are listed in Figure 3. We erform a fully factorial exeriment: for each combination of arameter values we generate 5 random roblem instances totaling to roblem instances to evaluate the roosed maings.

10 A. Comarison of M 1 and M 2 - The need for thermalaware scheduling Figure 3 comares, the end-to-end delay obtained for JUST schedules for maings M 1 and M 2 for the different values of the arameters. Since, we erform a fully factorial exeriment, we resent the results averaged for each value of each arameter. For each such set of values, we resent the end-to-end delay ratio, which is the ratio of the end-toend delay of the JUST schedule with maing generated by M 1 to that generated by M 2. The larger this ratio, the greater is the benefit of the thermal-aware maing ste. We also resent the number of cases where M 1 erforms better, equal and worse than M 2. Firstly, note that on average, the JUST schedule generated using maing M 2 has an end-to-end delay about 2.8 times smaller than that generated by maing M 1. This clearly indicates that for thermally constrained systems, the erformance enalty of being thermal-unaware even for the otimal scheduling strategy is severe. Since, the underlying HEFT is only a heuristic, we observe that for about 1% of the roblem instances M 1 erforms better than M 2. We now discuss the results for each arameter. The results indicate that the erformance ga between M 1 and M 2 is suer-linear in the size of the task grah. This imlies that the thermal-awareness is more critical in large and distributed task grahs. Results for variation in density and width arameters remain consistent across different values. With increasing nominal execution time, the relative erformance of M 2 imroves. This confirms the intuitive result that thermal-awareness is more critical when for longer eriods of time, the rocessors are run without reemtion. Relative erformance of M 2 is oor for very small values of CCR. This erhas is because with lower CCR values higher communication delays cause greater inaccuracies in the conservative estimate of (13). Relative erformance of M 2 greatly increases with higher rocessor frequency variation. To moderate the results, very small values of f var, u to 0.3, were chosen. For higher variations of frequency, which are not atyical in today s heterogeneous networks, the benefit of thermal-aware scheduling would be even more significant. A similar deendence on B var is not as evident. Thus, overall M 2 substantially out-erforms M 1. The advantage is exected to further increase for larger and more heterogeneous networks. B. Comarison of M 2, M 3 and M 4 For the same set of random test cases, we comare the end-to-end delay for JUST schedules generated for the three resented thermal-aware maings M 2, M 3 and M 4. The results averaged across the arameter variations are shown in Figure 4. As before, a higher end-to-end delay ratio indicates that the more sohisticated methods, M 3 and M 4, erform better than M 2. Comarison End-to-end delay ratio Better Equal Worse M 2 vs M M 2 vs M Figure 4. Comarison of end-to-end delay of JUST schedule of maings M 2, M 3 and M 4 As exected, the average otimal end-to-end delay of the different sets of maings is highest for M 2, which is followed by M 3 and M 4, in order. In ercentage terms the reduction in otimal end-to-end delay of maing M 3 and M 4 in comarison with that of M 2 is 3% and 9%, resectively: a modest gain in comarison to the 2.8x imrovement from M 1 to M 2. The average comutation overhead of generating maings M 3 and M 4 with resect to generating maing M 2 was found to be 1.1x and 2.8x, resectively. Furthermore, the number of solutions where M 2 is better or equal to M 3 and M 4 are substantial (45% and 25%, resectively). Thus, the more sohisticated maing strategies used for generating M 3 and M 4 rovide modest imrovements in end-to-end delay over the maing M 2 and at significant comutation overhead. The heuristic nature of the maing strategies resented here do not enable us to comment on the generality of these results. Perhas, other maing heuristics can be derived, which exlicitly model the thermal behavior and generate maings that greatly outerform M 2. VI. CONCLUSIONS With increasing ower densities in today s systems, thermal-aware scheduling must be considered to alleviate the roblem of high on-chi temeratures. In this aer, we studied the roblem of minimizing end-to-end delay of a task grah when scheduled on a arallel heterogeneous system, while working under secified thermal constraints, and with no seed scaling. When the maing (binding and ordering of tasks) of the task grah on to the rocessors is fixed, we showed that the roosed JUST schedule otimally reduces the end-to-end delay. Under such a schedule, enforcing thermal constraints was reduced to checking, before executing a task, if the temerature of the rocessor is under a re-comuted target temerature: a check that can be easily imlemented using on-chi temerature sensors. We then studied the maing roblem. We resented a modification of the timing roerties of the original task grah to model secified thermal constraints. On an extensive set of roblem instances, we showed that using a standard maing heuristic, HEFT, on the modified task grah gave maings with much smaller otimal end-toend delays than the maings generated using the original task grah. We also resented more sohisticated thermallyaware maing strategies which exlicitly modeled the thermal behavior of the rocessors. With results, we concluded

11 that these methods generate maings which rovide only small imrovements over the maings generated by the simle aroach of using the modified task grah. A ossible further work is to study imlementation of thermally constrained scheduling without the hel of thermal sensors, i.e., without the use of temerature interruts to enable the next task. This requires off-line comuted idle times to be inserted between the execution of consecutive tasks, such that the thermal constraints are satisfied for all ossible execution atterns. Considering that tasks always run for their WCETs may not rovide the correct value of the idle times. Another interesting direction forward is to consider systems where rocessors exchange heat, such as multicore systems. For such systems, the lower bound on γ min derived in (10) would deend on the state of the other rocessors. Conservative aroximations without much deviation from otimality have to be exlored. ACKNOWLEDGEMENTS This work was suorted by EU FP7 rojects EURETILE and PRO3D, under grant numbers and REFERENCES [1] Y.-W. Wu and et al, Joint exloration of architectural and hysical design saces with thermal consideration, in ISLPED, [2] H. Su, F. Liu, A. Devgan, E. Acar, and S. R. Nassif, Full chi leakage estimation considering ower suly and temerature variations, in ISLPED, [3] K. Skadron, M. R. Stan, W. Huang, S. Velusamy, K. Sankaranarayanan, and D. Tarjan, Temerature-aware microarchitecture, in ISCA, [4] D. Brooks and M. Martonosi, Dynamic thermal management for high-erformance microrocessors, in HPCA, [5] N. Bansal, T. Kimbrel, and K. Pruhs, Dynamic seed scaling to manage energy and temerature, in FOCS, [6] S. Wang and R. Bettati, Reactive seed control in temerature-constrained real-time systems, Real-Time Systems, vol. 39, no. 1-3, [7] J.-J. Chen, S. Wang, and L. Thiele, Proactive seed scheduling for real-time tasks under thermal constraints, in RTAS, [8] S. Zhang and K. S. Chatha, Aroximation algorithm for the temerature-aware scheduling roblem, in ICCAD, [9] Y. Xie and W.-L. Hung, Temerature-aware task allocation and scheduling for embedded multirocessor systems-on-chi (msoc) design, VLSI Signal Processing, vol. 45, no. 3, [10] Z. Wang and S. Ranka, A simle thermal model for multicore rocessors and its alication to slack allocation, [11] H. Tocuoglu, S. Hariri, and M.-Y. Wu, Performanceeffective and low-comlexity task scheduling for heterogeneous comuting, IEEE Trans. Parallel Distrib. Syst., vol. 13, no. 3, [12] Y. Liu, R. P. Dick, L. Shang, and H. Yang, Accurate temerature-deendent integrated circuit leakage ower estimation is easy, in DATE, [13] Y.-K. Kwok and I. Ahmad, Static scheduling algorithms for allocating directed task grahs to multirocessors, ACM Comut. Surv., vol. 31, no. 4, [14] C.-Y. Yang, J.-J. Chen, L. Thiele, and T.-W. Kuo, Energy-efficient real-time task scheduling with temeraturedeendent leakage, in DATE, 2010.