IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH

Transcription

1 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH 24 Dynamic On-Chi Therma Sensor Caibration Using Performance Counters Shiting (Justin) Lu, Russe Tessier, Senior Member, IEEE, and Wayne Bureson, Feow, IEEE Abstract Numerous sensors are currenty deoyed in modern rocessors to coect therma information for fine-grained dynamic therma management (DTM). To enhance rocessor erformance and avoid overheating, accurate temeratures must be obtained from therma sensors. Due to rocess variation and siicon aging, on-chi therma sensors require eriodic caibration before use in DTM. However, the caibration cost for therma sensors can be rohibitivey high as the number of on-chi sensors increases. In this work, a mode which is suitabe for on-ine cacuation is emoyed to estimate the temeratures of mutie sensor ocations on the siicon die. The estimated sensor and actua sensor therma rofie show a very high simiarity with correation coefficient.9 for most tested benchmarks. Our caibration aroach combines otentiay inaccurate temerature vaues obtained from two sources: temerature readings from therma sensors and temerature estimations using system erformance counters. A data fusion strategy based on Bayesian inference, which combines information from these two sources, is demonstrated aong with a temerature estimation aroach using erformance counters. The average absoute error of the corrected sensor temerature readings is <.5 o C and the standard deviation of error is ess than <.5 o C for tested benchmarks. Index Terms therma sensors, erformance counters, DTM, therma estimation, sensor caibration I. INTRODUCTION THERMAL management is a critica robem for modern microrocessors due to high transistor density. This characteristic increases ower consumtion and heat density in a sma siicon area causing erformance degradation and decreased system reiabiity. As a resut, dynamic therma and ower management strategies are often emoyed to tacke run-time therma and ower issues []. On-chi therma sensors deoyed in microrocessors are currenty used to assist DTM. For exame, there are five therma sensors er core imemented in the Power 7 EnergyScae infrastructure [2] and 2 sensors on each CPU core in the Inte Sandy Bridge [3]. Recent trends indicate increased future use to assess therma gradients and erform fine-grained therma management with more on-chi therma sensors. The efficiency and effectiveness of a DTM strategy reies on accurate therma sensor measurements. However, on-chi therma sensors are sensitive to rocess variations and can reort temerature vaues which deviate from actua ones. A This work was suorted by the Semiconductor Research Cororation under Task S. Lu, R. Tessier and W. Bureson are with the Deartment of Eectrica and Comuter Engineering, University of Massachusetts, Amherst, MA, 3, USA e-mai: ju, tessier, bureson@ecs.umass.edu. Coyright (c) 24 IEEE. Persona use of this materia is ermitted. However, ermission to use this materia for any other uroses must be obtained from the IEEE by sending an emai to ubs-ermissions@ieee.org. ouar imementation of an on-chi therma sensor uses a ring osciator whose frequency can be maed to a temerature vaue. In some cases, the temerature reading error of uncaibrated therma sensors can be substantia (u to 34 o C at 95 o C [4]) which adversey imacts DTM strategy. Two chaenges exist in using these sensors: () detecting if a sensor is roviding erroneous readings and (2) recaibrating the sensor, if necessary. Our focus in this work is the second chaenge. Often, therma sensor caibration invoves erforming therma imaging using an infrared camera whie caturing the hysica readings of therma sensors [5]. As the sensor count on a siicon die increases, the er-chi caibration cost can be rohibitivey high, eading to on-chi therma sensor use without individua sensor caibration. Even if therma sensors are initiay we-caibrated, their readings graduay drift away from actua temerature vaues due to device wear-out. Often, the degree of aging varies across the chi due to the activity variation of different subcircuits. Therefore, recaibration on therma sensors is needed to regain the required accuracy. In genera, it is not ractica to erform in-fied caibration with therma imaging since end users tyicay do not have exensive caibration equiment. In this aer, we take a different aroach to temerature estimation which is otimized for on-ine caibration of therma sensors. To dynamicay account for changing activities of the rocessor, coections of erformance counter vaues are used to estimate the chi therma rofie at run time. A erformance counter seection method is emoyed to reduce the intercorreations between readings and imrove estimation accuracy. Our resuts show that the correation coefficient between estimated and actua therma rofies is above.95 on a coection of benchmarks. In this aer, estimated temerature is excusivey used to refer to a temerature obtained from therma estimation using erformance counters. Estimated temerature vaues derived from erformance counter vaues are used to vaidate and correct divergent sensor readings via a muti-sensor coaborative caibration agorithm (MSCCA). This agorithm can be executed at run time using a bock of consecutive sensor readings. Corrected temerature vaues obtained from the agorithm are then used to adjust the maing of therma sensor arameters to temerature readings. A Bayesian technique integrated into MSCCA utiizes the imicit hysica roximity of the estimated temerature ocations (satia correation) to correct sensor reading errors. An increased number of on-chi sensors rovides increased agorithm accuracy since more satia correation information is catured within the estimated temeratures. To vaidate our agorithm, architectura and therma simua-

2 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH 24 2 tors are used to coect erformance counter and temerature vaues, resectivey. Resuts show that the average absoute error of therma sensor readings can be effectivey reduced to.5 o C for a tested benchmarks and the standard deviation of the errors in the corrected temeratures is reduced to an accetabe eve (from 3 4 o C to.5 o C). For comarison, the comutationa comexity and estimation accuracy of our new aroach is evauated against a method which is based on ower estimation and Kaman fitering [6]. Our resuts show that the new strategy has comarabe accuracy (better in many cases) but is more efficient (at east faster). II. BACKGROUND There are severa ways to caibrate on-chi therma sensors to achieve better measurement accuracy. We divide different aroaches into three main categories: therma imaging, design for caibration, and therma estimation. A. Therma Imaging The most traditiona way of caibrating on-chi therma sensors measures the therma rofie of a running chi using therma imaging technoogies and reorts the sensor readings at that time instant [7]. The mode arameters can then be obtained by soving a series of equations or by using statistica arameter inference [8]. Usuay, the caibration cost associated with this aroach is very high since every chi exeriences a different therma imaging resonse and the amount of effort increases with the number of on-chi therma sensors. Athough the aroach coud be used once after chi fabrication, it cannot be used effectivey at run time. B. Design for Caibration The second caibration technique uses design-forcaibration. This aroach is imemented by integrating dedicated hardware circuits on chi which monitor the rocess variation around the therma sensors [9]. With the knowedge of the chi rocess variation, the errors in therma sensor readings can be comensated and mode arameters can be otimized to refect the hysica reationshi between the temerature and hysica quantities. Since the rocess variation monitoring hardware consumes siicon rea estate, it raises the chi cost when a arge number of sensors are integrated. C. Therma Estimation A third aroach uses accurate on-chi therma estimation instead of therma imaging to determine actua temeratures. This information can then be used for caibration of secific sensors. In Liu [] and Cochran and Reda [], the authors describe methods to construct therma estimates for numerous oints in a rocessor from measurement data from a sarse set of therma sensors. In Ranieri et a. [2], the overa therma ma is recovered from a reduced set of sensors by seecting rincia eigenvectors of the whoe-chi temerature vector. In Zhou et a. [3], an information-theoretic framework is roosed to find the otima ocation for sensor deoyment and fu-chi therma monitoring. Since the therma sensor measurements are subject to noise, the amount of error at each secific sensor can be difficut to determine. As a resut, most recent aroaches for sensor caibration use a combination of sensor readings and other on-chi information to generate estimates of actua temerature. Kumar et a. [4] use the erformance counters in an Inte Pentium-4 rocessor to estimate the overa chi temerature. For mutie sensor caibration, it is necessary to estimate the temerature at the micro-architectura eve, so an estimation strategy with finer granuarity is needed. Lee et a. [5] roosed a run-time temerature sensing strategy using erformance counters in high-erformance rocessors. In this strategy, erformance counters are used to estimate the ower dissiation for each hardware comonent and the estimated ower traces are then used to estimate the temerature trace based on the therma mode imemented in a therma simuator. The maing from ower to temerature requires a comex therma mode which characterizes the therma RC network of the given chi. In two recent aers [6][6], two sources of temerature information are combined to generate temerature estimates: () noisy sensor readings and (2) ocaized ower consumtion which is reated to temerature. The technique used to integrate data from these two data sources is Kaman fitering (KF). Athough ower traces can be accuratey estimated at run time [7], a therma RC mode is required to determine the maing coefficients required to convert ower dissiation to temerature in the rediction ste of KF aroaches. Unfortunatey, the derivation of this mode is not trivia due to the comexity of siicon materias [8]. KF-based aroaches have shown the abiity to track the temerature rofie of a chi at a high comutationa cost since KF is erformed each time a temerature estimation is made. In Zhang and Srivistava [9], the temerature for noisy sensors is estimated using statistics. Like other aroaches in the revious aragrah, our new caibration method fits into the third category of caibration aroaches. Unike revious techniques, we directy use information from erformance counters for temerature estimation rather than using ower consumtion as an intermediate vaue for conversion between erformance counter information and estimated temerature. This direct aroach reduces run time and eiminates the need to estimate er-functiona unit ower consumtion. A merging agorithm is used to combine our temerature estimates and sensor reading data. This work greaty exands uon the materia in our revious ICCAD 22 aer [2] which targeted simiar goas. In this aer we track incrementa changes in temerature rather than absoute temerature, dramaticay increasing accuracy. Aso, we consider temerature gradients between sensors based on earier estimated temeratures in addition to erformance counter vaues in determining current temerature estimates for sensors. To vaidate the effectiveness of the aroach, 5 benchmarks are used to erform temerature verification and the training rocess for our mode deveoment is erformed using 6 benchmarks, rather than one.

3 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH 24 3 Trained Therma Estimation Mode (β arameters) Temerature Recordings via Therma Imaging Temerature Mode Training Performance Counters Performance Counters Design or Post-Siicon Phase.5.5 Brach rate Load rate Store rate Integer rate Foating Point rate IPC ICache miss rate ICache read miss DCache miss rate Dcache read miss Temerature Estimation Estimated Temerature Sensor Readings On-Chi Therma Sensor Recording Run Time Phase Initia Temeratures (for temerature change estimation) Merging Agorithm Fig. 2. Correation between temerature and some system statistics for the radix benchmark across 24 therma sensors Fig.. Corrected Temerature Our dynamic caibration scheme for on-chi therma sensors D. Our Caibration Strategy: Aroach Overview Fig. shows our strategy for dynamic on-chi sensor caibration. This fow can be broken down into four stes, one which is erformed once during the design or ost-siicon hase and three which are erformed reetitivey at run time. ) Temerature mode training - In the design or ostsiicon hase, a therma estimation mode is deveoed based on accurate temerature recordings through therma imaging technoogy and system statistics from erformance counters. The mode training oututs a set of arameters caed β arameters. These β arameters define the reationshi between erformance counter vaues and estimated temeratures. 2) Temerature estimation - The β arameters are used in a series of inear equations to convert erformance counter vaues to temerature estimates. Athough usefu, temeratures obtained from this mode often do not meet accuracy requirements since erformance counters cannot cature a on-chi therma detais recisey. 3) On-chi therma sensor recording - Potentiay noisy therma sensor readings are coected from on-chi therma sensors. 4) Merging agorithm - To caibrate a therma sensor, we combine therma estimations and sensor readings using a Bayesian-based fusion agorithm. This MSCCA agorithm generates corrected temerature vaues and identifies how much a therma sensor shoud be adjusted in caibration, if needed. In the foowing section, temerature estimation and mode training are considered. Section IV describes our merging agorithm and the techniques used for on-chi sensor reading. III. TEMPERATURE ESTIMATION AND MODEL TRAINING USING PERFORMANCE COUNTERS In a microrocessor, erformance counters monitor runtime system statistics, such as the oad/store rate, branch rediction miss rate, amount of cache misses, and instructions er cyce (IPC), among others, for various system management uroses. Since these statistics contain the activity information of functiona units in the rocessor, they can be used to estimate the ower consumtion at a er-structure granuarity using inear regression [7] or unit ower consumtion [5][2]. Unike ower consumtion estimation, functiona unit temerature estimation is more comex due to intercomonent correation resuting from heat fow across the chi. It can be shown that the temeratures of functiona units are correated with vaues read from on-chi erformance counters. Fig. 2 shows the correation coefficients of comonent temeratures and various system statistics for the SPLASH-2 radix benchmark [22]. Most temerature-statistic airs show non-zero correation coefficients. For exame, the foating oint rate shows a negative correation with integer units (e.g. the integer scheduer) and a ositive correation with foating oint comonents (e.g. the foating oint scheduer). To exore the correation between the aication characteristics and temerature changes over a short time eriod, system statistics and a temerature trace were recorded for every miisecond via simuation using SESC [23] and HotSot [24]. Fig. 3 iustrates the reation between the system statistics: IPC (the first row), integer instruction rate (the second row) and foating oint instruction rate (the third row), and temerature change rates for three functiona units: integer scheduer (eft coumn), foating oint scheduer (midde coumn) and L data cache (right coumn). The erformance counter data was obtained by reeatedy running the equake benchmark from the SPEC2 benchmark suite using the SESC simuator. The temerature trace was generated by HotSot. As seen in the figure, higher average IPC (hase A in the to-eft sub-figure) resuts in a higher temerature change rate for the integer scheduer at the start of hase A. However, this change rate is negative for the foating oint scheduer at the same oint. The foating oint instruction rate is higher in hase B and the scheduer is more active during this hase. eading to a foating oint scheduer instruction temerature surge every time the aication transitions from hase A to hase B, athough it is short. In this case, the temerature of the comonent is imacted by its surroundings due to heat fow. The temerature difference (referred to as therma gradient in the figure) between the secific comonent and a neighboring comonent is otted to iustrate this oint in Fig. 4. The therma gradient in the figure is obtained by taking the maximum temerature difference among a neighboring bocks. In the integer scheduer, for exame, the temerature surge

4 temerature change therma gradient temerature change f rate temerature change integer rate temerature change IPC IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH temerature change A Integer Scheduer Foating Point Scheduer.2 B Integer Scheduer.5 Foating Point Scheduer sec temerature change 2 3sec. temerature change sec sec x -3 L Data Cache x 3 L Data Cache 2.5 3sec x sec.2 2 3sec 2 3sec x x sec.2 2 3sec.2 2 3sec 2 3sec x sec sec 2 3sec x -3 Fig. 3. Runtime recording of some system statistics and temerature change rates for three function units: integer scheduer (eft coumn), foating oint scheduer (midde coumn) and L data cache (right coumn) for the equake benchmark. Temerature changes are in o C 2 3sec.2 2 3sec 2 3sec Integer Scheduer.2 2 3sec temerature change 2 3sec Foating Point Scheduer sec sec IPC integer rate f rate.4 3 x 3 L Data Cache 2 3sec sec.4 therma gradient Fig. 4. Runtime recording of therma gradient and temerature change rates for three function units. Temerature changes are in o C causes an increase in the therma gradient which acceerates heat fow from the integer scheduer to its neighbors. A new therma baance is quicky reached after a short time. A. Temerature Estimation Using Performance Counters By virtue of a variety of inter-unit therma comexities, the temerature at a secific osition on the chi is generay not ineary reated to one articuar erformance counter. However, we demonstrate that a inear mode can be used to estimate on-chi temerature changes at secific temerature sensors using mutie erformance counters if the time interva is sma. This inear aroximation is shown to be effective emiricay in deveoing an on-chi therma rofie. These therma estimates can then be merged (Section IV) with sensor readings to reduce satia (across sensors) and temora (across time) noise. In the foowing derivation we are interested in determining T estimates for secific sensors over a time interva, rather than absoute T vaues. ) Linear Temerature Estimator: In deveoing a inear mode for a secific therma sensor i over a time interva, a row vector (x i ) contains recorded erformance counter vaues for the interva, M (isted in Tabe I), therma gradient information, G i, and temerature, T i. Vaues G i measure the therma gradient between other therma sensors and sensor genera rate cache TABLE I PERFORMANCE COUNTERS PROVIDED BY SESC buffer and queue usage branch IPC, integer rate, oad rate store rate, foating oint rate Dcache read miss rate, Dcache write miss rate, Icache miss rate oad queue, store queue, ROB, Iwin, TLB BTB, RAS i at the beginning of the interva. Vaue T i measures the temerature of sensor i at the beginning of the interva. Since the correct temerature before the first interva is unknown, the therma gradients and T i can be aroximated by using therma sensor readings at the start of caibration. For other intervas, the temerature estimation from the revious interva is used. The combination of M, G i, and T i forms x i : x i = [M, G i, T i ] () For exame, for the integer scheduer (unit 8), G 8 incudes a temerature differences between the integer scheduer and other comonents at the beginning of the measurement interva. Here, the suerscrits of T indicate the hardware comonents in the fooran (Fig. in Section V). The therma gradient vector for the integer unit is given by (2). There are 24 architectura comonents in the studied rocessor,

5 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH 24 5 so G 8 contains 23 eements which are temerature differences between the comonents and the integer scheduer, excet itsef. The erformance counter vector M can be reresented by (3), where u is the number of erformance counters used in the mode. G 8 =[T T 8,..., T 7 T 8, T 9 T 8,..., T 24 T 8 ] (2) M = [P, P 2,..., P u ] (3) The T i vaue in () is used to take static ower (which is deendent on temerature) into account. Therefore, the samed vector at a articuar time ste is given by (4) for the integer scheduer. It is aarent that both hardware activities (as measured by erformance counters) and therma gradients imact temerature change during the saming interva. x 8 =[P,..., P u,t T 8,..., T 7 T 8, T 9 T 8,..., T 24 T 8, T 8 ] (4) Performance counter vaues reresent changes in the resective event counters during the saming interva. Ony events haening in a secific interva are evauated for the corresonding erformance counter monitors. Using the above x vector, it is ossibe to estimate the temerature change of a articuar comonent which contains the therma sensor during the saming interva using a inear equation. For exame, the equation for therma sensor i is: T i = x i β i (5) and for the sensor in the integer scheduer: T 8 = x 8 β 8 (6) Here, β 8 is a coumn vector whose eements are coefficients of the inear mode. The coefficient vector β 8 can be determined through mode training which wi be discussed in the next subsection. Each sensor i is trained searatey to obtain its own β coefficient vector. The inear mode ony needs mutiications and additions to cacuate the resuts, so the time cost is ow and cacuations can be done in rea time. B. Linear Mode Training β i As mentioned earier in this section, the first ste in deveoing a reationshi between erformance counter vaues and estimated temeratures (e.g. β vectors) invoves training. The accuracy of the coefficient vector β i imacts the mode accuracy for sensor i. In this training ste, accurate known temeratures for the sensors must be avaiabe to deveo the reationshis. As mentioned in Section II, these reationshis can be determined via architectura and therma simuation during design once hysica characteristics of the chi have been determined or during ost-fabrication testing using therma imaging. In ost-fabrication testing, it is ossibe to feed rea workoads to the system and read erformance counter registers. At the same time temerature vaues can be catured through infrared imaging of the running system. Unike er-chi caibration, it is ony necessary to erform data caturing on a sma amount of same chis to get the genera information of a articuar chi series. We assume that the secific information of an individua chi caused by rocess variation is refected in the therma sensors. In the foowing training anaysis, we assume that accurate temeratures and x vaues consisting of M and G are avaiabe for a sensors. The most straightforward way to train the inear mode to determine β vectors in (5) is to use an ordinary east square method (OLS) [2]. The coefficient vector obtained through OLS is given by (7). β i os = (X it X i ) X it y i (7) Here, X i is a matrix consisting of row vectors x i cacuated over a series of N saming intervas. Each row of X i reresents x i for one same interva. y i is a coumn vector comrised of accurate actua temerature changes for sensor i which occur during the resective training intervas. Athough OLS is caabe of training the inear mode, its somewhat simistic formuation does not consider the intercorreation of erformance counters, imiting accuracy. A more advanced, iterative mathematica aroach can be used to determine β vaues. As an aternative to OLS, we use automatic reevance determination (ARD), which was deveoed by MacKay [25] and Nea [26]. The coefficient vector β i for sensor i can be reresented by the foowing exressions (8) and (9). β i = δ 2 SX it y i (8) S = (A + δ 2 X it X i ) (9) In (9), A = diag(α,..., α M ), which is a diagona matrix. Each α j in A reresents the reevance of an inut vector variabe to the resut such that: δ 2 = α j = α js jj β i j N n= (yi n x i n β i n) 2 N M j= ( α js jj ) 2 () () Since (8) and (9) deend on () and () and vice versa, mutie iterations are needed to achieve convergence of the β i unknowns. These iterations cacuate α j in diagona matrix A and δ. In the above equations, S jj are eements of S. y n is the n th eement of y i and x n is the n th row vector of X i. N is the number of training sames and M is the ength of vector x i and the dimension of the matrix S. Vaues for α j and δ 2 are determined by aternating evauation of the above four equations unti convergence. From our exeriments, around four iterations are erformed unti these arameters reach convergence. C. Evauation of Mode Training Accuracy Using Therma Estimation To evauate the accuracy of the inear mode, Tabes II and III show the estimation error for one time interva. In this case, (5) is evauated for one time interva, using known T vaues to determine G gradients and measured erformance counter vaues P. Errors between the actua T and T vaues determined with (5) are then cacuated. Tabe II gives

6 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH 24 6 TABLE II AVERAGE ESTIMATION ERROR FOR EACH SENSOR OVER ALL BENCHMARKS Avg. abs. error ( o C) Std. abs. error ( o C) Avg. change ( o C) Std. change ( o C) Avg.abs. error ( o C) Std. abs.error ( o C) Avg. change ( o C) Std. change ( o C) TABLE III AVERAGE ESTIMATION ERROR FOR EACH BENCHMARK OVER ALL SENSORS benchmark mcf vortex swim art asi radiosity ocean radix Avg. abs. error ( o C) Std. abs. error ( o C) Avg. change ( o C) Std. change ( o C) benchmark arser twof vr amm au barnes fft water-satia Avg.abs. error ( o C) Std. abs. error ( o C) Avg. change ( o C) Std. change ( o C) the average absoute error and error standard deviation for each sensor for a singe interva using the β vaues determined through training. The error is averaged over a 6 test benchmarks (benchmarks described in more detai in Section V). Using information from this tabe it is ossibe to evauate the trained modes for a sensors. Sensor 8 (integer scheduer) and sensor 2 (FP scheduer) reort reativey high error and error variation comared with other sensors due to their high activity. The error can otentiay be reduced further with a smaer time ste, but this aroach increases the comutationa cost corresondingy. Tabe III gives the average absoute error and the associated standard deviation for each benchmark. The error is averaged over a sensors for each benchmark. From this tabe, we can evauate how the trained modes work for a benchmarks. In genera, average absoute error and standard deviation are ow in the tabes. During exerimentation we found that the trained mode was most effective for benchmarks which have simiar execution characteristics to the benchmark training set. However, the use of a broad cass of benchmarks for training hes minimize error across a arger number of benchmarks. D. Therma Estimation Resuts The SPLASH-2 and SPEC2 benchmark suites were used to vaidate the effectiveness of our inear mode. Mixed sames from a subset of benchmarks were used to train the inear mode (find β vaues) and the trained mode was tested for estimation on the rest of the benchmarks. Detaied simuator setu information is rovided in Section V. ) Estimated Therma Profie: Fig. 5 shows the therma rofie across a sensors at four time instances of the sensor 2 actua sensor 2 estimation sensor 8 actua sensor 8 estimation sensor 2 actua sensor 2 estimation (a) correation coefficient vortex swim art asi ocean radix twof amm (b) Fig. 6. (a) Temerature estimation over 6 seconds for sensor 8 (integer scheduer), 2 (foating oint scheduer) and 2 (ALU). The data is coected through simuation by running au on the AMD fooran. (b) The correation between estimated and actua temerature rofies for various benchmarks. SPLASH-2 ocean benchmark. Exeriments with other benchmarks created simiar grahs. At the first time oint, the estimated rofie is inaccurate due to the ack of knowedge of initia temeratures. In succeeding time oints, the estimated temerature rofie more cosey matches the actua rofie. At the 3 rd second, a cose temerature rofie match is achieved. It shoud be noted that whie an absoute temerature match is not achieved, a reative match across the sensors is rovided. In Section IV, this systematic drift is offset by adding constant vaues to temeratures estimated from sensor readings. 2) Temora Evoution of Temerature Estimation: Fig. 6(a) shows how the estimated temerature rogresses over time for three sensors (integer scheduer, ALU and foating oint scheduer). Other sensors show simiar trends. Since the initia temeratures are randomy chosen around o C for a

7 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH estimate actua Time sec Time sec Time 2 sec Time 3 sec Fig. 5. The estimated and actua rocessor temerature rofie at four time instances for the SPLASH-2 ocean benchmark. The estimated therma rofie exhibits a simiar shae as the actua rofie but with an offset. Each of the 24 therma sensors in the rocessor are reresented on the horizonta axis for the time instance. The numbered comonents of AMD rocessor can be found in the fooran of Fig.. The estimates track the actua temerature with a fixed offset er sensor. This offset w can be determined (Section IV-B) and comensated. avg abs error ( o C) rincie comonents rincie comonents 4 rincie comonents mcf vortex swim art asi radiosityocean radix arser twof vr amm au barnes fftwater satia Fig. 7. Prediction accuracy comarison for different numbers of rincia comonents used in the mode. sensors, the estimation mainy refects heat diffusion during the first 3 seconds. Over time, the temeratures of these three comonents are corrected to match their aroximate reative vaues (the foating oint scheduer is the hottest and ALU is the cooest). Fig. 6(b) shows the correation coefficient between these two vaues over time. To evauate accuracy, the effect of imiting the number of erformance counters used to generate therma estimates is aso considered. Tabe IV indicates the average absoute error over a sensors and benchmarks for different numbers of erformance counters used to generate estimates. Fourteen of the counters rovide itte benefit in terms of absoute error. TABLE IV AVERAGE ABSOLUTE ERROR OF TEMPERATURE ESTIMATION FOR ALL SENSORS AND BENCHMARKS IF THE NUMBER OF PERFORMANCE COUNTERS IS LIMITED TO SPECIFIC QUANTITIES No. counters Abs. error ( o C) E. Princia Comonents of Performance Counter Vectors In Section III-A it was shown that combinations of erformance counter changes and therma gradients can be combined to estimate temerature changes. However, it has reviousy been determined [27] that erformance counter vaues are correated, otentiay eading to mode instabiity. For exame, a branch miss rediction may ead to a ieine fush which imacts IPC. To exore the imact of this issue, exeriments were erformed to reace the P i vaues in (3) and (4) with uncorreated rincia comonents [28]. Princia comonent anaysis (PCA) transforms an inut vector (in this case u erformance counter vaues) into a new vector set by mutiying the inut vaues with a matrix of the eigenvectors derived from the set, as shown in (2). P u = P u C (2) P u is the origina vector of u erformance counter vaues coected from the rocessor and C is a coefficient matrix of eigenvectors determined during mode training. P u is the rincia comonent vector. In many cases, deending on the eigenvaues of the origina data set, some of the P u set may be ignored, eading to a reduced dimension vector P v. Rather than inserting P u erformance counters into the inear mode in (5), the reduced dimension rincia comonent estimates are inserted instead. Since variabes in P v are orthogona with each other, the muticoinearity robem is eiminated. Exerimentation showed that the argest 4 rincia comonent estimates corresond to non-zero eigenvaues, but the remaining 2 have eigenvaues cose to zero. In genera, to maintain maximum accuracy, the number of rincia comonents (e.g. the dimension of v in P v) used in the

8 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH 24 8 β(s) Fig. 8. ) β oints cubic fitting ine fan seed (rm) One β mode using cubic fitting for integer modue (bock 8 in Fig. mode shoud incude the number of non-zero eigenvaues, in this case 4. Fig. 7 shows the therma estimation error for different numbers of rincia comonents used in the mode. As exected, accuracy is imroved as the number of rincia comonents is increased from 5 to 4. Princia comonent count increases beyond this vaue do not imrove accuracy. To assess the benefits of PCA, the exeriments described in Section III-C were erformed using the fourteen PCA vaues in ace of the thirty-four P i vaues as art of the x i vector in (5) after mode retraining. In a our exeriments, the estimated temerature resuts were neary identica, indicating the negigibe effect of erformance counter correation. As a resut, the rest of our reorted resuts use P i vaues in (5) rather than PCA vaues. F. Dynamic Mode for Changing Cooing Conditions In contemorary comuter systems, a variety of cooing technoogies (e.g. fans, iquid) are used to efficienty remove heat from the microrocessor and rotect it from overheating. Often, the amount of cooing (e.g. fan seed, fuid fow seed) is dynamicay adjusted based on a rocessor s therma situation. As a resut, the β arameters determined through training in Section III-A are ony vaid for a secific cooing amount. In systems with mutie cooing eves, effectivey (5) for therma sensor i can be restated as: T i = x i β(s) i (3) where β(s) vaues have been determined using the training method described in Section III-B for a secific cooing amount (e.g. fan seed). In this case, β(s) training (Section III-B) is erformed at each cooing amount, s. In erforming caibration, the aroriate set of β(s) vaues can be used based on the current cooing amount. The drawback of this method is that it increases storage cost incurred by storing mutie mode arameters. Athough this mutie training aroach can be effectivey used for mutie, discrete cooing amounts, it does not address the issue of a arge number of ossibe cooing amounts. The mode to dynamicay adat β(s) for an s which was not reviousy trained can be achieved using curve fitting. Fig. 8 shows known β arameters determined through training for integer scheduer as bue stars. The exame shown in Fig. 8 exhibits a cubic fit. By buiding β(s) modes, ony a few β arameters for secific s cooing amounts must be stored, saving storage sace. IV. MULTI-SENSOR COLLABORATIVE CALIBRATION ALGORITHMS (MSCCA) Resource-imited therma sensors, such as ring osciators, often are affected by noise due to rocess variation and gradua device wear-out. As a resut, their readings may drift away from accurate vaues. In this section we show that sensor readings can be combined with estimates derived using the erformance counter aroach from the ast section to generate more accurate corrected temerature readings. Athough it is exected that sensor readings wi track corrected readings for ong eriods of time, if the reading for a secific sensor significanty differs from its corrected readings for a number of sames, the sensor can be recaibrated. In this section, to determine accurate temerature vaues, estimated temerature vaues obtained in Section III and readings taken from sensors are merged using a Bayesian inference based agorithm. The corrected temeratures can then be used for therma caibration. A. Probem Formuation Bayes theorem resents the reationshi between a known (riori) robabiity distribution and a osterior robabiity distribution; it is widey used for arameter inference. The unknown arameter distribution is reresented by (θ), which reresents the rior knowedge of θ and the distribution of random variabe x for a given θ is (x θ). The distribution of θ after an observation can be cacuated using the foowing formua. (θ x) = (x θ)(θ) (4) (x) For our sensor caibration robem, the actua temeratures of sensors are unknown attributes which are estimated by Bayesian inference. The foowing definitions are used for the formuation of the sensor caibration robem. t and (t) : the random vector of the actua temeratures and its robabiity distribution; r and (r) : the random vector of the therma sensor readings and its robabiity distribution; e and (e) : the random vector of the estimated temeratures and its robabiity distribution; Σ r : the covariance matrix of the random vector r; Σ e : the covariance matrix of the random vector e; (r t): the robabiity distribution of the sensor readings given the actua temeratures (sensor noise distribution); (t r): the robabiity distribution of the actua temeratures given the sensor readings (statistica inference after an observation); The robabiity distribution of the actua temerature t is given by the foowing formua. Note that t and r are mutivariate random variabes. (t r) = (r t)(t) (5) (r) In the above equation, the riori knowedge of the actua temerature distribution is (t), which can be obtained via therma estimation discussed in Section III. So, the riori knowedge is (e). The osteriori inference of an actua temerature after an observation is (t r).

9 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH 24 9 Since the temerature change rate is ess than. o C er miisecond [6], we assume that the actua temerature kees constant during a miisecond eriod. For today s high erformance rocessors, this corresonds to severa miion cock cyces and enough sensor and erformance counter readings can be obtained to erform the caibration agorithm. The corrected temerature is defined as the exected vaue of the conditiona random vector t r which is cacuated by the foowing equation. µ t = E(t r) = t (t r)dt (6) The covariance matrix of the corrected temerature is given as: Σ t = E[(t µ t )(t µ t ) ] (7) The robabiity distribution can be characterized by coecting a time series of sensor readings. Because there are many factors, such as suy votage, rocess variation and ambient temerature fuctuation which imact the sensor readings, the noise of a therma sensor foows a Gaussian distribution, i.e. r t N (t, Σ r ). In the Gaussian case, (6) and (7) have cosed form reresentations as foows [29]. µ t = µ e + Σ e (Σ e + Σ r ) (r µ e ) (8) Σ t = Σ e Σ e (Σ e + Σ r ) Σ e (9) Thus, the exected actua temerature given r, (µ t ), and its covariance (Σ t ) can be determined directy from sensor readings and estimated temerature vaues from erformance counters. B. MSCCA Agorithm The goa of the MSCCA agorithm is to determine the corrected temerature (µ t ) and covariance (Σ t ) for each temerature sensor once er sames. As seen in Agorithm, during each of sames, temerature sensor readings r and erformance counter vaues are read. For the same, the sensor readings from a temerature sensors form a row in an R matrix (ine 8). Additionay, the erformance counter vaues are converted to estimated temerature changes for each sensor using (5) (ine 5). These temerature changes are added to the estimated temeratures from the revious same (ine 6) and the resuts for each sensor is stored in an E matrix (ine 7). After rocessing sames, corrected temerature vaues for each temerature sensor are determined (ine 4) using (8) and (9). Vectors r and µ e used in the corrected temerature cacuation are determined from the coumnwise mean of the E and R matrices (ines and 2). As noted in Section III and shown in Fig. 5, the use of erformance counters to estimate temerature shows a strong reative match, athough an absoute offset for the actua temerature is often resent. To address this issue, a er-sensor offset vaue w is added to each r reading. Athough we found that w vaues are constant for each sensor across time and across benchmarks, the vaues are recacuated in the agorithm for consistency. TABLE V OPERATIONS REQUIRED BY MSCCA AND KALMAN FILTERING FOR SETS OF SAMPLE READINGS FOR 24 THERMAL SENSORS Oer- Estim- MSCCA Aroach KF Aroach ation ation corr. tota corr. tota scaar (444 48) (44 48) add. 86 scaar (3 + 48) (2 + 48) mut. 86 mat. add. 2 2 mat. mut. mat. -vect. 3 3 mut. mat. inv. vector add. 3 3 In our exerimentation, cacuation was erformed over tota sames with sames er invocation. A tota of invocations are used for the same set. As shown in Fig., temerature change estimation requires an initia temerature rofie of the siicon die which may not be avaiabe at runtime. For initiaization of the estimation aroach, it is ossibe to assign an arbitrary temerature to each therma sensor or to use therma sensor readings as initia temeratures. Agorithm Muti-Sensor Coaborative Caibration Agorithm MSCCA : Initiaize w. 2: Initiaize temerature rofie 3: whie Invocation count do 4: for i = ; i < ; i++ do 5: Estimate temeratures determined from erformance counters using (5) 6: Add temeratures to revious corrected temeratures and get udated temerature rofie. 7: Store the udated temeratures as a row in E matrix. 8: Store sensor readings as a row in R matrix. 9: end for : Adjust R matrix by adding offset w to each row. : The vector r is the coumnwise mean of R. 2: The vector µ e is the coumnwise mean of E. 3: Cacuate the covariance matrices Σ r and Σ e. 4: Perform Bayesian inference using Equations (8) and (9), and get the corrected temerature µ t. 5: w µ t - r. 6: end whie C. Comutationa Cost Evauation This section anayzes the comutationa comexity of the MSCCA aroach and comares it with the comexity of using Kaman fitering to generate corrected temeratures for therma sensors. Athough a fu discussion of the KF agorithm for temerature estimation can be found in [8], we rovide a brief overview of the required oerations here.

10 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH 24 The KF aroach requires two estimation stes to convert erformance counter vaues to estimated temerature. First, the ower consumtion of individua functiona units is determined using a inear set of equations which have been determined via inear regression [7]. Per-functiona unit ower vaues are then converted to estimated temerature via a second set of inear equations [8] whose derivation require the difficut aroximation of therma resistance and caacitance for on-chi functiona units. To deveo corrected temerature vaues from estimates and sensor readings, KF then uses cross correation with reviousy-determined noise vaues to merge the estimates and readings together. Unike our aroach, where corrected temeratures are generated every sames, KF requires corrected temerature evauation for every same, a significant time enaty. Thus, our aroach has two significant ractica benefits versus KF: Estimated temeratures are determined directy from erformance counter vaues rather than requiring ower as an intermediate vaue. The eimination of ower as a transition metric aso eiminates the need for comicated therma resistance and caacitance cacuation. MSCCA requires many fewer oerations and can be erformed ess frequenty reducing run time.. The comutationa cost for MSCCA and KF aroaches (not considering mode training which takes ace ony once at design time) can be broken down into two arts: temerature (or ower) estimation and temerature correction. The comutationa cost of the ower estimation for KF and temerature estimation for MSCCA is the time required to cacuate a inear combination of scaed erformance counter vaues. As a resut, the estimation comexity is O(n), where n is the number of erformance counters and is the number of same sets. The estimation coumn in Tabe V shows the number of oerations needed to erform this estimation (therma for MSCCA and ower for KF). There are n = 34 erformance counters incuded in our inear regression mode, so we secify comexity in terms of this vaue. The MSCCA aroach stores sames and erforms Bayesian estimation once er time stes. Tabe V shows the number of oerations erformed for sets of readings. For the MSCCA, time instances (sets) of readings er invocation are used. As noted in the revious subsection, caibration can be simutaneousy erformed for mutie consecutive sensor readings for each sensor in one invocation. In our imementation, there are m=24 therma sensors, so the matrix dimensions of Σ r and Σ e are If the matrix oerations are converted to scaar oerations, there are about, additions and, mutiications required for the KF method. In our method, the numbers of additions and mutiications are about (44 + 4, ) and (2 + 4, ). In contrast, KF-based agorithms redict and udate the temerature for each set of same readings, resuting in more matrix oerations. In Tabe V, the 34 scaar oerations for KF reresent the oerations to convert ower estimates to temerature estimates for a singe sensor. The remaining oerations reresent merging comutations for temerature estimates and temerature sensor readings. D. Memory Cost Evauation Since sames must be stored in matrices for a eriod of time before they are rocessed, MSCCA does require more memory usage than the KF-based aroach. In genera, the KF aroach does not require storage for the ower estimates and sensor reading sames. At each time ste, the therma sensor sames (sensor readings) and temerature estimates determined from ower estimations are used to udate temeratures, and then these sames are thrown away. The MSCCA aroach must store therma sensor sames R and temerature estimates E for sames in memory unti the next MSCCA evauation. As a resut, the memory comexity for the KF aroach is O() and for MSCCA is O(m). In our exeriments, is severa hundred and m = 24 sensors are used. So the memory storage of sames is around severa kiobytes. E. Imementation Issues The use of therma caibration raises concerns about overburdening the hardware and oerating system of the target rocessor. However, the nature of our caibration aroach and recent trends in on-chi monitoring for microrocessors essen this concern. In genera, therma sensor caibration is exected to be erformed once every few seconds, rather than miiseconds. In Section VI, it is shown that agorithm execution time is on the order of tens of miiseconds for evauation that is erformed every ten seconds. This overhead imits the oerating system and rocessor-eve ower and temerature imact of the agorithm itsef. Indeendent of this overhead imit, recent trends indicate that microrocessors increasingy incude dedicated circuitry to erform monitoring and monitor data rocessing which is searate from the main OS/rocessor comute atform. For exame, IBM EnergyScae [2] uses temerature and critica ath monitors aong with a microcontroer for sensor data rocessing. Inte s Active Management Technoogy rovides a searate on-chi communications channe and controer to monitor device oeration and contro system resonses at the oerating system eve. Often, these monitoring and monitor data rocessing infrastructures can be quite sma comared to the main rocessing infrastructure (e.g..2% of overa rocessor area [3]), imiting system erformance imact. These effects can be weighed against the benefits of a more accurate DTM aroach due to imroved therma sensor caibration. V. INFRASTRUCTURE AND EXPERIMENTAL APPROACH In this work, a simuation-based method is emoyed for data coection, mode construction and verification. Our dynamic sensor caibration strategies are verified with the data from simuation. In this section, we describe the two simuators used by this work and other exerimenta infrastructure. A. Architectura Simuator We use the SESC simuator [23] as the infrastructure for coecting system statistics. SESC is a cyce-accurate simuator

11 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. X, NO. Y, MONTH 24 HotSot Goden Temerature Satia&Temora Noise Sensor Readings Merging Agorithm Corrected Temerature SESC Simuator System Statistics Therma Estimation Mode Estimated Temerature Fig. 9. Sensor data and therma estimation merging scheme. Sensor readings are artificiay generated shown in the eft fow; Estimated temerature from erformance counter are obtained by the right fow. ower estimate to the SESC ower estimate for each functiona unit. First, a dynamic ower trace of a function units for a secific aication is generated. A ercentage of rocessor dynamic ower (4% based on the rediction for nm) is used to estimate static ower and a ortion of this ower is added to the dynamic ower trace for each functiona unit (roortiona to area). To account for the effects of temerature on static ower, the ower trace is fed to HotSot and therma simuation is erformed. The static ower for each functiona unit is then adjusted using therma deendency inearization [32]. A combination of the adjusted static ower and the dynamic ower is then used for mode training and verification using HotSot. We have found that the effects of temeraturedeendent static ower are sma over the temerature change range considered. Fig.. Fooran of the Athon 64 rocessor [3] which modes a fu out-of-order ieine with branch rediction, caches, buses, and other comonents of a modern rocessor. It can aso reort ower traces of system comonents which are used for therma simuation. The simuator was modified to suort the on-the-fy duming of erformance counter recordings which are synchronized with ower traces. The SESC simuator rovides abundant system statistics for architectura anaysis. Tabe I ists a subset of these statistics. Some simuation-reated metrics, e.g. simuation seed, are not used in our strategy since it woud not be avaiabe to a tyica many-core user at run-time. The seection of erformance counters is critica for achieving a good temerature estimation. Performance counters that give itte correation with temerature for most functiona units are excuded from the estimation. The erformance counter seection rocedure invoves a seect-and-test iteration during the mode training eriod, i.e. train the mode using a set of seected erformance counters and erform a cross-benchmark test (different benchmarks are used for training and testing) on the trained mode. During inear mode training for β arameters and for mode verification, SESC is used to record the ower trace for aications. This information is used by HotSot [24], a therma simuator, to determine actua temeratures that can be used for training or for comarisons versus therma estimates to verify our aroach (Section V-B). However, since ony dynamic ower consumtion is reorted by the simuator and static eakage ower accounts for a non-negigibe art of tota ower dissiation for submicron technoogy nodes (about 4% for our chosen node of nm), we add a static B. Therma Simuator The HotSot simuation too, which takes ower traces from SESC, target rocessor geometry and materia arameters as inuts, is used to generate accurate goden temeratures. As mentioned in the revious subsection, it is assumed that the HotSot generated temerature vaues are the actua temeratures considering the sohisticated therma diffusion mode imemented by HotSot (Fig. 9). An AMD Athon64 rocessor is used to assess our aroach. The fooran of AMD Athon64 rocessor is shown in Fig.. The rocessor incudes 24 functiona bocks, each of which is abeed in the figure. Each bock contains a therma sensor. According the rocessor secification of AMD Athon 64 fabricated under 3 nm SOI technoogy, the reorted die size is 93 mm 2 [33]. After technoogy scaing, the area of the rocessor is estimated to be 24 mm 2 in nm technoogy. The frequency of the rocessor is configured at GHz in simuation and the overa initia temerature of the rocessor is set to o C. C. Satia and Temora Noise Variations in sensor accuracy across temerature sensors on the die (satia noise) is mainy caused by rocess variation which is reativey static, so we consider satia noise to be constant for short time eriods (severa hours). Unike satia noise, variations in a secific sensor s accuracy over time (temora noise) is caused by environmenta effects ike votage and ambient temerature fuctuation, so its vaue varies for each temerature same. VI. RESULTS For training and verification of our new caibration aroach and for comarison to KF, we use the aications isted in Tabe VI from the SPEC2 and SPLASH-2 benchmark suites. SPEC2 is an industry-standardized CPU-intensive benchmark suite which incude both integer and foating oint aications. To diversify the test benchmarks, we mixed SPEC2 and SPLASH-2 in the same test sets. These benchmarks were randomy divided into four sets: Set I, Set II, Set III and Set IV. Our therma estimation mode (β vaues) was trained using benchmark sets I and II. Our modes and agorithms are verified with the remaining sets.