COMPACT THERMAL MODEL FOR THE TRANSIENT TEMPERATURE PREDICTION OF A WATER-COOLED MICROCHIP MODULE IN LOW CARBON EMISSION COMPUTING

Transcription

1 Numerical Heat Transfer, Part A, 59: , 2011 Copyright # Taylor & Francis Group, LLC ISSN: print= online DOI: / COMPACT THERMAL MODEL FOR THE TRANSIENT TEMPERATURE PREDICTION OF A WATER-COOLED MICROCHIP MODULE IN LOW CARBON EMISSION COMPUTING A. Kubilay 1, S. Zimmermann 1, I. Zinovik 1, B. Michel 2, and D. Poulikakos 1 1 Laboratory of Thermodynamics in Emerging Technologies, Department of Mechanical and Process Engineering, ETH Zürich, Zürich, Switzerland 2 IBM Research Zurich, Rüschlikon, Switzerland This article presents a compact computational model for the rapid determination of the junction temperature of a chip cooled with a heat sink, exploring the concept of hot water cooled electronics as a strategy to reduce the carbon footprint of data centers. The model aims at rapid simulations of variations of the chip, as well as the heat sink outlet water temperatures during transient heat loads. The model is validated by experimental tests with a water-cooled manifold microchannel (MMC) heat sink, which is designed to cool the processors of state-of-the-art servers. The chip temperature is determined subject to periodic heat loads as large as 100 W with frequencies in the range from 1 to 10 Hz. The results show that to calculate 1 s of real temperature variation requires less than 20 s of computational time on a Quad-Core AMD Opteron 2350, 2 GHz desktop PC with 4 GB RAM. The thermal response of the heat sink to real-time power traces with durations up to 200 s is modeled for different flow rates. The simulations indicate that application of a flow-control feedback loop could achieve more than 50% reduction in water flow rate, without compromising the maximal chip temperatures. 1. INTRODUCTION Recent developments in the high performance computer industry are characterized by an ever-increasing computational power resulting in higher energy consumption, and the pressing need to reuse the large amounts of thermal energy generated in data centers. An important strategy regarding energy reuse is the use of novel heat sink concepts at chip level to enable the removal of thermal energy Received 21 January 2011; accepted 14 March A.K. and S.Z. contributed equally to this article. We acknowledge G. I. Meijer and S. Paredes for sharing the mechanical design parameters and thermal simulations of the cold plates, which are used in the IBM hot water-cooled IBM BladeCenter 1 Server Cluster Aquasar. We also acknowledge financial support for this work by the Swiss Center of Competence for Energy and Mobility (CCEM). Address correspondence to D. Poulikakos, Laboratory of Thermodynamics in Emerging Technologies, Department of Mechanical and Process Engineering, ETH Zürich, Zürich 8092, Switzerland. dimos.poulikakos@ethz.ch 815

2 816 A. KUBILAY ET AL. NOMENCLATURE A contact area, m 2 C volumetric specific heat, J=m 3 K D hydraulic diameter, m h heat transfer coefficient, W=m 2 K K matrix with the inverse resistance coefficients as entries k thermal conductivity, W=m K L cell thickness, m N number, ( ) Nu Nusselt-number ( ) Q heat source term, W=m 3 q local heat flux, W R thermal resistance, m 2 K=W S vector specifying the heat sources for the discretized cells T temperature, K x, y, z direction, m at the highest possible temperature (and therefore exergy) levels. To this end, the hot water leaving the heat sink after chip cooling is subsequently directed through a heat exchanger to make its energy usable for another heating application (process industry, district heating, etc.). After this, the resulting lower temperature warm water is introduced back to the inlet of the heat sink as the coolant. Such heat sinks and the associated electronics operate apparently at elevated temperature levels but still within allowable thresholds. It is important to note that in this new energy conscious strategy, the elimination of chillers alone (used traditionally to cool the air before entering the board region) results in significant energy savings [1]. Currently, heat management at the board and system level is extended from the processors to various electronic parts including on-board internal memory, integrated circuits, power converters, and Ethernet switches. The growing complexity of energyaware designs requires development of models and software design tools enabling both reliable, as well as time-efficient virtual determination of important performance parameters (such as temperature) at all component, board, and system levels. Since the 1990s, compact thermal modeling was employed to predict the thermal behavior of electronic components and systems. This modeling approach focuses on the simulation of a limited set of the system parameters, which are of interest within a particular design context, and it relies on well thought out approximations of the related heat transport phenomena. A detailed review of the large number of compact models reported in the literature is beyond the scope of this introduction. Instead, we only mention representative examples reflecting different levels of complexity in the field of compact thermal modeling. The system level thermal models utilize the electro-thermal analogy, where a thermal resistance coefficient is attributed to every board component [2] and the system is considered as a network of thermal resistors. This approach is essentially zero-dimensional and each electronic part is characterized by a single temperature value. The need for evaluation of spatially nonuniform temperature distributions initiated the development of models that solve the steady-state three-dimensional heat conduction equation numerically in every element of the simulated system. In these models, the speed-up of the solution process compared to the general finite volume and finite difference algorithms is achieved by fine tuning the algebraic solvers, which take advantage of the simple block geometry of electronic components and their layered structure with constant heat conductivity within every layer [3]. Heat fluxes generated by the components change over time due to variation of the computational load on the server. In the simulations, the time-dependent

3 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE 817 temperature profile of the components is usually obtained from a numerical solution of the transient heat conduction equation. In the compact model, computational efficiency is facilitated by reducing the problem complexity by a partial discretization of the heat conduction equation [4, 5]. In this approach, spatial derivatives of the equation are approximated using finite differences on a relatively coarse grid representing the geometry of the package, while time derivatives are left intact. Thus, the heat conduction problem is reduced to a set of ordinary differential equations (ODE) simulating the temperature evolution in the grid nodes, which correspond to the rectangular blocks of the domain discretization. The high computational efficiency of the model is due to the following factors: 1) the ODE system is a sparse matrix, and 2) since heat conductivity coefficients are assumed to be constants, the equations are linear. The accuracy of the ODE thermal modeling of electronic component systems is analyzed in reference [6]. This investigation concludes that depending on the problem under study, the limited accuracy of the model may still be sufficient for the thermal analysis of the system in question. It is also shown that finer discretization of the computational domain approaches the results corresponding to solutions of finite difference and the finite element models. Increasing attention to energy reuse and to minimization of the overall system level carbon footprint, led to approaches which combine the resistor network model for solid elements with detailed simulations of heat convection and fluid flow in the heat sinks. While this multi-domain modeling proved to be a successful research method for steady state analysis [7, 8], the solution of the transient Navier-Stokes equations as a part of a package thermal model remains too time-consuming to be used in the design phase. A hybrid approach combining the ODE thermal model with an approximate treatment of the fluid dynamics problem is suggested in reference [9] for three-dimensional stacked integrated circuits. In this hybrid model, hydrodynamic and thermal layers in a set of parallel channels of the heat sink are assumed to be laminar, fully developed and can be described with the corresponding correlations for the Nusselt number and the friction coefficient. The suggested simplifications allowed for a reduction of the heat convection partial differential equations to a set of ODE similar to the equations in the thermal ODE model. In the simulations [9], the computed junction temperature compared very well with the results of a commercial CFD finite volume solver for the corresponding conjugate heat transfer problem. Miniaturization of electronic components leads to ever-increasing heat fluxes which will soon exceed the heat removal capacity of traditional cooling methods for data centers, such as those based on forced air convection [10]. Switching to liquid cooling addresses both challenges by rendering most of the additional components for air cooling such as computer room air conditioners obsolete and reducing the thermal resistance by at least a factor of 5 [11]. The concept of liquid cooling is now well established in the literature, starting from the work of Tuckerman and Pease [12], with steady progress accomplished since in the thermal performance of the heat sinks. Manifold micro-channel heat sinks using a hierarchical design to supply and collect the coolant from micro-channel structures are proposed as an improvement by Escher et al. [13, 14], who optimized the manifold and the micro-channel design for an ultra-thin heat sink. Liquid cooling with high coolant temperature improves the exergetic efficiency and enables energy re-use as mentioned earlier. Such an energy reuse strategy can

4 818 A. KUBILAY ET AL. minimize the overall system level carbon footprint by reducing the power consumption of a data center and eliminating the requirement of fossil fuels for building heating [15]. In reference [16], this concept was explored experimentally by demonstrating the feasibility of hot water cooled electronics as a strategy to reduce the carbon footprint of data centers and to enhance the exergetic efficiency of the cooling unit. CFD simulations [17] showed that the complicated hierarchical structure of a microchannel (MMC) heat sink induces a flow field with high turbulent intensity accompanied by an increased pressure drop. High pressure drops across the heat sink require an adequate increase of water pumping power. Therefore, a promising measure to further reduce the energy consumption of water cooled servers is to tailor the water pumping power to the time-dependent heat generation in the chips which are running software applications. The prediction of the performance of these new generation heat sinks demands efficient compact thermal models that are able to simulate the temporal evolution of the packages during relatively long time spans typical for running server software. The goal of the present study is to develop and experimentally validate such a compact model of the heat transfer in an MMC heat sink for time-efficient simulations of the transient temperature during run-time of typical software applications. The model is also applied for simulations of a flow rate controller in combination with chip power traces typical for runs of a compiler and a session of an internet browser. 2. MODEL DEVELOPMENT The heat sink (Wolverine Tube Inc., Huntsville, USA) described in detail elsewhere [16, 17] consists of a micro-channel coldplate connected to a manifold layer, which is oriented perpendicular to the micro-channels in the coldplate. The MMC heat sink is shown schematically in Figure 1, where the arrows depict the direction Figure 1. Schematic of the heat sink with flow direction indicated by arrows.

5 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE 819 of the coolant water flow. The coolant water is fed centrally into the inlet manifold, from which it is directed to the micro-channel structure through a slot nozzle in the bottom wall. The slot nozzle induces jet impingement of the coolant on the micro-channel enhancing the heat transfer performance of the heat sink. Due to the symmetry of the design, the coolant evenly branches to both sides and leaves the coldplate laterally through the two collection manifolds. The two streams rejoin and leave the heat sink through the outlet port. The chip cooled by the MMC heat sink is attached to the sink below the micro-channel structure with a thermal interface material (TIM) placed on the top of processor chips (IBM BladeCenter 1 Server, HS22 and QS22). Since the chip temperature is critical and should not exceed the threshold specified by the industrial standards, thermal modeling of the package has to reliably predict a characteristic chip temperature generated when it is running a computation. On the other hand, assessment of the potential energy reuse requires simulation of the average water temperature of the heat sink. A thermal model has to consider heat transfer in the four compartments of the package. In the chip generating heat, in the TIM, in the MMC heat sink, and in the cooling water inside the heat sink. In our model, the temperature fields in the chip, TIM, and the solid part of the heat sink are computed using the heat diffusion equation with a source term for heat generation in the chip. Detailed modeling of the heat convection in the water flow inside the MMC heat sink requires solving the Reynolds-averaged turbulent flow Navier-Stokes equations [17]. Since the rapid solution of the Navier-Stokes equations for three-dimensional unsteady turbulent flows inside the complex heat sink structure is computationally unfeasible, a semi-empirical approach is adopted in the suggested model of the package. In the model, we compute the heat dissipated to the water and the corresponding average temperature of the flow at the sink outlet based on a given inlet water temperature, flow rate, and a heat transfer coefficient specified at the internal walls of the MMC heat sink. The latter is an area-averaged heat transfer coefficient at the fluid-solid interface and is a fitting parameter, which has to be obtained via calibration of the model. The second similar fitting parameter of the model is an average heat transfer coefficient which defines the heat transfer at the external boundaries of the package. The two fitting parameters serve as an input to the boundary conditions of the heat diffusion equation for the solid parts of the package. In order to obtain a fast and robust solution, the unsteady heat diffusion problem was reduced to a system of differential equations for an equivalent thermal resistance network following the approach which is employed in the ODE thermal compact modeling concept [4, 5]. A simplified heat sink geometry composed of seven compartments is utilized in the model. Four compartments (numbered 1 4 in Figure 2a) represent the water flow path consisting of the three main channels (1 3) and the micro-channel array below them (4). The remaining three compartments (5 7 in Figure 2a) correspond to the solid parts of the package. The chip, the TIM, and the solid material of the heat sink. The heat conduction problem is solved in the solid parts of the package excluding the micro-channel array and the water path. The governing equation for the temperature field T inside the computational domain is written as follows. C qt qt ¼rkrT þ Q ð1þ

6 820 A. KUBILAY ET AL. Figure 2. (a) Model geometry of package with MMC heat sink. 1 3: water flow path manifolds, 4: micro-channels, 5: TIM, 6: Chip; and 7: MMC cooper compartment. (b). Example of discretized layer with equivalent thermal resistance network. where C, k, and Q are the volumetric specific heat of the material, the thermal conductivity, and a heat source term, respectively. The thermal conductivity and the heat capacity used in the simulations for the chip, TIM, and the solid material part of the heat sink are shown in Table 1 together with the geometrical parameters of the computational domain. Every solid part of the geometry (compartments 5 7 in Figure 2a) is subdivided into a set of horizontal layers which are discretized with rectangular cells. The partial differential equation of heat conduction (1) is then reduced to a system of ordinary differential equations by applying the finite volume formulation for the spatial variables. Due to the well-known analogy between heat and electrical conduction, the system of ordinary differential equations is identical to the governing equations for an electrical circuit where voltage and electric current represent temperature and heat flux, respectively. An example of a discretized layer with the rectangular cells and the equivalent two-dimensional thermal resistance network is shown in Table 1. Model geometry and material parameters Chip TIM Heat sink Width mm Length mm Thickness 16 mm Width mm Length mm Thickness 16 mm Width 47.5 mm Length 47.5 mm Thickness: Base 1.2 mm Thickness: Microchannels 1.7 mm Thickness: Manifold 4 mm Thickness: Uppercap 2 mm Thermal conductivity 130 W=m K Heat capacity J=m 3 K Thermal conductivity 3.73 W=m K Heat capacity J=m 3 K Thermal conductivity 380 W=m K Heat capacity J=m 3 K

7 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE 821 Figure 2b. If the node temperature in the example grid is denoted as T mn, the equation for this node can be written as follows. CDxDyDzðdT mn =dtþ ¼kDyDzðT m 1n T mn Þ=Dx þ kdydzðt mþ1n T mn Þ=Dx þ kdxdzðt mn 1 T mn Þ=Dy þ kdxdzðt mþ1n T mn Þ=Dy þ qdxdydz ð2þ where Dx, Dy, and Dz denote the dimensions of the cell. The system of equations (2) for every node describes a thermal resistance network with the following node thermal resistance. R ¼ L=kA ð3þ where k is the thermal conductivity of the cell material, L is the thickness of the rectangular cell, and A is the contact area between neighboring cells. In this model, the node l resistance defined in Eq. (3) represents the thermal resistance to the heat flow at the position of the node in one of the six directions parallel to the axis coordinates. Depending on the direction, the thickness of the cell L equals Dx, Dy,orDz, respectively. The discretized geometry of the package results in a set of N rectangular elements with a grid at the centers of the elements for which all equations are written in vector form as follows. C _T ¼ KT þ S where T is the N-dimensional vector representing the temperature in the N grid nodes, and C and S are the vectors specifying the thermal capacitances and the heat sources for the discretized cells, respectively. The entries of the matrix K are the inverse resistance coefficients 1=R, where the resistances R are defined in Eq. (3). Since every cell is connected to six neighbors with mutual contact interfaces, the matrix K is a Laplacian matrix with only six entries in each row (column). In the source term, the vector elements represent either the heat generated in the chip or the heat flux at the boundaries of the domain. The local heat flux at the external boundary of node i with temperature T i is calculated as follows. ð4þ q ext ¼ h ext AðT a T i Þ ð5þ where h ext is the external heat transfer coefficient, and T a is the ambient temperature. The convective heat flux at the fluid-solid interface in the MMC heat sink is defined in the same way. q int ¼ h int AðT i T inlet Þ ð6þ where h int is the heat transfer coefficient for the internal boundaries, and T inlet is the inlet water temperature. The resulting ODE system is numerically integrated to obtain the time dependent temperature at all nodes of a grid which represents a discretization of the geometry of the MMC heat sink and the attached chip including

8 822 A. KUBILAY ET AL. TIM. The solution is obtained using subroutine ode45 (MATLAB 2009b), which employs an explicit Runge-Kutta algorithm. 3. MODEL VALIDATION The estimation of the two unknown heat transfer coefficients is carried out iteratively based on the results of a series of the experimental tests reported in reference [16]. Schematics of the flow loop and the actual test section designed to evaluate the MMC heat sink performance are shown in Figures 3a and 3b. The fluid inlet temperature T f,in was controlled using a heat exchanger, which is connected to a separate flow loop where the temperature is regulated with accuracy of 0.1 C using a heater= chiller (Proline RP 855, Lauda, Germany). The flow was measured with an accuracy of 0.2% for entire range ( l=min) of operation using a Coriolis flow meter (Emerson, Switzerland). A differential pressure sensor (Honeywell, USA) and two thermocouples (Omega Engineering Inc., USA) were used to measure the pressure drop with a precision of bar, and inlet and outlet temperatures with a cross calibrated accuracy of 0.1 C. This precise calibration was needed for the evaluation of the heat sink efficiency because the relative errors increased significantly due to relatively low water temperature rise at high water flow rates. For example, an error of 0.1 C already corresponds to a 7 W heat flux error at a water flow rate of 1 l=min, which is significant given the 130 W maximum thermal load associated with the chip (IBM BladeCenter 1 Server, HS22). A 7 mm pore filter (Swagelok, Solon, USA) was used to keep the coolant free of large particles. Additionally, 14 integrated resistance temperature detectors (RTDs) in the heating test chip (2.31 cm 2 ) were used to determine the temperature field of the chip. Two different types of thermal greases were applied as thermal interface material (TIM) between the chip and the heat sink, Figure 3. (a) Test section for MMC heat sink performance evaluation; and (b) schematic of the experimental flow loop.

9 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE 823 which were attached together using spring loaded screws. The spring loading maintained a constant force of N on to the chip. The thermal grease improved the heat spreading and helped make a good thermal connection. The average thickness of the TIM layer was measured with an accuracy of 1.5 mm using the average value of four inductive length probes (P2001, Mahr, Goettingen, Germany). These probes, located at the four edges, were a fixed part of the holder. In the tests, the temporal dynamics of the chip and water temperature exhibit three different phases. In the beginning of a test, the chip load is increased gradually causing an increase of the chip temperature, and the fluid temperature in the water loop. When the heat load reaches its maximum and the heat transfer in the package is at equilibrium, the system attains a steady-state with the chip and water temperature remaining constant. In the final phase, the heat load is turned off abruptly leading to a sharp drop of the chip temperature. The results of the temperature measurements in the steady-state phase of the tests serve as calibration data to determine the two model heat transfer coefficients at the boundaries. After the calibration, the model is applied for predicting the transient behavior of the system, which is observed at the beginning and at the end of the test. The model calibration is carried out in two steps. In the first step, the heat dissipated to water and the average temperature of the chip surface are calculated for a set of values of the two fitting parameters. In the second step, the parameters are adjusted for every flow rate to reproduce the chip temperature and the dissipated heat measured in the tests. An example calculation for the chip heat load of 90 W is shown in Figure 4. It is assumed that the values of the Nusselt number in the main channels and in the micro-channels of the manifold are the same and the heat transfer coefficient h int Figure 4. Heat dissipated by water (closed symbols) and chip temperature (open symbols) for a set of model fitting Nusselt number, and external heat transfer coefficient h ext.1)60w=m 2 K, 2) 90 W=m 2 K, and 3) 120 W=m 2 K.

10 824 A. KUBILAY ET AL. at the solid-water interface is calculated by h int ¼ Nu k=d, where k is the heat conductivity of water and D is the hydraulic diameter of the channels. In the calculations, the change in Nusselt number was set to span over the range that was determined in the CFD simulations of the tested MMC heat sink [17]. The results in Figure 4 (see closed symbols) indicate that the value of heat dissipated to the water was only marginally affected by the choice of the Nusselt number and remained almost constant for Nu > 2. The simulations show that the increase of the heat transfer coefficient defining the boundary condition of the problem enhances heat flow at the boundary. The enhancement leads to a lower temperature level T i at the wall and a corresponding decrease of the local heat flux defined by Eq. (5). The weak dependence of the dissipated heat on the Nusselt number allows to define the value of the external heat transfer coefficient h ext as a first step of the model fitting procedure. It was found that the calculated heat dissipated by water corresponds to the measured value within 5% if h ext ¼ 90 W=(m 2 K). For the experimental setup, the value of h ext represents a lumped thermal resistance, which combines the heat transfer through the holder of the MMC heat sink and the natural convection at the external interfaces of the package. In a second step of the calibration, the value of the internal heat transfer coefficient is adjusted until the calculated temperature of the chip coincides with the measured chip temperature. The resulting dependence of the fitted Nusselt number on the flow rate is shown in the Figure 5 along with the data used for the calibration. The increase of the Nusselt number for higher flow rates (see closed symbols in the figure) reflects an enhancement of the heat transfer due to the disruption of boundary layers in the heat sink channels and intensification of the flow turbulence that was also observed in the CFD simulations of the MMC heat sink [17]. The calibrated model was applied to predict the outlet water temperature in the tests with flow rates in the range from 0.5 l=min to 1 l=min and an inlet water temperature of 60 C. In these tests, the water temperature measured at the outlet varied from Figure 5. Fitted Nusselt number (closed symbols) and measured chip temperature (open symbols) used for the fitting for different flow rates in MMC heat sink with h ext ¼ 90 W=m 2 K.

11 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE C to 62.2 C and the predicted values deviated from the data by less than 0.1 C corresponding to the accuracy of calibration of the sensors in the setup. The sensitivity of the model to discretization of the domain was checked by computing the maximum temperature of the chip for grids with the number of nodes increasing from 2,000 to 60,000. In these runs, the maximum of the chip temperature was decreasing from C to C with a change less than 0.1 C after the node number exceeded 4,000. Since the suggested compact model is not focused on reproducing smooth spatial temperature distribution in the package, but is intended to provide time efficient estimates of transient dynamics of the characteristic parameters of the system, most of the simulations are carried out with a 8,000 node grid. An additional series of simulations was performed to assess the impact of the discretization used in the model on calculations of the average and maximum temperature of the chip. In this series, the heat load was locally adjusted in the individual nodes until the chip temperature computed at the 14 locations of the RTD s matched the measurements obtained in the experiment. The iterative procedure of the adjustments resulted in a reversed engineering power map that was composed of a hot spot W=cm 2 located slightly off center of the chip, four adjacent zones with heat load W=cm 2 and the load of W=cm 2 for the rest of chip. The maximum temperature and the average temperature measured in the test were C and C with the computed temperatures being only slightly higher: 1.6 C for the maximal temperature and 0.2 C for the average temperature, respectively. The validation of the model in transient conditions was carried out using the unsteady phases of the experimental testing. The simulations span a 200 s interval, which is the typical run-time of a C-compiler and internet browser executing a set of standardized software tests [18]. Calculated temperature contours in a cross-section of the package after the heat load was turned off are shown in Figure 6. The contour plots show that the temperature on the chip and TIM drops rapidly by 10 C within 0.1 s. In the simulations, the average chip temperature was predicted within 0.5 C during both gradual increase of the heat load and its abrupt turnoff (see Figure 7). The heat dissipated to the water was calculated with accuracy better than 5% during both gradual increase and abrupt turnoff of the heat load (see Figure 8). In the latter case, the simulations accurately reproduced a negative heat flux when the rapidly cooled heat sink was effectively heated by still warm water Figure 6. Temperature contours with heat load turned off at t ¼ 0 s. The two lower layers corresponding to the chip and TIM are not in scale (extended in the vertical direction).

12 826 A. KUBILAY ET AL. Figure 7. Average chip temperature when heat load 90 W is turned on (experimental data: closed symbols, simulation: thick solid line; right axis) and off (data: open symbols, simulation: thin solid line; left axis). pumped through the heat sink. The model was sensitive enough to capture the change of water temperature when the actual flow rate in the test was changed from 0.5 to 0.7 l=min (see slight decrease of the dissipated heat at t ¼ 125 s). The validation shows that the compact model presented in this work can rely on steady-state calibration to successfully predict transient behavior of water and Figure 8. Heat dissipated by water when heat load 90 W is turned on (data: closed symbols, simulation: thick solid line; right axis) and off (data: open symbols, simulation: thin solid line; left axis).

13 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE 827 average chip temperature of the package with the MMC heat sink. It is also important to notice that the ratio of the model run time on a desktop PC divided by the simulated phenomenon time does not exceed 20:1, i.e., 1 s of the real time interval requires less than 20 s of computation time on a desktop PC. This ratio of physical to simulation time is several orders of magnitude better than the ratio in transient CFD runs or transient compact thermal modeling [9], which focus on detailed simulation of spatial temperature fields in all package components. 4. OSCILLATING HEAT LOAD The measurements of power traces of various software packages running standard benchmarks show that most of the time the heat load of the computer CPU oscillates between 50 and 70% of its maximum value with the frequency of oscillations between 1 and 10 Hz [19]. In water-cooled microelectronic packages the heat load oscillations cause a transient response of the chip temperature, of the water temperature, and corresponding oscillations of the heat removed by the water in the heat sink. To exemplify the dynamic response of the MMC package to an oscillating heat load, a series of simulations were carried out for a set of model heat loads. In the simulations, the model heat load was specified as a sequence of rectangular pulses with a given frequency and amplitude. The induced response of chip temperature and dissipated heat reached quasi-periodic oscillations within 20 s. An example of the heat load oscillating between 40 and 90 W is shown in Figure 9. The frequencies of three consecutive sections of the load are 0.4 Hz, 1 Hz, and 4 Hz with pulse periods of 2,500 ms, 1,000 ms, and 250 ms, respectively. The results indicate that due to the high thermal conductivity of the heat sink material and the efficient heat transfer between heat sink and cooling water, the response time of the package is relatively Figure 9. Model heat load for simulations of transient response of the package with MMC heat sink.

14 828 A. KUBILAY ET AL. Figure 10. Maximum chip temperature for oscillating heat load with inlet water 60 C and flow rate 0.5 l=min. Dashed lines depict maximum temperature for corresponding steady-state heat loads. short and the chip temperature oscillates with the same frequency as the heat load (see Figure 10). The maximum of the oscillating chip temperature reaches the same level as in the steady-state case with the corresponding maximum heat load if the load frequency is low. When the frequency increases to 4 Hz, the maximum chip temperature is about 1 C lower than in the steady-state simulations (see dashed lines in Figure 10), and this difference (marked by d in Figure 10) rises with the frequency. The heat dissipated to water exhibits similar qualitative behavior as is observed for the chip temperature (see Figure 11). In both cases, there is a frequency threshold above which the maximum of the chip temperature as well as the water temperature at the outlet are below the corresponding levels for steady-state conditions. The results show that the relative deviation of the dissipated heat is greater than the deviation of maximum chip temperature. The limiting factor responsible for the damped response of the removed heat to oscillating loads is the intensity of heat transfer between the sink material and water. The average heat flux to water is not as sensitive to the heat load oscillations as the flux at the TIM-heat sink interface, since the heat sink has a large fluid-solid interface, which leads to relatively small amplitudes of temperature oscillations at the wall surface. The impact of the oscillation frequencies on the deviation d of the maximum chip temperature from the steady-state case is illustrated in Figure 12. For frequencies lower than 2 Hz, the chip temperature always reaches its steady-state level while at 10 Hz, the maximum temperature oscillation is 1.4 C lower than for the steady-state run. The variation of flow rate in the range from 0.5 l=min to 1.0 l=min affects only marginally (less than 0.5 C) the value of deviation for all specified frequencies up to 10 Hz (see Figure 12). The chip temperature with the steady-state heat loads is also not very sensitive to the flow rate change. In the experiments, an increase of the flow rate by a factor of two caused only a two degree drop of the average chip temperature from 72.5 C to 70.5 C (see Figure 5). On the other hand, the pumping power

15 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE 829 Figure 11. Heat dissipated by water for oscillating heat load with inlet water 60 C and flow rate 0.5 l=min. Dashed lines depict dissipated heat for corresponding steady-state heat loads. required to sustain a given flow rate scales faster than the flow rate. Thus, one possible strategy to minimize the overall energy consumption needed for operating a computer server cooled by water is to keep a minimal flow rate which is adjusted on-the-fly to compensate for spikes of the chip heat load. The following examples present applications of the compact model to the simulation of the MMC package assuming that the inlet water temperature or the water flow rate is controlled via a simple feedback loop in order to minimize the energy consumption. Figure 12. Deviation d of maximum chip temperature from corresponding steady-state case with constant heat load: 1) 0.5 l=min, 2) 0.7 l=min, and 3) 1.0 l=min.

16 830 A. KUBILAY ET AL. 5. ON/OFF CONTROLLER FOR INLET WATER TEMPERATURE The simplest feedback loop in control systems is modeled as an on=off controller with two states and one adjustable parameter. The controller states and the value of the parameter are changed every time when the controlled variable passes one of two given thresholds. Such controllers, in which continuous dynamics of the controlled variable interacts with the discreet dynamics of the system states, are studied extensively in the context of hybrid control systems [20, 21]. A schematic of the on=off controller adapted to the studied package with the MMC heat sink is shown in Figure 13. The controller works as a thermostat, which can switch the inlet temperature of the water from a low to a high level and vice versa as soon as the maximum chip temperature is passing a lower or upper threshold, respectively. In the model equations, a corresponding step function was implemented to model the switching of the inlet temperature depending on the maximum temperature calculated in the chip. In systems with an on=off thermostat, the frequency of temperature oscillations depends on the choice of the system thresholds as well as the levels of the adjustable parameters. In the case of a constant heat load, the simulations of the package with the MMC heat sink show that the frequency is proportional to the difference between the thresholds and inversely proportional to the difference between the parameter levels. An example of the transient chip temperature in the package equipped with the on=off controller and constant heat load 90 W is shown in the left plot of Figure 14. The maximal chip temperature oscillates periodically with the frequency 3.3 Hz between the specified controller thresholds 75 C and 80 C. In case of an oscillating heat load, the resulting profile of the chip temperature is a superposition of the oscillations defined by the characteristic frequency of the controller, the heat load frequency and the phase shift between the two. The right plot in Figure 14 shows the oscillating temperature of the chip for the heat load specified as a sequence of rectangular pulses with the frequency of 1 Hz and amplitudes from 0 to 90 W. In this simulation, the superposition of the heat and controller oscillations induces periodic changes of the chip temperature with a double peak profile. Since the maximum of the oscillating heat load is the same as in the case of the constant Figure 13. Schematic representation of the states of on=off controller.

17 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE 831 Figure 14. Chip temperature in package with on=off controller for constant heat load 90 W (left), and heat load oscillating from 0 to 90 W with frequency 1 HZ (right). Inlet water temperature switches from 60 Cto 70 C. heat load, the chip temperature does not exceed the upper threshold. In contrast, cooling the heat sink during the idle phase of the heat load leads to a decrease in the chip temperature below the lower threshold of the controller. The simulations demonstrate that the adjustment of the water temperature to a lower level during phases with smaller computational loads of the chip could be a method to save energy due to a lower overall demand for hot water flow. The implementation of an on=off controller is based on the assumption that the changes of the inlet temperature affect the entire fluid-solid interface in the heat sink immediately. The assumption is valid for the controllers which have a negligibly small hydraulic residence time, i.e., the time required to fill the system with water with the adjusted temperature is relatively small compared to the characteristic time of the problem. If the residence time has to be taken into account, a delay parameter should be added into the model of the controller. Hardware to manipulate water flow rates is much simpler than equipment to adjust the water temperature and is thus an attractive possibility to reduce pumping power and, consequently, the overall energy consumption of the system. In order to illustrate the possible impact of flow rate controllers on the package chip temperature, a flow rate feedback loop was incorporated into the developed model of the package. 6. PROPORTIONAL CONTROLLER FOR WATER FLOW RATE: SIMULATION OF THERMAL RESPONSE WITH C-COMPILER AND INTERNET BROWSER REAL TIME POWER TRACES In the MMC heat sink, the flow rate affects the formation of the thermal boundary layers and the corresponding heat transfer efficiency at the inner walls of the sink (see Figure 5). Since the wall heat transfer coefficient is proportional to the water flow rate, the flow rate controller of the package increases the flow rate according to the increase of the computational load on the chip. The results in

18 832 A. KUBILAY ET AL. Figures 10 and 12 show that the chip temperature for transient heat loads does not exceed the temperature in steady-state cases, whereas the maximum for both cases was set to the same power level. Consequently, in a conservative controller the instant flow rate corresponding to a specific value of transient heat load should be at the level which is sufficient to keep the chip temperature below the safety threshold in the steady-state conditions with the same heat load. A series of simulations of the package with steady-state heat loads was carried out to determine the flow rates which guarantee the maximum chip temperature to remain at a safe operational level. The results show that if the flow rate increases with a slope of 1 l=min per 40 W heat load, the maximum chip temperature remains at 72 C up to a heat load of 100 W. Based on the simulations, the following feedback loop was implemented in the package model. If the transient chip heat load remains below 65 W, the flow rate is kept at a base level of 0.5 l=min otherwise the flow rate is increased proportional to the heat load above 65 W. The slope of the proportional increase is set to 1 l=min per 40 W as it was determined in the steady-state simulations. In the model, the implementation defines the internal heat transfer coefficients h int at the source terms of the equations for temperature in the nodes of the internal walls. At every node, the time-dependent value of the coefficient is calculated as a function of the instant flow rate accordingly to the calibration curve shown in Figure 5. In order to investigate the thermal behavior of the MMC package for realistic power traces, the model heat load of the chip is imitating the heat loads in CPUs during typical software applications. The model heat load for simulations with the controller is based on power traces measured in reference [18] in an Intel P4 chip running open source browser Mozilla and C-compiler GCC, as shown in Figure 15. In the model equations, the heat load profiles define the source terms in the equations for the grid nodes representing the chip of MMC package. The power trace reaches 80 W for the running compiler and 100 W for the running browser. If the MMC Figure 15. Chip heat loads corresponding to power traces of Mozilla (solid line) and C-compiler GCC (dashed line).

19 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE 833 Figure 16. (a) Chip temperature and (b) water temperature at outlet for power traces of compiler with proportional controller (solid lines 1) and with constant flow rate 1 l=min (dashed lines 2). package works without the controller, the corresponding steady-state flow rates that are required to keep the chip temperature at 72 C will be 0.6 l=min and 1.0 l=min, respectively. On the other hand, the controller enforces a proportional increase of flow rate above 0.5 l=min only when the load exceeds 65 W. Subsequently, the time-averaged flow rate of the package with the controller is lower and the average flow rates are 0.52 l=min and 0.51 l=min for the compiler and browser, respectively. The results of the simulations of chip temperature and water temperature at the outlet for the browser power trace are shown in Figure 16. In both cases with and without the controller, the maximum chip temperature is below the chosen safety threshold of 72 C. The mean values of the chip and water temperature with controller only marginally (<0.6 C) exceed the temperature with the steady-state flow rate. Figure 17. Chip temperature for power traces of Mozilla: 1) controller with min flow rate 0.3 l=min, 2) controller with min flow rate 0.5 l=min, and 3) constant flow rate 1 l=min.

20 834 A. KUBILAY ET AL. Similar results are obtained for the power trace of the compiler. The maximum chip temperature for two controllers with a different base level flow rate is compared to the chip temperature without the controller in Figure 17. The simulations show that even if the base level of the controller is decreased from 0.5 l=min to 0.3 l=min, the average chip temperature rises less than 0.4 C and remains below the safety threshold. At the same time, the average flow rate of the second controller is as low as 0.33 l=min compared with 1.0 l=min in the steady-state case. While the simulations include an idealized model of the flow controller, the results demonstrate that the significant (more than 50%) flow rate reduction can be exploited to minimize pumping power consumption, and thus to reduce the overall carbon footprint of data centers cooled by the warm to hot water return flow from a space heating system. 7. CONCLUSION The goal of this study was to evaluate the feasibility and performance of a compact heat transfer model approach for a hot water-cooled heat sink utilized to explore a new strategy to reduce the carbon footprint of data centers. A compact model for an example package with a MMC heat sink was developed and experimentally validated using transient temperature measurements for the package. It is shown that a MATLAB desktop implementation of the model allows for time-efficient simulations with a ratio of real thermal phenomenon time to needed computer simulation time of the same less than 1:20. The model is applied to simulate the thermal response of the package to heat loads, which are typical for standard software runs lasting up to 200 s. The simulations indicate that the application of a flow-control feedback loop can lead to a more than 50% reduction in water flow rate, and thus to the corresponding savings of pumping power without compromising allowable industrial specifications for maximum chip temperature. The rapid simulation run-time proves that the developed model can be used as a time-efficient simulation tool for aggressive dynamic power management of microelectronic packages designed to cool the processors of state-of-the-art computer servers. REFERENCES 1. IDC Corporate USA, Farmingham, MA, # , The Impact of Power and Cooling on the Datacenter Infrastructure, C. Lasance, Ten Years of Boundary-Condition-Independent Compact Thermal Modeling of Electronic Parts: A Review, Heat Transfer Eng., vol. 29, no. 2, pp , P. Li, L. T. Pileggi, M. Asheghi, and R. Chandra, Efficient Full-Chip Thermal Modeling and Analysis, Proc. ICCAD, pp , W. Huang, S. Ghosh, S. Velusamy, K. Sankaranarayanan, K. Skadron, and M. R. Stan, HotSpot: A Compact Thermal Modeling Methodology for Early-Stage VLSI Design, IEEE Trans. on Very Large Scale Integration (VLSI) Systems, vol. 14, no. 5, p. 501, W. Huang, K. Sankaranarayanan, K. Skadron, R. J. Ribando, and M. R. Stan, Accurate, Pre-RTL Temperature-Aware Design Using a Parameterized, Geometric Thermal Model, J. IEEE Trans. on Computers, vol. 57, no. 9, pp , D. Fetis and P. Michaud, An Evaluation of Hotspot-3.0 Block-Based Temperature Model, Proc. WDDD, In Conjunction with ISCA, available at < caps/people/michaud/hs.pdf> (accessed September 2010).

21 TEMPERATURE OF A WATER-COOLED MICROCHIP MODULE R. Gupta and V. Eswaran, Integrating Compact Thermal Models in CFD Simulations of Electronic Packages, J. Electronic Packaging, vol. 130, pp , R. Wälchli, T. Brunschwiler, B. Michel, and D. Poulikakos, Combined Local Microchannel-Scale CFD Modeling and Global Chip Scale Network Modeling for Electronics Cooling Design, Int. J. Heat Mass Transfer, vol. 53, pp , A. Sridhar, A. Vincenzi, M. Ruggiero, T. Brunschwiler, and D. Atienza, 3D-ICE: Fast Compact Transient Thermal Modeling for 3D ICs with Inter-tier Liquid Cooling, Proc. Int. Conf. Computer-Aided Design (ICCAD 2010), vol. 1, no. 1, pp , G. I. Meijer, Cooling Energy-Hungry Data Centers, Science, vol. 328, pp , E. G. Colgan, B. Furman, M. Gaynes, W. S. Graham, N. C. LaBianca, J. H. Mageklein, R. J. Polastre, M. B. Rothwell, R. J. Bezama, R. Choudhaky, K. C. Marston, H. Toy, J. Wakil, J. A. Zitz, R. R. Schmidt, A Practical Implementation of Silicon Microchannel Coolers for High Power Chips, IEEE Trans. on Components and Packaging Technologies, vol. 30, no. 2, pp , D. B. Tuckerman and R. F. W. Pease, High-Performance Heat Sinking for VLSI, Electron Device Lett., vol. 2, no. 5, pp , W. Escher, et al., Experimental Investigation of an Ultra-Thin Manifold Microchannel Heat Sink for Liquid-Cooled Chips, ASME J. Heat Transfer, vol. 132, no. 8, pp , W. Escher, T. Brunschwiler, B. Michel, and D. Poulikakos, A Novel High Performance, Ultra Thin Heat Sink for Electronics, Int. J. Heat Mass Transfer, vol. 31, no. 4, pp , T. Brunschwiler, B. Smith, E. Ruetsche, and B. Michel, Toward Zero-Emission Data Centers through Direct Reuse of Thermal Energy, IBM J. Research & Devlopement, vol. 53, no. 3, p. 11, S. Zimmermann, M. K. Tiwari, F. Ott, B. Michel, I. Meijer, S. Paredes, and D. Poulikakos, Experimental Investigation of a Hot Water-Cooled Heat Sink for Efficient Data Center Cooling: Towards Electronic Cooling with High Exergetic Utility, Proc. 2nd European Conf. Microfluidics Microfluidics 2010, Toulouse, France, P. Kasten, S. Zimmermann, M. K. Tiwari, B. Michel, and D. Poulikakos, Hot Water-Cooled Heat Sinks for Efficient Data Center Cooling: Towards Electronic Cooling with High Exergetic Utility, Frontiers in Heat and Mass Transfer, vol. 1, p , C. Isci and M. Martonosi, Runtime Power Monitoring in High-End Processors: Methodology and Empirical Data, Proc. 36 Int. Symp. Microarchitecture, San Diego, CA, December F. J. Mesa-Martinez, E. K. Ardestani, and J. Renau, Characterizing Processor Thermal Behavior, ACM SIGPLAN Notices, vol. 45, no. 3, pp , R. Alur, C. Courcoubetis, T. A. Henzinger, and P.-H. Ho, Hybrid Automata: An Algorithmic Approach to the Specification and Verification of Hybrid Systems, Lecture Notes in Computer Science, vol. 736, pp , J. Lygeros, An Overview of Hybrid System Control, in D. Hristu-Varsakelis and W. Levine (eds.), Handbook of Networked and Embedded Control Systems, pp , Computational Sciences and Engineering, Boston, Birkhauser, 2005.