Modeling and Metrology for Expedient Analysis and Design of Computer Systems

Transcription

1 Modeling and Metrology for Expedient Analysis and Design of Computer Systems Cullen Bash Chandrakant Patel Hewlett-Packard Laboratories Hewlett-Packard Laboratories 1501 Page Mill Road, M/S 3U Page Mill Road, M/S 3U-7 Palo Alto, California Palo Alto, California Phone: Phone: Fax: Fax: Abstract Recent increases in component level power dissipation have elevated system level thermal analysis to an important and increasingly indispensable role in the design of computers. Indeed, in the near future, even low end desktop personal computer workstations will contain microprocessors that will dissipate more than 100 Watts of heat. With design cycles as small as six months, coupled with an increase in local and system power dissipation, it is critical to have a thermal design tool that will allow the engineer to efficiently and accurately predict system level thermal performance before the prototype phase has begun. In 1997, an experimental method, dubbed Thermo-Volume Resistance, was developed by Chandrakant Patel and Christian Belady of Hewlett-Packard Co 1. This technique allows a computer component (such as heat sink, power supply) to be characterized thermally and hydrodynamically by relating the heat transfer and pressure drop characteristics of the component to the volume flow rate through the component. This paper extends their experimental methodology to the modeling of computer systems and demonstrates how the engineer can quickly obtain and use Thermo-Volume Resistance data to predict system level thermal performance by coupling this data with standard computational fluid dynamics (CFD) packages. By employing this technique, dense, tightly packed system components can be accurately modeled with a coarse computational grid that significantly reduces the time to numerical convergence. To demonstrate this methodology, the thermal design of a Hewlett-Packard quad processor server is described in this publication. Experimental and analytical results are discussed and the accuracy of the method is assessed. Key words: Compact Modeling, Thermal Design, System Analysis, and Computational Fluid Dynamics. 1. Introduction Increased microprocessor power dissipation and decreased computer system design cycles have brought about the requirement for a thermal design tool that will allow the design engineer to quickly and accurately assess the performance of a heat sink in a system. Currently, the data provided by many heat sink vendors take the form of a thermal resistance ( C/W) versus an approach velocity. This thermal resistance is typically measured in a non-standard wind tunnel and provides results that are only valid for the particular test environment in which the measurement was taken 1. The data are useful for comparing the performance of heat sinks that have undergone identical testing, but are rendered invalid when the heat sink is taken out of the test environment and placed in a computer system that contains multiple complex airflow paths. Consequently, the design engineer must increasingly rely on CFD tools to accurately predict heat sink performance within a system. Dense heat sinks contain a large amount of surface area that is used to transfer the heat from the component to the cooling fluid. In addition, the density of the heat sink imparts a pressure drop to the flow moving through it. To accurately model this heat transfer and pressure drop, the temperature and velocity gradients from the surface of the heat sink to the fluid must be adequately captured using CFD. Since CFD tools solve the discrete Navier-Stokes equations only at specified points in the spatial domain of the model, it is necessary for the analyst to create a mesh within the solution domain. 157

2 Intl. Journal of Microcircuits and Electronic Packaging The mesh consists of a network of nodes and the Navier-Stokes equations are solved only at the nodes of the defined mesh 2. Therefore, to capture the heat transfer from a surface, the mesh must be denser near the surface than in the free stream, where temperature and velocity gradients are not as dramatic. Due to the large surface area of dense heat sinks, it becomes necessary to create a very large mesh to resolve the heat transfer and pressure drop within the heat sink (usually much greater than 100,000 nodes at the system level). In fact, one study suggests as many as five grid cells are required between each fin to fully resolve the velocity gradient 3. Models of this size and detail take hours if not days to converge even on the fastest workstations and make parametric studies impractical to perform. Figure 1 shows a typical system, a Hewlett-Packard four-way server, that would require thermal modeling in the design stage. The side panel has been partially cut away to display the processors, heat sinks, and exhaust fans. Figure 1. A Midrange HP server. p l R T V = Duct Perimeter = Length of Heat sink in the Flow Direction = Thermal Resistance = Temperature = Volume Flow Rate. 3. Experimental Procedure In order to expedite the analysis of a system, a volume resistance CFD model is proposed. The model, constructed using the volume resistance functionality available in most CFD tools, requires the user to apply the characteristics of the space the volume resistance occupies. This section describes the gist of the experimental approach used to determine those characteristics 1. The aim of the experimental method is twofold, namely; Determination of the flow resistance of the heat sink. Determination of the thermal resistance of the heat sink. The simplest representation of the experimental information sought is described in Figure 2. A calibrated airflow chamber provides the measured volume flow rate through a tightly ducted heat sink 1. The ratio of volume flow rate through the duct to the duct cross sectional area is the approach velocity, m duct. The graph of pressure drop versus approach velocity is the flow resistance and provides the volume resistance characteristic that can now be applied to a heat sink modeled as a box in a CFD model of a system. To reduce the time to convergence of these large system level models, a semi-empirical technique has been proposed by Patel and Belady that utilizes the volume resistance functionality of CFD tools to model the pressure drop and volume flow rate through a heat sink 1. The case temperature of the component to which the heat sink is attached is then estimated using the numerically determined volume flow rate and empirical thermal resistance data. Since the surface area of the heat sink is not directly modeled, a coarse mesh can be used, thus, dramatically reducing the solution convergence time. Metered air source to deliver quantified measure of air through the heat sink, m 3 /sec Heat Sink P hs Heat Source Air Heat Sink Temperature Measurement 2. Nomenclature R hs, o C/W P hs, Pa V, m 3 /s v duct, m/s P = Static Pressure x = Flow Direction of Fluid Through Heat sink q = Fluid Density Figure 2. Thermo-Volume resistance. f = Moody (or Darcy) Friction Factor f l = Friction Factor (Laminar Flow) An additional important piece of information gathered in the experimentation is the thermal performance of the heat sink. The heat f t = Friction Factor (Turbulent Flow) m duct = Approach Velocity to Volume Resistance source simulates the power into the heat sink. The thermal resistance versus the volume flow rate through the heat sink, V, is deter- d = Hydraulic Diameter of Volume Resistance A c = Duct Cross Sectional Area mined by measuring the temperature of the incoming air and the 158

3 average temperature of the heat sink base (or the surface temperature of the heat source if it is desirable to include the interface thermal resistance with the heat sink thermal resistance measurement). Upon execution of the volume resistance CFD model, one can deduce the volume flow rate through a heat sink used in a system, for example, for a given heat sink configuration the measure of air bypassing or, alternately, going through the heat sink, can be extracted. With that knowledge, and the thermal resistance curve, the thermal performance of the heat sink in a system can be determined. Alternatively, one can build a virtual test bed using a CFD tool and estimate the thermo-volume resistance numerically. Although not as accurate as a careful experimental measurement, a numerical calculation can serve as a good substitute if one does not have the necessary experimental apparatus, or if one does not have parts to test. It is important to note, however, that although this method works well for forced convection environments, it is not optimized for environments in which buoyancy induced convection is a significant method of heat transfer from the heat sink. This is due primarily to the fact that the close ducting around the heat sink during the thermal resistance measurement inhibits the movement of air against gravity. Further research is necessary in order to quantify the changes that are required to the duct cross section before the method can be extended to environments with minimal forced air flow. 4. Thermo-Volume Resistance The thermo-volume resistance technique is a semi-empirical analysis tool. In general, the analyst determines the functional dependence of pressure on the approach velocity to the heat sink. A volume resistance consisting of a defined rectilinear volume within the computational domain of the CFD model is created. The volume resistance should have the cross sectional area of the heat sink duct used for the flow resistance measurement and the length of the heat sink. Figure 3 is a view of a dense heat sink placed within the boundary of the volume resistance used to model it. Once the volume resistance boundaries have been defined, the characteristic dependence of pressure on velocity through the heat sink is added to the volume resistance by specifying the coefficients of the pressure/velocity equation determined from the thermo-volume resistance measurements. This allows a portion of the approaching fluid to bypass the volume resistance if a lower resistance path is available. In essence, measurement is used to determine the effect of friction on the flow through the heat sink, which would normally require a dense grid in a CFD model, while a CFD simulation is used to determine the path of the flow around the heat sink, which can be accomplished with a coarse grid. In general, the flow through a volume resistance resembles internal flow through a pipe or rectangular duct. The pressure drop through the volume can therefore be characterized with the following equation, In equation (1), f is the Moody (or Darcy) friction factor and d is the hydraulic diameter 4. The hydraulic diameter of the volume resistance is equivalent to the hydraulic diameter of the duct used in the flow resistance measurement since they have identical cross sections. For a rectangular cross sectional area, it is defined as follows 5, If the friction factor can be determined from the experimental flow resistance data, the volume resistance within the simulation can be completely defined. v duct Volume Resistance Figure 3. Heat sink with volume resistance. v bypass The friction factor, however, can be difficult to determine since f is generally a function of duct velocity. This functionality, in turn, is dependent upon the flow regime. Specifically, when the flow is laminar, f varies inversely with velocity, and when its turbulent, it is a constant value 6. In the transition region between laminar and turbulent flow, f is a non-linear function of velocity. Since the flow regime within the heat sink, upon deployment in a system, is generally not known when its flow resistance is measured, the test data used to define f may span multiple regimes. Therefore, a de-coupling of the friction factor and flow regime is sought. This is accomplished by breaking up the friction factor into its constituent components, such as, In equation (3), f l and f t are both constants, where f l represents the laminar component of friction, and f t the turbulent component. If equation (3) is substituted into equation (1), the following is obtained, (1) (2) (3) (4) 159

4 Intl. Journal of Microcircuits and Electronic Packaging Though unconventional, equation (4) is useful since it captures flow in transition and allows the pressure drop to be characterized over multiple flow regimes. The analyst is not required to estimate whether the flow through the heat sink will be laminar or turbulent when deployed in a system. If equation (4) is integrated over the length of the heat sink, l, one can obtain, The first half of equation (5) corresponds to the pressure drop that occurs in the laminar flow regime, while the second half describes the effect of turbulent flow. The laminar and turbulent friction factors, f l and f t, respectively, are determined by curve fitting the measured heat sink flow resistance data to equation (5). The resulting values are then placed into the CFD tool and the volume resistance is, at that point, hydrodynamically characterized. Although equation (4) is very useful for characterizing the pressure drop over a broad velocity range, some CFD packages do not allow its use in volume resistance characterization. Rather, equation (4) is truncated and made fully linear as in equation (6), (5) built, the volume resistances in the model must be characterized with the appropriately calculated friction factors. If the pressure drop through the volume resistance is based on equations (6) or (7), the numeric hydrodynamic performance (such as pressure drop and volumetric flow rate) of the volume resistance in the converged CFD solution must be compared with the measured flow resistance data. If they do not match, the friction factor will have to be adjusted until an adequate match is realized. (Note that if equation (4) is used, iteration is not necessary.) When the volume resistance has been adjusted such that a correct representation of the heat sink is obtained, the heat sink base temperature, (or component case temperature), can be calculated using the measured thermal resistance curve and the modeled volume flow rate through the heat sink. Start Obtain Thermo-Volume Resistance Curves Build CFD Model Or, the linear portion is left off and it is made fully quadratic as in equation (7), (6) (7) Extract the Friction Factor(s) From the Flow Resistance Curve Eq (4) Or Estimate the Friction Factor Using Given Pressure/Velocity Curve Eqs (6) or (7) Modify Friction Solve the Model Factor In these cases, unless the flow through the heat sink is either fully laminar (equation (6)) or fully turbulent (equation (7)) within the velocity range of the measurement, the empirical flow resistance Compare Numeric data will not precisely fit the pressure/velocity function. If this occurs, the data can be fit to the given function, and the correlation can Hydrodynamic Performance to Measured Performance be evaluated. If there is a high correlation, the friction factor can be extracted and used to characterize the volume resistance as described No above. If the correlation is poor, the volume flow rate through the Within Error heat sink must be estimated using experience, intuition, or other tools. Then, using the flow resistance data, the friction factor can be calculated based on the measured pressure drop at the estimated flow Yes rate. This best estimate of a friction factor is then used as a starting point to characterize the volume resistance. Upon convergence Calculate Component Temperature of the CFD solution, the pressure drop over the volume resistance and volume flow rate through it must be compared to the measured End flow resistance data. If they do not lie within some tolerated error, iteration will be required. In the next section, the method will be applied to an example heat sink and the three types of curve fits will be distinguished. Figure 4 is a flow chart that describes how to use the thermovolume resistance technique in a typical CFD package to predict Figure 4. Analysis flow chart. component temperature and airflow patterns within the system. Once In addition to the provision of a route to expedient compact CFD the thermo-volume resistance has been measured and a CFD model modeling, the thermo-volume resistance method allows one to con- 160

5 ceive a system without choosing a heat sink. Upon acquiring sufficient experience, one can build an entire system composed of volumes and then fit the volume to a heat sink of a particular characteristic. This can allow appropriate air handling design in the early crucial stage of system design. 5. Case Study of an HP Quad Processor Server P (Pa) Laminar Curve Fit - eq (4) Composite Curve Fit - eq (1) Turbulent Curve Fit - eq (5) Approach Velocity, v duct (m/s) To demonstrate this technique, the design of the Hewlett-Packard server shown in Figure 1 will be discussed. The heat sink used in the initial thermal investigation of the server is displayed in Figure 3. The heat sink is a 5.2 cross-cut extrusion, 4.2 wide and 1.0 tall, with a fin pitch of.17. With four of these heat sinks per system, a very large mesh would be required to resolve the heat transfer from each fin on the heat sink to the cooling fluid, therefore, the thermo-volume resistance method was used to expedite the analysis. The heat sink s flow resistance and thermal resistance were measured in the Thermal Sciences Laboratory at HP Laboratories in Palo Alto using a CFM airflow metering device manufactured by Airflow Measurement Systems 7. The heat sink was placed on an aluminum heater block with a 3 X 4 heating surface. The effect of heat spreading from the heater block surface to the larger heat sink surface was found to be insignificant. A 2 inch long cartridge heater with a diameter of.25 inches was used to provide heat to the heater block. The heater block was housed in an Ultem base to minimize heat loss. Three type T thermocouples were used to record the surface temperature of the heater block. Holes were drilled through the heat sink, normal the heater block surface to probe the surface temperature. Thermal grease with moderate conductivity (~1W/mK) was used at the interface. Since the surface temperature of the heater block was measured, it must be noted that the thermal resistance of the heat sink includes the resistance of the heater block/ heat sink interface. The locations on the heater block surface chosen for temperature recording coincided with the processor manufacturers recommended thermal measurement locations on the component surface to which the heat sink was designed to attach. Figure 5 displays the flow and thermal resistance curves of the heat sink. The flow resistance curve shows the pressure drop as a function of the approach velocity to the heat sink. The solid line represents a curve fit of the data to equation (5) (f l /d = 43.2 s -1 and f t /d = 67.7 m -1 ). The dotted lines show curve fits of the data to equations (6) (f l /d =157 s -1 ) and (7) (f t /d =3.2 m -1 ), the fully laminar and turbulent equations, respectively, integrated over the length of the heat sink. The error in the laminar and turbulent curve fits indicates there is transition from laminar to turbulent flow in the velocity range within which the measurements were taken. Depending upon where the operating point lies on the flow resistance curve within the system environment, iteration of the friction factor will likely be required when using a laminar or turbulent curve fit with this particular heat sink. The International Journal of Microcircuits and Electronic Packaging, Figure Volume 6. 22, CFD Number model. 2, Second Quarter 1999 (ISSN ) Volume Flow Rate, V (m 3 /s) Figure 5. Thermo-volume resistance curves. The server was modeled using Fluent Inc. s CFD package IcePak. The system was designed such that the airflow through the processor bay would be isolated from the rest of the system. This allowed the processor region of the computer to be modeled separately from the rest of the system. Figure 6 displays the CFD model of the system s processor bay. The model consists of an intake vent, four 50 W processors, heat sinks modeled as volume resistances directly below each processor, two 92 mm fans mounted immediately downstream of the processors, and two 120 mm exhaust fans evacuating the system. Vents were modeled with an open area of 50%. Non-functional fan impedance was measured in a similar manner as heat sink flow resistance, thus allowing planar resistances to be used to model the effect of a non-functional fan with a locked rotor. Functional fans were modeled as mass sources characterized by their given fan curves. The software used the overall system information to determine the fan operating points. 161

6 Intl. Journal of Microcircuits and Electronic Packaging The system was designed to be thermally redundant. Thus, any one of the four fans could fail without causing a critical reduction in cooling. The CFD model was used to perform quick parametric studies to determine the effect of fan failure, processor pitch and ducting on processor thermal performance. The model contained approximately 60,000 cells and converged in around 15 minutes running on a Hewlett-Packard J282 PA-RISC Server. Upon convergence, the average volume flow rate through each heat sink was extracted from the solution. Each value was compared with the thermal resistance curve of Figure 5 to determine the heat sink thermal resistance. intake air temperature. The uncertainty in the temperature measurements is estimated to be ±1 C. A power of 50 W (±1 W ) was provided to each module. 6. Results and Discussion a. Intake side. The results from several different solutions of the CFD model were compared with experimentation to assess the accuracy of the thermo-volume resistance technique. Figure 7 shows the experimental setup used to access model accuracy. It consists of a prototype server chassis with the processor bay physically isolated from the rest of the system, four thermal test modules packaged identically to the processors being simulated, two 92 mm fans immediately downstream of the processors, and two 120 mm fans exhausting the system. Resistive heating elements were used to provide power to the individual components on each module. Type T thermocouples were used to record the case temperature of the processor modules exactly as had been done in the heat sink thermal resistance experiment, (at the processor b. Exhaust side. manufacturer s recommended locations), and identical thermal grease was used. Thermal data were recorded using a multi channel Figure 7. Experimental setup. HP data acquisition system and HP VEE software. Flow velocity data were measured with hot wire anemometers from Cambridge AccuSense. Table 1. Velocity comparisons. Tables 1 and 2 compare modeled velocity and temperature results to experimental data, respectively. In Table 1, velocities were Intake 1.89 ± Configuration Location Expmt (m/s) Model (m/s) measured two inches downstream of the intake vent, and approximately.8 upstream of each heat sink. Measurements were taken at All Fans On Processor ± Processor ± three positions in front of each heat sink along its width, and an Processor ± Processor 4 average was taken. The uncertainty in the velocity measurements is 1.18 ± Intake 1.26 ± estimated to be approximately 10% of the reading. Each processor Bottom Processor ± is numbered according to Figures 6 and 7. Exhaust Fan Processor ± In Table 2, R ca is the thermal resistance from the processor case Off Processor ± to the intake air. Each processor was packaged on a multichip module with an aluminum thermal plate (the case) conducting the heat Intake 1.65 ± Processor ± Top Processor 1 from the components on the module to the heat sink. Since the heat 1.11 ± Processor Fan Processor ± input to the heat sink was non-uniform, it was necessary to account Off Processor ± for the spreading of the heat flux through the thermal plate 8. Therefore, R ca Processor ± is composed of heat spreading due to non-uniform power dissipation into the thermal plate, and the heat sink thermal resistance from Figure 5 corresponding to the average volume flow (V) through the heat sink. The resistance also includes the interface resistance since that was included in the heat sink thermal resistance measurement. DT ca is, therefore, the temperature difference between the average case temperature, measured at three locations, and the 162

7 Table 2. Temperature comparisons. Config Lctn Expt Model T ca ± 1.4C V (m 3 /s) R ca (C/W) T ca All Fans Proc On Proc Bot Exst Proc Off Proc Top Exst Proc Off Proc Bot Proc Proc Off Proc Top Proc Proc Off Proc All temperatures in C Agreement between measured and modeled velocities in front of the processors is, in general, good, with an average agreement of 7.4% and a maximum error of 19%. Velocity trends corresponding to fan dysfunction also show good correlation to experiment. Agreement among velocities measured at the intake, however, is not as good with an average disagreement of 22%, outside of the error in the experiment. The reason for this error is unknown, but it is suspected that it may be attributed to the increase in velocity that occurs due to mass conservation as air moves through the small vent holes in the chassis. In the CFD model, the vent is modeled as a large hole in the chassis with a given resistance to flow, therefore, the increase in speed is not captured. Agreement between measured and modeled temperatures is very good with an average agreement of 4% (~.5 C) and a maximum of 9% (~1.3 C), both are within experimental error. This close correlation indicates that the method works very well at estimating component temperatures within the system. 7. Summary and Conclusions This paper has detailed how to apply the thermo-volume resistance method of heat sink characterization, described by Patel and Belady, to system level design and analysis of electronics. The method has been applied to the design of a Hewlett-Packard server. Comparison to an experimental mock up of the system shows good correlation of velocities near the heat sinks, with an average error of 7.4%, and excellent correlation of temperatures with an average error of just 4%. search activities at Hewlett-Packard Labo- Chandrakant Patel leads the thermal re- It should be realized, however, that the method is not limited to ratories. He is responsible for devising the heat sinks alone, but can be employed on any component that impedes airflow. Power supplies, for example, are excellent candi- Packard Labs, and has 15 years of indus- cooling research strategies at Hewlettdates and, indeed, have been modeled with success in this manner trial experience in mechanical product by Hewlett-Packard in the past. design, electronics packaging and cooling Given the reduction in computational grid size that comes with research. the employment of this technique and the corresponding reduction in the time to numerical convergence, the method shows tremendous promise as a process of standardizing forced convection heat sink characterization and as a means of expediting system level analysis. References 1. C.D. Patel, C. L. Belady, Modeling and Metrology in High Performance Heat Sink Design, Proceedings of the Electronic Components and Technology Conference, ECTC 97, San Jose, California, pp , May S.V. Patankar, Numerical Heat Transfer and Fluid Flow, Taylor & Francis Publishers, V. Mansingh, N. Kazuhiro, S. Shidore, S. Addison, Heat Sinks for High Power Multichip Modules, Proceedings of the International Electronics Packaging Society Conference, IEPS 96, pp , F.P. Incropera, D.P. De Witt, Fundamentals of Heat and Mass Transfer, Wiley, pg. 472, M.N. Ozisik, Heat Transfer: A Basic Approach, McGraw- Hill, Mexico, pg. 293, L.F. Moody, Friction Factors for Pipe Flow, Transactions of the ASME, Vol. 66, No. 8, pp , November Airflow Measurement Systems, P.O. Box 2491, Current Updated Literature, Chula Vista, California R. R. Tummala E. J. Rymaszewski, Microelectronics Packaging Handbook, Van Nostrand Reinhold, New York, pp , About the authors Cullen Bash is a member of technical staff at Hewlett-Packard Laboratories and is involved in thermal modeling, metrology and cooling technology research. Before joining Hewlett-Packard Labs, he worked on the thermal design of HP systems and PA-RISC microprocessor modules. 163