EEC 686/785 Modeling & Performance Evaluation of Computer Systems. Lecture 11

Transcription

1 EEC 686/785 Modeling & Performance Evaluation of Computer Systems Lecture Department of Electrical and Computer Engineering Cleveland State University (based on Dr. Raj Jain s lecture notes) Outline Move nd midterm to Nov.? Review of lecture 0 Other regression models Eperimental design

2 Midterm # 3 P P P3 P P5 P6 P7 P8 Total Workload Characterization Techniques Workload characterization: the process of studying the real-user environments, observe the key characteristics, and develop a workload model that can be used repeated The measured workload data consists of services requested or the resource demands of a number of users on the system The term user denotes the entity that makes the service requests at the SUT interface

3 5 Workload Characterization Techniques In workload characterization literature, the term workload component or workload unit is used instead of the user The workload component should be at the SUT interface Workload parameters or workload features Measured quantities, service requests, or resource demands For eample: transaction types, instructions, packet sizes, source-destinations of a packet, and page reference pattern Each component should represent as homogeneous a group as possible 6 Workload Characterization Techniques Averaging Single-parameter histograms Multiparameter histograms Principal component analysis Markov models Clustering 3

4 Principal Component Analysis 7 Key idea: use a weighted sum of parameters to classify the components Let ij denote the ith parameter for jth component y j = Principal component analysis assigns weights w i s such that y j s provide the maimum discrimination among the components The quantity y j is called the principal factor The factors are ordered. First factor eplains the highest percentage of the variance n w i ij Finding Principal Factors 8 Find the correlation matri Find the eigenvalues of the matri and sort them in the order of decreasing magnitude Find corresponding eigenvectors. These give the required loadings

5 Markov Models 9 Markov => the net request depends only on the last request Described by a transition matri Given the same relative frequency of requests of different types, it is possible to realize the frequency with several different transition matrices Clustering 0 Take a sample, that is, a subset of workload components Select workload parameters Select a distance measure Remove outliers Scale all observations Perform clustering Interpret results Change parameters, or number of clusters, and repeat 3-7 Select representative components from each cluster 5

6 Multiple Linear Regression Models A multiple linear regression model allows one to predict a response variable y as a function of k predictor variables,,, k using a linear model: Given a sample {(,,, k,y ),,( n, n,, kn,y n )} of n observations, the model consists of the following n equations: y = b0 + b + b+ + bkk+ e y = b + b + b + + b + e 0 k k y = b + b + b + + b + e n 0 n n n k kn n Multiple Linear Regression Models In vector notation, we have: y k b e y k b e..... = yn n n kn bn en or y = Xb+ e b= X X X y T T ( ) ( ) 6

7 Regression with Categorical Predictors 3 Eamples of categorical variables: CPU types To represent a categorical variable that can take only values, we can define a binary variable, j, that takes levels: + and -, i.e., first value j = + second value For a categorical variable that takes 3 values, cannot simply define a three-value variable because it implies an order type A = type B 3 type C Regression with Categorical Predictors The recommended coding is to use predictor variables: if type A if type B = = 0 otherwise 0 otherwise The 3 types can be represented by (, ) pairs: (, ) = (,0) type A (, ) = (0,) type B (, ) = (0,0) type C 7

8 Regression with Categorical Predictors The regression model for a 3-level categorical variable: y = b 0 + b + b + e The average responses for the three types are: ya = b0 + b yb = b0 + b yc = b0 Parameter b represents the difference between average responses with types A and C Parameter b represents the difference between average responses with types B and C Parameter b 0 represents average response with type C 5 Regression with Categorical Predictors 6 To represent a categorical variable with k levels (or k categories), we need to define k binary variables as follows: j if jth value = 0 otherwise The kth value is defined by = = = k- =0. The regression parameter b 0 represents the average response with the kth alternative The parameter b j represents the difference between the average responses with alternatives j and k 8

9 Curvilinear Regression 7 Curvilinear regression: if the nonlinear function can be converted into a linear form, then the regression can be carried out using the simple or multiple linear regression techniques Nonlinear Linear y = a + b/ y = a+ b(/) y = /(a+b) (/y) = a + b y = /(a+b) (/y) = a + b y = ab ln y = ln a + (ln b)ln y = a + b n y = a + b( n ) y = b a ln y = ln b + a ln Transformations 8 Transformation: when some function of the measured response variable y is used in place of y in a model E.g., squire root transformation: y = b0 + b + b + + bkk + e 9

10 When Transformations Are Needed 9 If it is known from physical considerations of the system that a function of the response rather than the response itself is a better variable to use in the model Measured the interarrival times y for requests and it is known that the number of requests per unit time (/y) has a linear relationship to a certain predictor If the range of the data covers several orders of magnitude and the sample size is small => use transformation to reduce the range of variability If the homogeneous variance assumption of the residuals is violated Eperimental Design and Analysis 0 Design a proper set of eperiments for measurement or simulation Develop a model that best describes the data obtained Estimate the contribution of each alternative to the performance Isolate the measurement errors Estimate confidence intervals for model parameters Check if the alternatives are significantly different Check if the model is adequate 0

11 Terminology Response Variables: outcome E.g., throughput, response time Factors: variables that affect the response variable E.g., CPU type, memory size, number of disk drives, workload used, and user s educational level Also called predictor variables or predictors Levels: the values that a factor can assume E.g., the CPU type has three levels: 68000, 8080, Z80 Also called treatment Terminology Primary Factors: the factors whose effects need to be quantified E.g., CPU type, memory size only, and number of disk drives Secondary Factors: factors whose impact need not be quantified Replication: repetition of all or some eperiments Design: the number of eperiments, the factor level and number of replications for each eperiment E.g., full factorial design with 5 replications: or 3 eperiments, each repeated five times

12 Terminology 3 Eperimental Unit: any entity that is used for eperiments. Usually only those eperimental units that are considered as one of the factors are of interest E.g., users hired to use the workstation while measurements are being performed can be considered as the eperimental unit. Generally, no interest in comparing the units Goal: minimize the impact of variation among the units Terminology Interaction: effect of one factor depends upon the level of the other Eample: two factors A and B, each has two levels

13 Types of Eperimental Designs 5 Given k factors, with ith factor having n i levels Simple Designs: vary one factor at a time Number of eperiments n = + Not statistically efficient Wrong conclusions if the factors have interaction Not recommended k ( n i ) Types of Eperimental Designs 6 Full Factorial Design: all combinations Number of eperiments = Can find the effect of all factors Too much time and money Ways to reduce the number of eperiments Reduce the number of levels for each factor, e.g., levels per factor Reduce the number of factors Use fractional factorial designs k n i 3

14 Types of Eperimental Designs 7 Fractional Factorial Designs: save time and epense Less information May not get all interactions Not a problem if negligible interactions Types of Eperimental Designs 8 A sample fractional factorial design 9 eprs (3 - design) instead of 8 (3 design)

15 k Factorial Designs 9 k factors, each at two levels Easy to analyze Helps in sorting out impact of factors Good at the beginning of a study Valid only if the effect of a factor is unidirectional, i.e., the performance either continuously decreases or continuously increases as the factor is increased from min to ma E.g., memory size, the number of disk drives Factorial Designs: Eample 30 Two factors, each at two levels 5

16 Factorial Designs: Model 3 y = q 0 + q A A + q B B + q AB A B 5 = q 0 q A q B + q AB 5 = q 0 + q A q B + q AB 5 = q 0 q A + q B q AB 75 = q 0 + q A + q B + q AB Unique solution for q A and q B : y = A + 0 B + 5 A B Interpretation: Mean performance = 0 MIPS Effect of memory = 0 MIPS Effect of cache = 0 MIPS Interaction between memory and cache = 5 MIPS Computation of Effects 3 Model: y = q 0 + q A A + q B B + q AB A B Substitution: y = q 0 q A q B + q AB y = q 0 + q A q B + q AB y 3 = q 0 q A + q B q AB y = q 0 + q A + q B + q AB 6

17 Computation of Effects 33 Solution: q 0 = ¼ (y +y +y 3 +y ) q A = ¼ (-y +y -y 3 +y ) q B = ¼ (-y +y +y 3 +y ) q AB = ¼ (y -y -y 3 +y ) Notice that effects are linear combinations of responses. Sum of the coefficients is zero => contrasts Notice: q A = Column A Column y q B = Column B Column y q AB = Column A Column B Column y Sign Table Method 3 For a design, the effects can be computed easily by preparing a sign matri as shown above Net, multiply the entries in column I by those in column y and put their sum under column I; perform similar calculation for other columns The sums under each column are divided by to give the corresponding coefficients of the regression model 7

18 Allocation of Variation 35 Importance of a factor = proportion of the variation eplained ( ) Sample Variance of = = yi y y sy Variation of y Δ Numerator = i = ( y i y) = sum of squares total ( SST ) Allocation of Variation 36 For a design SST = + qa + qb qab Variation due to A=SSA= Variation due to B=SSB= qa q B Variation due to interaction AB=SSAB= SST = SSA + SSB + SSAB Fraction eplained by A = SSA / SST Variation Variance qab 8

19 Derivation 37 Model: y q + q + q + q Notice i = 0 A The sum of entries in each column is zero The sum of the squares of entries in each column is : B = 0; = 0; = ; = ; ( AB = 0 ) = Derivation Notice (continued) The columns are orthogonal (inner product of any two columns is zero): Sample mean = 0; ( ) = 0; y = = = = q 0 y i ( q 0 + q A q0 + q A + q B ( + q + q AB B ) = 0 ) + q AB 38 9

20 Derivation 39 Variation of y = = = = q = q ( y A A i ( q A ( q A y) ( + q ) ) B + q + B + q + q B AB + q B AB ( q ( ) ) + ) + q ( q AB AB ( ) ) + product terms + 0 Eample 0 Memory-cache study: y = ( ) = 0 Total variation = Total variation = 00 Variation due to memory = 600 (76%) Variation due to cache = 00 (9%) Variation due to interaction = 00 (5%) = ( y i y) = ( ) = 00 0

21 Case Study: Interconnection Nets Memory interconnection networks: Omega and Crossbar Memory reference patterns Random (with uniform probability) Matri multiplication problem in which each process is doing a part of the multiplication Case Study: Interconnection Nets Fied factors: Number of processors was fied at 6 Queued requests were not buffered but blocked Circuit switching instead of packet switching Random arbitration instead of round robin Infinite interleaving of memory => no memory bank contention

22 Design for Interconnection Networks 3 Interpretation of Results Average throughput = More effective factor = B = reference pattern => The address patterns chosen are very different Reference pattern eplains ±0.57 (77%) of variation Effect of network type = Omega networks = average Crossbar networks = average Difference between the two = 0.9 Slight interaction (0.036) between reference pattern and network type

23 General k Factorial Designs 5 k factors at two levels each k eperiments k effects: k main effects k two factor interactions k three factor interactions 3 k Design Eample 6 Three factors in designing a machine Cache size Memory size Number of processors Factor A Memory Size B Cache Size C Number of Processors Level - MB kb Level 6MB kb 3

24 k Design Eample 7 k Design Eample - Analysis 8 3 SST = ( qa + qb + qc + qab + qbc + qac + qabc ) = 8( ) = = 8% + % + 7% + % + % + % + 0% = 00% Number of processors (C) is the most important factor