CS 147: Computer Systems Performance Analysis

Transcription

1 CS 147: Computer Systems Performance Analysis Advanced Regression Techniques CS 147: Computer Systems Performance Analysis Advanced Regression Techniques 1 / 31

2 Overview Overview Overview Common Transformations Handling Outliers Common Transformations Handling Outliers 2 / 31

3 Linear regression assumes a linear relationship between predictor and response What if it isn t linear? You need to fit some other type of function to the relationship Linear regression assumes a linear relationship between predictor and response What if it isn t linear? You need to fit some other type of function to the relationship 3 / 31

4 4 / 31 When To Use When To Use When To Use Easiest to tell by sight Make a scatter plot If plot looks non-linear, try curvilinear regression Or if non-linear relationship is suspected for other reasons Relationship should be convertible to a linear form Easiest to tell by sight Make a scatter plot If plot looks non-linear, try curvilinear regression Or if non-linear relationship is suspected for other reasons Relationship should be convertible to a linear form

5 5 / 31 Common Transformations Types of Common Transformations Types of Types of Many possible types, based on a variety of relationships: y = ax b y = a + b/x y = ab x Etc., ad infinitum Many possible types, based on a variety of relationships: y = ax b y = a + b/x y = ab x Etc., ad infinitum

6 6 / 31 Common Transformations Transform Them to Linear Forms Common Transformations Transform Them to Linear Forms Transform Them to Linear Forms Apply logarithms, multiplication, division, whatever to produce something in linear form I.e., y = a + b something Or a similar form If predictor appears in more than one transformed predictor variable, correlation is likely! Apply logarithms, multiplication, division, whatever to produce something in linear form I.e., y = a + b something Or a similar form If predictor appears in more than one transformed predictor variable, correlation is likely!

7 Common Transformations Sample Transformations Common Transformations Sample Transformations Sample Transformations For y = ae bx take logarithm of y, do regression on log y = b0 + b1x, let b = b1, a = e b 0 For y = a + b log x, take log of x before fitting parameters, let b = b1, a = b0 For y = ax b, take log of both x and y, let b = b1, a = e b 0 For y = ae bx take logarithm of y, do regression on log y = b 0 + b 1 x, let b = b 1, a = e b 0 For y = a + b log x, take log of x before fitting parameters, let b = b 1, a = b 0 For y = ax b, take log of both x and y, let b = b 1, a = e b 0 7 / 31

8 8 / 31 Common Transformations Corrections to Jain p. 257 (Early Editions) Common Transformations Corrections to Jain p. 257 Corrections to Jain p. 257 (Early Editions) Nonlinear y = a + b/x y = 1/(a + bx) y = x(a + bx) y = ab x Linear y = a+b(1/x) (1/y) = a + bx (x/y) = a + bx ln y = ln a + x ln b y = a + bx n y = a + b(x n ) Nonlinear Linear y = a + b/x y = a+b(1/x) y = 1/(a + bx) (1/y) = a + bx y = x(a + bx) (x/y) = a + bx y = ab x ln y = ln a + x ln b y = a + bx n y = a + b(x n )

9 Use some function of response variable y in place of y itself Curvilinear regression is one example But techniques are more generally applicable Use some function of response variable y in place of y itself Curvilinear regression is one example But techniques are more generally applicable 9 / 31

10 When To Transform? When To Transform? When To Transform? If known properties of measured system suggest it If data s range covers several orders of magnitude If homogeneous variance assumption of residuals (homoscedasticity) is violated If known properties of measured system suggest it If data s range covers several orders of magnitude If homogeneous variance assumption of residuals (homoscedasticity) is violated 10 / 31

11 Transforming Due To (Lack of) Homoscedasticity Transforming Due To (Lack of) Homoscedasticity Transforming Due To (Lack of) Homoscedasticity If spread of scatter plot of residual vs. predicted response isn t homogeneous, Then residuals are still functions of the predictor variables Transformation of response may solve the problem If spread of scatter plot of residual vs. predicted response isn t homogeneous, Then residuals are still functions of the predictor variables Transformation of response may solve the problem 11 / 31

12 12 / 31 What Transformation To Use? Compute standard deviation of residuals Plot as function of mean of observations Assuming multiple experiments for single set of predictor values Check for linearity: if linear, use a log transform If variance against mean of observations is linear, use square-root transform If standard deviation against mean squared is linear, use inverse (1/y) transform If standard deviation against mean to a power is linear, use power transform More covered in the book What Transformation To Use? What Transformation To Use? Compute standard deviation of residuals Plot as function of mean of observations Assuming multiple experiments for single set of predictor values Check for linearity: if linear, use a log transform If variance against mean of observations is linear, use square-root transform If standard deviation against mean squared is linear, use inverse (1/y) transform If standard deviation against mean to a power is linear, use power transform More covered in the book

13 13 / 31 General Transformation Principle General Transformation Principle General Transformation Principle For some observed relation between standard deviation and mean, s = g(y): 1 let h(y) = g(y) dy transform to w = h(y) and regress on w For some observed relation between standard deviation and mean, s = g(y): 1 let h(y) = g(y) dy transform to w = h(y) and regress on w

14 14 / 31 Example: Log Transformation Example: Log Transformation Example: Log Transformation If standard deviation against mean is linear, then g(y) = ay 1 So h(y) = ay dy = 1 a ln y If standard deviation against mean is linear, then g(y) = ay 1 So h(y) = ay dy = 1 a ln y

15 15 / 31 Confidence Intervals for Nonlinear Regressions Confidence Intervals for Nonlinear Regressions Confidence Intervals for Nonlinear Regressions For nonlinear fits using general (e.g., exponential) transformations: Confidence intervals apply to transformed parameters Not valid to perform inverse transformation before calculating intervals Must express confidence intervals in transformed domain For nonlinear fits using general (e.g., exponential) transformations: Confidence intervals apply to transformed parameters Not valid to perform inverse transformation before calculating intervals Must express confidence intervals in transformed domain

16 16 / 31 Handling Outliers Outliers Handling Outliers Outliers Outliers Atypical observations might be outliers Measurements that are not truly characteristic By chance, several standard deviations out Or mistakes might have been made in measurement Which leads to a problem: Do you include outliers in analysis or not? Atypical observations might be outliers Measurements that are not truly characteristic By chance, several standard deviations out Or mistakes might have been made in measurement Which leads to a problem: Do you include outliers in analysis or not?

17 17 / 31 Handling Outliers Deciding How To Handle Outliers Handling Outliers Deciding How To Handle Outliers Deciding How To Handle Outliers 1. Find them (by looking at scatter plot) 2. Check carefully for experimental error 3. Repeat experiments at predictor values for each outlier 4. Decide whether to include or omit outliers Or do analysis both ways Question: Is last point in last lecture s example an outlier on rating vs. year plot? 1. Find them (by looking at scatter plot) 2. Check carefully for experimental error 3. Repeat experiments at predictor values for each outlier 4. Decide whether to include or omit outliers Or do analysis both ways Question: Is last point in last lecture s example an outlier on rating vs. year plot?

18 Handling Outliers Rating vs. Year Handling Outliers Rating vs. Year Rating vs. Year 8 6 Rating Year 8 6 Rating Year 18 / 31

19 in Regression in Regression in Regression Generally based on taking shortcuts Or not being careful Or not understanding some fundamental principle of statistics Generally based on taking shortcuts Or not being careful Or not understanding some fundamental principle of statistics 19 / 31

20 Not Verifying Linearity Not Verifying Linearity Not Verifying Linearity Draw the scatter plot If it s not linear, check for curvilinear possibilities Misleading to use linear regression when relationship isn t linear Draw the scatter plot If it s not linear, check for curvilinear possibilities Misleading to use linear regression when relationship isn t linear 20 / 31

21 21 / 31 Relying on Results Without Visual Verification Relying on Results Without Visual Verification Relying on Results Without Visual Verification Always check scatter plot as part of regression Examine predicted line vs. actual points Particularly important if regression is done automatically Always check scatter plot as part of regression Examine predicted line vs. actual points Particularly important if regression is done automatically

22 22 / 31 Some Nonlinear Examples Some Nonlinear Examples Some Nonlinear Examples

23 23 / 31 Attaching Importance to Parameter Values Attaching Importance to Parameter Values Attaching Importance to Parameter Values Numerical values of regression parameters depend on scale of predictor variables So just because a particular parameter s value seems small or large, not necessarily an indication of importance E.g., converting seconds to microseconds doesn t change anything fundamental But magnitude of associated parameter changes Numerical values of regression parameters depend on scale of predictor variables So just because a particular parameter s value seems small or large, not necessarily an indication of importance E.g., converting seconds to microseconds doesn t change anything fundamental But magnitude of associated parameter changes

24 Not Specifying Confidence Intervals Not Specifying Confidence Intervals Not Specifying Confidence Intervals Samples of observations are random Thus, regression yields parameters with random properties Without confidence interval, impossible to understand what a parameter really means Samples of observations are random Thus, regression yields parameters with random properties Without confidence interval, impossible to understand what a parameter really means 24 / 31

25 Not Calculating Coefficient of Determination Not Calculating Coefficient of Determination Not Calculating Coefficient of Determination Without R 2, difficult to determine how much of variance is explained by the regression Even if R 2 looks good, safest to also perform an F-test Not that much extra effort Without R 2, difficult to determine how much of variance is explained by the regression Even if R 2 looks good, safest to also perform an F-test Not that much extra effort 25 / 31

26 Using Coefficient of Correlation Improperly Using Coefficient of Correlation Improperly Using Coefficient of Correlation Improperly Coefficient of determination is R 2 Coefficient of correlation is R R 2 gives percentage of variance explained by regression, not R E.g., if R is.5, R 2 is.25 And regression explains 25% of variance Not 50%! Coefficient of determination is R 2 Coefficient of correlation is R R 2 gives percentage of variance explained by regression, not R E.g., if R is.5, R 2 is.25 And regression explains 25% of variance Not 50%! 26 / 31

27 27 / 31 Using Highly Correlated Predictor Variables Using Highly Correlated Predictor Variables Using Highly Correlated Predictor Variables If two predictor variables are highly correlated, using both degrades regression E.g., likely to be correlation between an executable s on-disk and in-core sizes So don t use both as predictors of run time Means you need to understand your predictor variables as well as possible If two predictor variables are highly correlated, using both degrades regression E.g., likely to be correlation between an executable s on-disk and in-core sizes So don t use both as predictors of run time Means you need to understand your predictor variables as well as possible

28 28 / 31 Using Regression Beyond Range of Observations Using Regression Beyond Range of Observations Using Regression Beyond Range of Observations Regression is based on observed behavior in a particular sample Most likely to predict accurately within range of that sample Far outside the range, who knows? E.g., regression on run time of executables smaller than size of main memory may not predict performance of executables that need VM activity Regression is based on observed behavior in a particular sample Most likely to predict accurately within range of that sample Far outside the range, who knows? E.g., regression on run time of executables smaller than size of main memory may not predict performance of executables that need VM activity

29 Measuring Too Little of the Range Measuring Too Little of the Range Measuring Too Little of the Range Converse of prevoius mistake Regression only predicts well near range of observations If you don t measure commonly used range, regression won t predict much E.g., if many programs are bigger than main memory, only measuring those that are smaller is a mistake Converse of prevoius mistake Regression only predicts well near range of observations If you don t measure commonly used range, regression won t predict much E.g., if many programs are bigger than main memory, only measuring those that are smaller is a mistake 29 / 31

30 30 / 31 Using Too Many Predictor Variables Using Too Many Predictor Variables Using Too Many Predictor Variables Adding more predictors does not necessarily improve model! More likely to run into multicollinearity problems So what variables to choose? It s an art Subject of much of this course Adding more predictors does not necessarily improve model! More likely to run into multicollinearity problems So what variables to choose? It s an art Subject of much of this course

31 Assuming a Good Predictor Is a Good Controller Assuming a Good Predictor Is a Good Controller Assuming a Good Predictor Is a Good Controller Often, a goal of regression is finding control variables But correlation isn t necessarily control Just because variable A is related to variable B, you may not be able to control values of B by varying A E.g., if number of hits on a Web page is correlated to server bandwidth, you might not boost hits by increasing bandwidth Often, a goal of regression is finding control variables But correlation isn t necessarily control Just because variable A is related to variable B, you may not be able to control values of B by varying A E.g., if number of hits on a Web page is correlated to server bandwidth, you might not boost hits by increasing bandwidth 31 / 31