COMP 551 Applied Machine Learning Lecture 3: Linear regression (cont d)

Transcription

1 COMP 551 Applied Machine Learning Lecture 3: Linear regression (cont d) Instructor: Herke van Hoof Slides mostly by: Class web page: Unless otherwise noted, all material posted for this course are copyright of the Instructors, and cannot be reused or reposted without the instructor s written permission.

2 What we saw last time Definition and characteristics of a supervised learning problem. Linear regression (hypothesis class, cost function, algorithm). Closed-form least-squares solution method (algorithm, computational complexity, stability issues). Alternative optimization methods via gradient descent. 2

3 Predicting recurrence time from tumor size This function looks complicated, and a linear hypothesis does not seem very good. What should we do? 3

4 Predicting recurrence time from tumor size This function looks complicated, and a linear hypothesis does not seem very good. What should we do? Pick a better function? Use more features? Get more data? 4

5 Input variables for linear regression Original quantitative variables X 1,, X m Transformations of variables, e.g. X m+1 = log(x i ) Basis expansions, e.g. X m+1 = X i2, X m+2 = X i3, Interaction terms, e.g. X m+1 = X i X j Numeric coding of qualitative variables, e.g. X m+1 = 1 if X i is true and 0 otherwise. In all cases, we can add X m+1,, X m+k to the list of original variables and perform the linear regression. 5

6 Example of linear regression with polynomial terms f w (x) = w 0 + w 1 x +w 2 x 2 X = x 2 x Y =

7 Solving the problem w =(X T X) 1 X T Y = apple apple = apple So the best order-2 polynomial is y =0.68x x Compared to y = 1.6x for the order-1 polynomial. 7

8 Order-3 fit: Is this better? 8

9 Order-4 fit 9

10 Order-5 fit 10

11 Order-6 fit 11

12 Order-7 fit 12

13 Order-8 fit 13

14 Order-9 fit 14

15 This is overfitting! 15

16 This is overfitting! We can find a hypothesis that explains perfectly the training data, but does not generalize well to new data. In this example: we have a lot of parameters (weights), so the hypothesis matches the data points exactly, but is wild everywhere else. A very important problem in machine learning. 16

17 Overfitting Every hypothesis has a true error measured on all possible data items we could ever encounter (e.g. f w (x i ) - y i ). Since we don t have all possible data, in order to decide what is a good hypothesis, we measure error over the training set. Formally: Suppose we compare hypotheses f 1 and f 2. Assume f 1 has lower error on the training set. If f 2 has lower true error, then our algorithm is overfitting. 17

18 Overfitting Which hypothesis has the lowest true error? d=1 d=2 d=3 d=4 d=5 d=6 d=7 d=8 18

19 Cross-Validation Partition your data into a Training Set and a Validation set. The proportions in each set can vary. Use the Training Set to find the best hypothesis in the class. Use the Validation Set to evaluate the true prediction error. Compare across different hypothesis classes (different order polynominals.) Train: Validate: X = Y =

20 k-fold Cross-Validation Consider k partitions of the data (usually of equal size). Train with k-1 subset, validate on k th subset. Repeat k times. Average the prediction error over the k rounds/folds. Source: 20

21 k-fold Cross-Validation Consider k partitions of the data (usually of equal size). Train with k-1 subset, validate on k th subset. Repeat k times. Average the prediction error over the k rounds/folds. Source: Computation time is increased by factor of k. 21

22 Leave-one-out cross-validation Let k = n, the size of the training set For each order-d hypothesis class, Repeat n times: Set aside one instance <x i, y i > from the training set. Use all other data points to find w (optimization). Measure prediction error on the held-out <x i, y i >. Average the prediction error over all n subsets. Choose the d with lowest estimated true prediction error. 22

23 Estimating true error for d=1 x Data y Cross-validation results 23

24 Cross-validation results Optimal choice: d=2. Overfitting for d > 2. 24

25 Evaluation We use cross-validation for model selection. Available labeled data is split into two parts: Training set is used to select a hypothesis f from a class of hypotheses F (e.g. regression of a given degree). Validation set is used to compare the best f from each hypothesis class across different classes (e.g. different degree regression). Must be untouched during the process of looking for f within a class F. 25

26 Evaluation After adapting the weights to minimize the error on the train set, the weights could be exploiting particularities in the train set: have to use the validation set as proxy for true error After choosing the hypothesis class to minimize error on the validation set, the hypothesis class could be adapted to some particularities in the validation set Validation set is no longer a good proxy for the true error! 26

27 Evaluation We use cross-validation for model selection. Available labeled data is split into parts: Training set is used to select a hypothesis f from a class of hypotheses F (e.g. regression of a given degree). Validation set is used to compare the best f from each hypothesis class across different classes (e.g. different degree regression). Must be untouched during the process of looking for f within a class F. Test set: Ideally, a separate set of (labeled) data is withheld to get a true estimate of the generalization error. Cannot be touched during the process of selecting F (Often the validation set is called test set, without distinction.) 27

28 Validation vs Train error [From Hastie et al. textbook] High Bias Low Variance Low Bias High Variance Prediction Error Test Sample Training Sample Low Model Complexity High FIGURE Test and training error as a function of model complexity. 28

29 Understanding the error Given set of examples <X, Y>. Assume that y = f(x) +, where is Gaussian noise with zero mean and std deviation σ. N (x µ, 2 ) f(x) 2 µ x 29

30 Understanding the error Consider standard linear regression solution: Err(w) = i=1:n ( y i - w T x i ) 2 If we consider only the class of linear hypotheses, we have systematic prediction error, called bias, whenever the data is generated by a non-linear function. Depending on what dataset we observed, we may get different solutions. Thus we can also have error due to this variance. This occurs even if data is generated from class of linear functions. 30

31 An example (from Tom Dietterich) The circles are data points. X is drawn uniformly randomly. Y is generated by the function y = 2sin(0.5x) +. 31

32 An example (from Tom Dietterich) With different sets of 20 points, we get different lines. 32

33 Bias-variance decomposition Compare f(x) (true value at x) to prediction h(x) Error: E[(h(x) f(x)) 2 ] Bias: f(x) h(x) How far is the average estimate from the true function? Variance: E[(h(x) h(x)) 2 ] How far are estimates (on average) from the average estimate Notes: Expectations over all possible dataset (noise realizations) h(x) =E[h(x)] 33

34 Bias-variance decomposition Can show that: Error = bias 2 + variance (e.g. What happens if we apply 1 st order model to quadratic dataset? No matter how much data we have, can t fit the dataset This means we have bias (underfitting) What happens if we apply 2 nd order model to linear dataset? The model can definitely fit the dataset! (no bias) Spurious parameters make the model sensitive to noise Higher variance than necessary (overfitting if dataset is small) 34

35 Bias-variance decomposition Can show that: Error = bias 2 + variance (e.g. What happens if we apply 1 st order model to quadratic dataset? No matter how much data we have, can t fit the dataset This means we have bias (underfitting) What happens if we apply 2 nd order model to linear dataset? The model can definitely fit the dataset! (no bias) Spurious parameters make the model sensitive to noise Higher variance than necessary (overfitting if dataset is small) 35

36 Validation vs Train error [From Hastie et al. textbook] High Bias Low Variance Low Bias High Variance Prediction Error Test Sample Training Sample Low Model Complexity High FIGURE Test and training error as a function of model complexity. 36

37 Bias-variance decomposition Can show that: Error = bias 2 + variance (e.g. What happens if we apply 1 st order model to quadratic dataset? No matter how much data we have, can t fit the dataset This means we have bias (underfitting) What happens if we apply 2 nd order model to linear dataset? The model can definitely fit the dataset! (no bias) Spurious parameters make the model sensitive to noise Higher variance than necessary (overfitting if dataset is small) What happens if we apply 1 st order model to linear dataset? 37

38 Gauss-Markov Theorem Main result: The least-squares estimates of the parameters w have the smallest variance among all linear unbiased estimates. 38

39 Gauss-Markov Theorem Main result: The least-squares estimates of the parameters w have the smallest variance among all linear unbiased estimates. Understanding the statement: Real parameters are denoted: w Estimate of the parameters is denoted: ŵ Error of the estimator: Err(ŵ) = E(ŵ-w) 2 = Var(ŵ) + (E(ŵ-w)) 2 Unbiased estimator means: E(ŵ-w)=0 There may exist an estimator that has lower error, but some bias. 39

40 Bias vs Variance Gauss-Markov Theorem says: The least-squares estimates of the parameters w have the smallest variance among all linear unbiased estimates. Insight: Find lower variance solution, at the expense of some bias. 40

41 Bias vs Variance Gauss-Markov Theorem says: The least-squares estimates of the parameters w have the smallest variance among all linear unbiased estimates. Insight: Find lower variance solution, at the expense of some bias. E.g. fix low-relevance weights to 0 41

42 Recall our prostate cancer example The Z-score measures the effect of dropping that feature from the linear regression. z j = ŵ j / sqrt(σ 2 v j ) where ŵ j is the estimated weight of the j th feature, and v j is the j th diagonal element of (X T X) -1 TABLE 3.2. Linear model fit to the prostate cancer data. The Z score is the coefficient divided by its standard error (3.12). Roughly a Z score larger than two in absolute value is significantly nonzero at the p =0.05 level. Term Coefficient Std. Error Z Score Intercept lcavol lweight age lbph svi lcp gleason pgg [From Hastie et al. textbook] 42

43 Subset selection Idea: Keep only a small set of features with non-zero weights. Goal: Find lower variance solution, at the expense of some bias. There are many different methods for choosing subsets. (More on this later ) Least-squares regression can be used to estimate the weights of the selected features. Bias as true model might rely on the discarded features! 43

44 Bias vs Variance Find lower variance solution, at the expense of some bias. Force some weights to 0 E.g. Include penalty for model complexity in error to reduce overfitting. Err(w) = i=1:n ( y i - w T x i ) 2 + λ model_size λ is a hyper-parameter that controls penalty size. 44

45 Ridge regression (aka L2-regularization) Constrains the weights by imposing a penalty on their size: ŵ ridge = argmin w { i=1:n ( y i - w T x i ) 2 + λ j=0:m w j2 } where λ can be selected manually, or by cross-validation. 45

46 Ridge regression (aka L2-regularization) Constrains the weights by imposing a penalty on their size: ŵ ridge = argmin w { i=1:n ( y i - w T x i ) 2 + λ j=0:m w j2 } where λ can be selected manually, or by cross-validation. Do a little algebra to get the solution: ŵ ridge = (X T X+λI) -1 X T Y 46

47 Ridge regression (aka L2-regularization) Re-write in matrix notation: f w (X) = Xw Err(w) = (Y Xw) T (Y Xw) + λw T w To minimize, take the derivative w.r.t. w: Try a little algebra: Err(w)/ w = -2 X T (Y Xw) 2λw = 0 47

48 Ridge regression (aka L2-regularization) Re-write in matrix notation: f w (X) = Xw Err(w) = (Y Xw) T (Y Xw) + λw T w To minimize, take the derivative w.r.t. w: Err(w)/ w = -2 X T (Y Xw) 2λw = 0 Try a little algebra: X T Y = (X T X + Iλ) w ŵ = (X T X + Iλ) -1 X T Y 48

49 Ridge regression (aka L2-regularization) Constrains the weights by imposing a penalty on their size: ŵ ridge = argmin w { i=1:n ( y i - w T x i ) 2 + λ j=0:m w j2 } where λ can be selected manually, or by cross-validation. Do a little algebra to get the solution: ŵ ridge = (X T X+λI) -1 X T Y The ridge solution is not equivariant under scaling of the data, so typically need to normalize the inputs first. Ridge gives a smooth solution, effectively shrinking the weights, but drives few weights to 0. 49

50 Lasso regression (aka L1-regularization) Constrains the weights by penalizing the absolute value of their size: ŵ lasso = argmin W { i=1:n ( y i - w T x i ) 2 + λ j=1:m w j } 50

51 Lasso regression (aka L1-regularization) Constrains the weights by penalizing the absolute value of their size: ŵ lasso = argmin W { i=1:n ( y i - w T x i ) 2 + λ j=1:m w j } Now there is no closed-form solution. Need to solve a quadratic programming problem instead. 51

52 Lasso regression (aka L1-regularization) Constrains the weights by penalizing the absolute value of their size: ŵ lasso = argmin W { i=1:n ( y i - w T x i ) 2 + λ j=1:m w j } Now there is no closed-form solution. Need to solve a quadratic programming problem instead. More computationally expensive than Ridge regression. Effectively sets the weights of less relevant input features to zero. 52

53 Comparing Ridge and Lasso Ridge Complexity: j=0:m w j 2 Lasso λ j=1:m w j Note that for lasso, reducing any weight by, say, 0.1 reduces the complexity by the same amount So, we d prefer reducing less relevant features In ridge regression, reducing high weights reduces complexity more than reducing a low weight So, trade-off between reducing less relevant features and larger weights. Tend to not reduce weights that are already small 53

54 Comparing Ridge and Lasso Ridge g regularization (2 pa w 2 Contours of equal regression error w 2 Lasso 1 w? w? w 1 w 1 Contours of equal model complexity penalty 54

55 A quick look at evaluation functions We call L(Y,f w (x)) the loss function. Least-square / Mean squared-error (MSE) loss: L(Y, f w (X)) = i=1:n ( y i - w T x i ) 2 Other loss functions? 55

56 A quick look at evaluation functions We call L(Y,f w (x)) the loss function. Least-square / Mean squared-error (MSE) loss: L(Y, f w (X)) = i=1:n ( y i - w T x i ) 2 Other loss functions? Absolute error loss: L(Y, f w (X)) = i=1:n y i w T x i 0-1 loss (for classification): L(Y, f w (X)) = i=1:n I ( y i f w (x i ) ) 56

57 A quick look at evaluation functions We call L(Y,f w (x)) the loss function. Least-square / Mean squared-error (MSE) loss: L(Y, f w (X)) = i=1:n ( y i - w T x i ) 2 Other loss functions? Absolute error loss: L(Y, f w (X)) = i=1:n y i w T x i 0-1 loss (for classification): L(Y, f w (X)) = i=1:n I ( y i f w (x i ) ) Different loss functions make different assumptions. Squared error loss assumes the data can be approximated by a global linear model with Gaussian noise. Loss function independent of complexity penalty (l1 or l2) 57

58 A quick look at evaluation functions Assume data generated as p(y x; w) =N (x T w, 1 (y x T = p exp w) ) 58

59 A quick look at evaluation functions Assume data generated as p(y x; w) =N (x T w, 1 (y x T = p exp w) ) Reasonable to try to maximize arg max w the probability of generating the labels y p(y x, w) = arg max log p(y x, w) w = arg max w const (y xt w) 2 59

60 Tutorial Programming with Python This Friday, 6-7 pm Stewart Biology building S3/3 60

61 What you should know Overfitting (when it happens, how to avoid it). Cross-validation (how and why we use it). Ridge regression (l2 regularisation), Lasso (l1 regularisation) 61

62 MSURJ announcement 62