Regression. Goal: Learn a mapping from observations (features) to continuous labels given a training set (supervised learning)

Size: px

Start display at page:

Download "Regression. Goal: Learn a mapping from observations (features) to continuous labels given a training set (supervised learning)"

Daisy Knight
6 years ago
Views:

1 Linear Regression

2 Regression Goal: Learn a mapping from observations (features) to continuous labels given a training set (supervised learning) Example: Height, Gender, Weight Shoe Size Audio features Song year Processes, memory Power consumption Historical financials Future stock price Many more

3 Linear Least Squares Regression Example: Predicting shoe size from height, gender, and weight For each observation we have a feature vector, x, and label, y x = x 1 x 2 x 3 We assume a linear mapping between features and label: y w 0 + w 1 x 1 + w 2 x 2 + w 3 x 3

4 Linear Least Squares Regression Example: Predicting shoe size from height, gender, and weight We can augment the feature vector to incorporate offset: x = 1 x 1 x 2 x 3 We can then rewrite this linear mapping as scalar product: 3 y ŷ = w i x i = w x i=0

5 Why a Linear Mapping? Simple Often works well in practice Can introduce complexity via feature extraction

6 1D Example Goal: find the line of best fit y x coordinate: features y coordinate: labels y ŷ = w 0 + w 1 x x Intercept / Offset Slope

7 Evaluating Predictions Can measure closeness between label and prediction Shoe size: better to be off by one size than 5 sizes Song year prediction: better to be off by a year than by 20 years What is an appropriate evaluation metric or loss function? Absolute loss: Squared loss: y ŷ (y ŷ) 2 Has nice mathematical properties

8 How Can We Learn Model (w)? Assume we have n training points, where denotes the ith point Recall two earlier points: Linear assumption: ŷ = w x We use squared loss: (y ŷ) 2 Idea: Find w that minimizes squared loss over training points: n min w i=1 (w y (i) ) 2 ŷ (i){

9 Given n training points with d features, we define: : matrix storing points X R n d y : real-valued labels R n ŷ : predicted labels, where R n ŷ = Xw w : regression parameters / model to learn R d Least Squares Regression: Learn mapping ( ) from features to labels that minimizes residual sum of squares: n min w Xw y 2 2 w Equivalent min w (w y (i) ) 2 by definition of Euclidean norm i=1

10 Find solution by setting derivative to zero n 1D: f(w) = wx y 2 2 = (w y (i) ) 2 n df dw (w) =2 i=1 (w y (i) ) wx x x y i=1 = 0 wx x x y = 0 w =(x x) 1 x y Least Squares Regression: Learn mapping ( ) from features to labels that minimizes residual sum of squares: min w Xw y 2 2 w Closed form solution: w =(X X) 1 X y (if inverse exists)

11 Overfitting and Generalization We want good predictions on new data, i.e., generalization Least squares regression minimizes training error, and could overfit Simpler models are more likely to generalize (Occam s razor) Can we change the problem to penalize for model complexity? Intuitively, models with smaller weights are simpler

12 Given n training points with d features, we define: : matrix storing points X R n d y : real-valued labels R n ŷ : predicted labels, where R n ŷ = Xw w : regression parameters / model to learn R d Ridge Regression: Learn mapping ( ) that minimizes residual sum of squares along with a regularization term: w Training Error Model Complexity min w Xw y λ w 2 2 free parameter trades off Closed-form solution: w =(X X + λi d ) 1 X y between training error and model complexity

13 Millionsong Regression Pipeline

14 training set model full dataset new entity test set accuracy prediction Obtain Raw Data Split Data Feature Extraction Supervised Learning Pipeline Supervised Learning Evaluation Predict

15 training set model full dataset new entity test set accuracy prediction Obtain Raw Data Goal: Predict song s release year from audio features Raw Data: Millionsong Dataset from UCI ML Repository Western, commercial tracks from timbre averages (features) and release year (label) Split Data Feature Extraction Supervised Learning Evaluation Predict

16 training set model full dataset new entity test set accuracy prediction Obtain Raw Data Split Data: Train on training set, evaluate with test set Test set simulates unobserved data Test error tells us whether we ve generalized well Split Data Feature Extraction Supervised Learning Evaluation Predict

17 training set model full dataset new entity test set accuracy prediction Obtain Raw Data Feature Extraction: Quadratic features Compute pairwise feature interactions Captures covariance of initial timbre features Leads to a non-linear model relative to raw features Split Data Feature Extraction Supervised Learning Evaluation Predict

18 Given 2 dimensional data, quadratic features are: x = x 1 x 2 = Φ(x) = x 2 1 x 1 x 2 x 2 x 1 x 2 2 z = z 1 z 2 = Φ(z) = z 2 1 z 1 z 2 z 2 z 1 z 2 2 More succinctly: Φ (x) = x 2 1 2x 1 x 2 x 2 2 Φ (z) = z 2 1 2z 1 z 2 z 2 2 Equivalent inner products: Φ(x) Φ(z) = x 2 1z x 1 x 2 z 1 z 2 + x 2 2z 2 2 = Φ (x) Φ (z)

training set model full dataset new entity test set accuracy prediction Obtain Raw Data Supervised Learning: Least Squares Regression Learn a mapping

19 training set model full dataset new entity test set accuracy prediction Obtain Raw Data Supervised Learning: Least Squares Regression Learn a mapping from entities to continuous labels given a training set Audio features Song year Split Data Feature Extraction Supervised Learning Evaluation Predict

20 Given n training points with d features, we define: : matrix storing points X R n d y : real-valued labels R n ŷ : predicted labels, where R n ŷ = Xw w : regression parameters / model to learn R d Ridge Regression: Learn mapping ( ) that minimizes residual sum of squares along with a regularization term: w Training Error Model Complexity min w Xw y λ w 2 2 Closed-form solution: w =(X X + λi d ) 1 X y

21 Ridge Regression: Learn mapping ( ) that minimizes residual sum of squares along with a regularization term: w Training Error Model Complexity min w Xw y λ w 2 2 free parameter trades off between training error and model complexity How do we choose a good value for this free parameter? Most methods have free parameters / hyperparameters to tune First thought: Search over multiple values, evaluate each on test set But, goal of test set is to simulate unobserved data We may overfit if we use it to choose hyperparameters Second thought: Create another hold out dataset for this search

22 training set model full dataset validation set new entity test set accuracy prediction Obtain Raw Data Evaluation (Part 1): Hyperparameter tuning Training: train various models Validation: evaluate various models (e.g., Grid Search) Test: evaluate final model s accuracy Split Data Feature Extraction Supervised Learning Evaluation Predict

23 Regulariza*on-Parameter-(--) λ Grid Search: Exhaustively search through hyperparameter space Define and discretize search space (linear or log scale) Evaluate points via validation error

24 Hyperparameter Regulariza*on-Parameter-(--) λ Hyperparameter-1 Grid Search: Exhaustively search through hyperparameter space Define and discretize search space (linear or log scale) Evaluate points via validation error

25 Evaluating Predictions How can we compare labels and predictions for n validation points? Least squares optimization involves squared loss, (y ŷ) 2, so it seems reasonable to use mean squared error (MSE): MSE = 1 n n i=1 (ŷ (i) y (i) ) 2 But MSE s unit of measurement is square of quantity being measured, e.g., squared years for song prediction More natural to use root-mean-square error (RMSE), i.e., MSE

training set model full dataset validation set new entity test set accuracy prediction Obtain Raw Data Evaluation (Part 2): Evaluate final model Training set: train

26 training set model full dataset validation set new entity test set accuracy prediction Obtain Raw Data Evaluation (Part 2): Evaluate final model Training set: train various models Validation set: evaluate various models Test set: evaluate final model s accuracy Split Data Feature Extraction Supervised Learning Evaluation Predict

27 training set model full dataset validation set new entity test set accuracy prediction Obtain Raw Data Split Data Predict: Final model can then be used to make predictions on future observations, e.g., new songs Feature Extraction Supervised Learning Evaluation Predict

28 Distributed ML: Computation and Storage

29 Challenge: Scalability Classic ML techniques are not always suitable for modern datasets 60" 50" Moore's"Law" Data Grows Faster 40" 30" 20" Overall"Data" Par8cle"Accel." DNA"Sequencers" than Moore s Law [IDC report, Kathy Yelick, LBNL] 10" 0" 2010" 2011" 2012" 2013" 2014" 2015" Data Machine Learning Distributed Computing

30 Least Squares Regression: Learn mapping ( ) from features to labels that minimizes residual sum of squares: min w Xw y 2 2 w Closed form solution: w =(X X) 1 X y (if inverse exists) How do we solve this computationally? Computational profile similar for Ridge Regression

31 Computing Closed Form Solution w =(X X) 1 X y Computation: O(nd 2 + d 3 ) operations Consider number of arithmetic operations ( +,,, / ) Computational bottlenecks: Matrix multiply of X X : O(nd 2 ) operations Matrix inverse: O(d 3 ) operations Other methods (Cholesky, QR, SVD) have same complexity

32 Storage Requirements w =(X X) 1 X y Computation: O(nd 2 + d 3 ) operations Storage: O(nd + d 2 ) floats Consider storing values as floats (8 bytes) Storage bottlenecks: X X and its inverse: O(d 2 ) floats X : O(nd) floats

33 Big n and Small d w =(X X) 1 X y Computation: O(nd 2 + d 3 ) operations Storage: O(nd + d 2 ) floats Assume O(d 3 ) computation and O(d 2 ) storage feasible on single machine X Storing and computing X X are the bottlenecks Can distribute storage and computation! Store data points (rows of X ) across machines Compute X X as a sum of outer products

34 Matrix Multiplication via Inner Products Each entry of output matrix is result of inner product of inputs matrices = = 28

35 Matrix Multiplication via Inner Products Each entry of output matrix is result of inner product of inputs matrices =

36 Matrix Multiplication via Inner Products Each entry of output matrix is result of inner product of inputs matrices =

37 Matrix Multiplication via Outer Products Output matrix is sum of outer products between corresponding rows and columns of input matrices =

38 Matrix Multiplication via Outer Products Output matrix is sum of outer products between corresponding rows and columns of input matrices =

39 Matrix Multiplication via Outer Products Output matrix is sum of outer products between corresponding rows and columns of input matrices =

40 Matrix Multiplication via Outer Products Output matrix is sum of outer products between corresponding rows and columns of input matrices =

41 d x (1) x (2) n x (n) d x (1) x (2) n X X = n = x (n) i=1 Example: n = 6; 3 workers workers: x (1) x (5) x (4) x (6) x (3) x (2) O(nd) Distributed Storage map: O(nd 2 ) Distributed O(d 2 ) Local Storage Computation reduce: ( ) -1 O(d 3 ) Local Computation O(d 2 ) Local Storage

42 d x (1) x (2) n x (n) d x (1) x (2) n X X = n = x (n) i=1 Example: n = 6; 3 workers workers: x (1) x (5) x (4) x (6) x (3) x (2) O(nd) Distributed Storage map: O(nd 2 ) Distributed O(d 2 ) Local Storage Computation reduce: ( ) -1 O(d 3 ) Local Computation O(d 2 ) Local Storage

43 Distributed ML: Computation and Storage, Part II

44 Big n and Small d w =(X X) 1 X y Computation: O(nd 2 + d 3 ) operations Storage: O(nd + d 2 ) floats Assume O(d 3 ) computation and O(d 2 ) storage feasible on single machine Can distribute storage and computation! Store data points (rows of X ) across machines Compute X X as a sum of outer products

45 Big n and Small d w =(X X) 1 X y Computation: O(nd 2 + d 3 ) operations Storage: O(nd + d 2 ) floats Assume O(d 3 ) computation and O(d 2 ) storage feasible on single machine Can distribute storage and computation! Store data points (rows of X ) across machines Compute X X as a sum of outer products

46 Big n and Big d w =(X X) 1 X y Computation: O(nd 2 + d 3 ) operations Storage: O(nd + d 2 ) floats X As before, storing and computing X X are bottlenecks Now, storing and operating on Can t easily distribute! X X is also a bottleneck

47 d x (1) x (2) n x (n) d x (1) x (2) n X X = n = x (n) i=1 Example: n = 6; 3 workers workers: x (1) x (5) x (4) x (6) x (3) x (2) O(nd) Distributed Storage map: O(nd 2 ) Distributed O(d 2 ) Local Storage Computation reduce: ( ) -1 O(d 3 ) Local Computation O(d 2 ) Local Storage

48 Big n and Big d w =(X X) 1 X y Computation: O(nd 2 + d 3 ) operations Storage: O(nd + d 2 ) floats X As before, storing and computing X X are bottlenecks Now, storing and operating on Can t easily distribute! X X is also a bottleneck 1st Rule of thumb Computation and storage should be linear (in n, d)

49 Big n and Big d We need methods that are linear in time and space One idea: Exploit sparsity Explicit sparsity can provide orders of magnitude storage and computational gains Sparse data is prevalent Text processing: bag-of-words, n-grams Collaborative filtering: ratings matrix Graphs: adjacency matrix Categorical features: one-hot-encoding Genomics: SNPs, variant calling dense : size : 7 >< sparse : indices : 0 6 >: values : 1. 3.

50 Big n and Big d We need methods that are linear in time and space One idea: Exploit sparsity Explicit sparsity can provide orders of magnitude storage and computational gains Latent sparsity assumption can be used to reduce dimension, e.g., PCA, low-rank approximation (unsupervised learning) d r d r Low-rank n n

51 Big n and Big d We need methods that are linear in time and space One idea: Exploit sparsity Explicit sparsity can provide orders of magnitude storage and computational gains Latent sparsity assumption can be used to reduce dimension, e.g., PCA, low-rank approximation (unsupervised learning) Another idea: Use different algorithms Gradient descent is an iterative algorithm that requires O(nd) computation and O(d) local storage per iteration

52 Closed Form Solution for Big n and Big d Example: n = 6; 3 workers workers: x (1) x (5) x (4) x (6) x (3) x (2) O(nd) Distributed Storage map: O(nd 2 ) Distributed O(d 2 ) Local Storage Computation reduce: ( ) -1 O(d 3 ) Local O(d 2 ) Local Computation Storage

53 Gradient Descent for Big n and Big d Example: n = 6; 3 workers workers: x (1) x (5) x (4) x (6) x (3) x (2) O(nd) Distributed Storage map: O(nd 2 ) Distributed O(d 2 ) Local Storage Computation reduce: ( ) -1 O(d 3 ) Local O(d 2 ) Local Computation Storage

54 Gradient Descent for Big n and Big d Example: n = 6; 3 workers workers: x (1) x (5) x (4) x (6) x (3) x (2) map:??? O(nd) Distributed Storage O(nd) O(nd 2 O(d) ) O(d 2 ) Local Distributed Storage Computation reduce: ( ) -1 O(d 3 ) Local O(d 2 ) Local Computation Storage

55 Gradient Descent for Big n and Big d Example: n = 6; 3 workers workers: x (1) x (5) x (4) x (6) x (3) x (2) map:??? O(nd) Distributed Storage O(nd) O(nd 2 O(d) ) O(d 2 ) Local Distributed Storage Computation O(d) O(d) reduce:? O(d 3 ) Local O(d 2 ) Local Computation Storage

Regression. Goal: Learn a mapping from observations (features) to continuous labels given a training set (supervised learning)

Regression. Goal: Learn a mapping from observations (features) to continuous labels given a training set (supervised learning) Linear Regression Regression Goal: Learn a mapping from observations (features) to continuous labels given a training set (supervised learning) Example: Height, Gender, Weight Shoe Size Audio features