Lasso, Ridge, and Elastic Net
|
|
- Lorraine Foster
- 6 years ago
- Views:
Transcription
1 Lasso, Ridge, and Elastic Net David Rosenberg New York University February 7, 2017 David Rosenberg (New York University) DS-GA 1003 February 7, / 29
2 Linearly Dependent Features Linearly Dependent Features David Rosenberg (New York University) DS-GA 1003 February 7, / 29
3 Linearly Dependent Features Suppose We Have 2 Equal Features Input features: x 1,x 2 R. Outcome: y R. Linear prediction functions f (x) = w 1 x 2 + w 2 x 2 Suppose x 1 = x 2. Then all functions with w 1 + w 2 = k are the same. give same predictions and have same empirical risk What function will we select if we do ERM with l 1 or l 2 constraint? David Rosenberg (New York University) DS-GA 1003 February 7, / 29
4 Linearly Dependent Features Equal Features, l 2 Constraint w 2 w 1 + w 2 = w 2 2 w 1 + w 2 = w 1 + w 2 = w 1 w 1 + w 2 = w 1 + w 2 = w 1 + w 2 = 2 2 Suppose the line w 1 + w 2 = corresponds to the empirical risk minimizers. Empirical risk increase as we move away from these parameter settings Intersection of w 1 + w 2 = 2 2 and the norm ball w 2 2 is ridge solution. Note that w 1 = w 2 at the solution David Rosenberg (New York University) DS-GA 1003 February 7, / 29
5 Linearly Dependent Features Equal Features, l 1 Constraint w2 w1 + w2 = w1 + w2 = 9 w 1 2 w1 + w2 = 7.25 w1 + w2 = 5.5 w1 w1 + w2 = 3.75 w1 + w2 = 2 Suppose the line w 1 + w 2 = 5.5 corresponds to the empirical risk minimizers. Intersection of w 1 + w 2 = 2 and the norm ball w 1 2 is lasso solution. Note that the solution set is {(w 1,w 2 ) : w 1 + w 2 = 2,w 1,w 2 0}. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
6 Linearly Dependent Features Linearly Related Features Same setup, now suppose x 2 = 2x 1. Then all functions with w 1 + 2w 2 = k are the same. give same predictions and have same empirical risk What function will we select if we do ERM with l 1 or l 2 constraint? David Rosenberg (New York University) DS-GA 1003 February 7, / 29
7 Linearly Dependent Features Linearly Related Features, l 2 Constraint w 2 2 w 2 w 1 + 2w 2 = 10/ w 1 + 2w 2 = 10/ w 1 + 2w 2 = 10/ w 1 + 2w 2 = 10/ 5 w 1 w 1 + 2w 2 = 10/ corresponds to the empirical risk minimizers. Intersection of w 1 + 2w 2 = 10 5 and the norm ball w 2 2 is ridge solution. At solution, w 2 = 2w 1. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
8 Linearly Dependent Features Linearly Related Features, l 1 Constraint w 2 w 1 + 2w 2 = 16 w 1 + 2w 2 = 12 w 1 2 w 1 + 2w 2 = 8 w 1 w 1 + 2w 2 = 4 Intersection of w 1 + 2w 2 = 4 and the norm ball w 1 2 is lasso solution. Solution is now a corner of the l 1 ball, corresonding to a sparse solution. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
9 Linearly Dependent Features Linearly Dependent Features: Take Away For identical features l 1 regularization spreads weight arbitrarily (all weights same sign) l 2 regularization spreads weight evenly Linearly related features l 1 regularization chooses variable with larger scale, 0 weight to others l 2 prefers variables with larger scale spreads weight proportional to scale David Rosenberg (New York University) DS-GA 1003 February 7, / 29
10 Linearly Dependent Features Empirical Risk for Square Loss and Linear Predictors Recall our discussion of linear predictors f (x) = w T x and square loss. Sets of w giving same empirical risk (i.e. level sets) formed ellipsoids around the ERM. With x 1 and x 2 linearly related, we get a degenerate ellipse. That s why level sets were lines (actually pairs of lines, one on each side of ERM). KPM Fig David Rosenberg (New York University) DS-GA 1003 February 7, / 29
11 Linearly Dependent Features Correlated Features Same Scale Suppose x 1 and x 2 are highly correlated and the same scale. This is quite typical in real data, after normalizing data. Nothing degenerate here, so level sets are ellipsoids. But, the higher the correlation, the closer to degenerate we get. That is, ellipsoids keep stretching out, getting closer to two parallel lines. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
12 Linearly Dependent Features Correlated Features, l 1 Regularization w2 w2 w 1 2 w 1 2 w1 w1 Intersection could be anywhere on the top right edge. Minor perturbations (in data) can drastically change intersection point very unstable solution. Makes division of weight among highly correlated features (of same scale) seem arbitrary. If x 1 2x 2, ellipse changes orientation and we probably hit a corner. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
13 Linearly Dependent Features A Very Simple Model Suppose we have one feature x 1 R. Response variable y R. Got some data and ran least squares linear regression. The ERM is What happens if we get a new feature x 2, but we always have x 2 = x 1? ˆf (x 1 ) = 4x 1. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
14 Linearly Dependent Features Duplicate Features New feature x 2 gives no new information. ERM is still ˆf (x 1,x 2 ) = 4x 1. Now there are some more ERMs: ˆf (x 1,x 2 ) = 2x 1 + 2x 2 ˆf (x 1,x 2 ) = x 1 + 3x 2 ˆf (x 1,x 2 ) = 4x 2 What if we introduce l 1 or l 2 regularization? David Rosenberg (New York University) DS-GA 1003 February 7, / 29
15 Linearly Dependent Features Duplicate Features: l 1 and l 2 norms ˆf (x 1,x 2 ) = w 1 x 1 + w 2 x 2 is an ERM iff w 1 + w 2 = 4. Consider the l 1 and l 2 norms of various solutions: w 1 w 2 w 1 w w 1 doesn t discriminate, as long as all have same sign w 2 2 minimized when weight is spread equally Picture proof: Level sets of loss are lines of the form w 1 + w 2 = c... David Rosenberg (New York University) DS-GA 1003 February 7, / 29
16 Correlated Features and the Grouping Issue Correlated Features and the Grouping Issue David Rosenberg (New York University) DS-GA 1003 February 7, / 29
17 Correlated Features and the Grouping Issue Example with highly correlated features Model in words: y is a linear combination of z 1 and z 2 But we don t observe z 1 and z 2 directly. We get 3 noisy observations of z 1. We get 3 noisy observations of z 2. We want to predict y from our noisy observations. Example from Section 4.2 in Hastie et al s Statistical Learning with Sparsity. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
18 Correlated Features and the Grouping Issue Example with highly correlated features Suppose (x, y) generated as follows: z 1,z 2 N(0,1) (independent) ε 0,ε 1,...,ε 6 N(0,1) (independent) y = 1 1.5z 2 + 2ε 0 3z { x j = z 1 + ε j /5 for j = 1,2,3 z 2 + ε j /5 for j = 4,5,6 Generated a sample of (x,y) pairs of size 100. Correlations within the groups of x s were around Example from Section 4.2 in Hastie et al s Statistical Learning with Sparsity. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
19 Correlated Features and the Grouping Issue Example with highly correlated features Lasso regularization paths: Lasso Paths Elastic-Net Paths coefficients coefficients λ λ 1 Lines with the same color correspond to features with essentially the same information Distribution of weight among them seems almost arbitrary David Rosenberg (New York University) DS-GA 1003 February 7, / 29
20 Correlated Features and the Grouping Issue Hedge Bets When Variables Highly Correlated When variables are highly correlated (and same scale, after normalization), we want to give them roughly the same weight. Why? Let their error cancel out How can we get the weight spread more evenly? David Rosenberg (New York University) DS-GA 1003 February 7, / 29
21 Correlated Features and the Grouping Issue Elastic Net The elastic net combines lasso and ridge penalties: 1 ŵ = arg min w R d n n { w T } 2 x i y i + λ1 w 1 + λ 2 w 2 2 We expect correlated random variables to have similar coefficients. i=1 David Rosenberg (New York University) DS-GA 1003 February 7, / 29
22 Correlated Features and the Grouping Issue Highly Correlated Features, Elastic Net Constraint w2.8 w w w1 Elastic net solution is closer to w 2 = w 1 line, despite high correlation. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
23 Correlated Features and the Grouping Issue Elastic Net - Sparse Regions Suppose design matrix X is orthogonal, so X T X = I, and contours are circles (and features uncorrelated) Then OLS solution in green or red regions implies elastic-net constrained solution will be at corner Fig from Mairal et al. s Sparse Modeling for Image and Vision Processing Fig 1.9 David Rosenberg (New York University) DS-GA 1003 February 7, / 29
24 Correlated Features and the Grouping Issue Elastic Net A Theorem for Correlated Variables Theorem a Let ρ ij = ĉorr(x i,x j ). Suppose ŵ i and ŵ j are selected by elastic net, with y centered and predictors x 1,...,x d standardized. If ŵ i ŵ j > 0, then ŵ i ŵ j y 2 λ 2 1 ρij. a Theorem 1 in Regularization and variable selection via the elastic net : David Rosenberg (New York University) DS-GA 1003 February 7, / 29
25 Correlated Features and the Grouping Issue Elastic Net Results on Model Lasso Paths Elastic-Net Paths coefficients coefficients coefficients coefficients λ 1 λ λ 1 λ 1 Lasso on left; Elastic net on right. Ratio of l 2 to l 1 regularization roughly 2 : 1. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
26 Extra Pictures Extra Pictures David Rosenberg (New York University) DS-GA 1003 February 7, / 29
27 Extra Pictures Elastic Net vs Lasso Norm Ball From Figure 4.2 of Hastie et al s Statistical Learning with Sparsity. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
28 Extra Pictures The ( l q ) q Constraint Generalize to l q : ( w q ) q = w 1 q + w 2 q. Note: w q is a norm if q 1, but not for q (0,1) F = {f (x) = w 1 x 1 + w 2 x 2 }. Contours of w q q = w 1 q + w 2 q : David Rosenberg (New York University) DS-GA 1003 February 7, / 29
29 Extra Pictures l 1.2 vs Elastic Net From Hastie et al s Elements of Statistical Learning. David Rosenberg (New York University) DS-GA 1003 February 7, / 29
Lasso, Ridge, and Elastic Net
Lasso, Ridge, and Elastic Net David Rosenberg New York University October 29, 2016 David Rosenberg (New York University) DS-GA 1003 October 29, 2016 1 / 14 A Very Simple Model Suppose we have one feature
More informationl 1 and l 2 Regularization
David Rosenberg New York University February 5, 2015 David Rosenberg (New York University) DS-GA 1003 February 5, 2015 1 / 32 Tikhonov and Ivanov Regularization Hypothesis Spaces We ve spoken vaguely about
More informationMachine Learning and Computational Statistics, Spring 2017 Homework 2: Lasso Regression
Machine Learning and Computational Statistics, Spring 2017 Homework 2: Lasso Regression Due: Monday, February 13, 2017, at 10pm (Submit via Gradescope) Instructions: Your answers to the questions below,
More informationClassification. The goal: map from input X to a label Y. Y has a discrete set of possible values. We focused on binary Y (values 0 or 1).
Regression and PCA Classification The goal: map from input X to a label Y. Y has a discrete set of possible values We focused on binary Y (values 0 or 1). But we also discussed larger number of classes
More informationStatistical Data Mining and Machine Learning Hilary Term 2016
Statistical Data Mining and Machine Learning Hilary Term 2016 Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: http://www.stats.ox.ac.uk/~sejdinov/sdmml Naïve Bayes
More informationLINEAR REGRESSION, RIDGE, LASSO, SVR
LINEAR REGRESSION, RIDGE, LASSO, SVR Supervised Learning Katerina Tzompanaki Linear regression one feature* Price (y) What is the estimated price of a new house of area 30 m 2? 30 Area (x) *Also called
More informationLinear Methods for Regression. Lijun Zhang
Linear Methods for Regression Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Outline Introduction Linear Regression Models and Least Squares Subset Selection Shrinkage Methods Methods Using Derived
More informationLinear regression COMS 4771
Linear regression COMS 4771 1. Old Faithful and prediction functions Prediction problem: Old Faithful geyser (Yellowstone) Task: Predict time of next eruption. 1 / 40 Statistical model for time between
More informationMSA220/MVE440 Statistical Learning for Big Data
MSA220/MVE440 Statistical Learning for Big Data Lecture 9-10 - High-dimensional regression Rebecka Jörnsten Mathematical Sciences University of Gothenburg and Chalmers University of Technology Recap from
More informationCOMS 4721: Machine Learning for Data Science Lecture 6, 2/2/2017
COMS 4721: Machine Learning for Data Science Lecture 6, 2/2/2017 Prof. John Paisley Department of Electrical Engineering & Data Science Institute Columbia University UNDERDETERMINED LINEAR EQUATIONS We
More informationLinear Regression. Aarti Singh. Machine Learning / Sept 27, 2010
Linear Regression Aarti Singh Machine Learning 10-701/15-781 Sept 27, 2010 Discrete to Continuous Labels Classification Sports Science News Anemic cell Healthy cell Regression X = Document Y = Topic X
More informationDATA MINING AND MACHINE LEARNING
DATA MINING AND MACHINE LEARNING Lecture 5: Regularization and loss functions Lecturer: Simone Scardapane Academic Year 2016/2017 Table of contents Loss functions Loss functions for regression problems
More informationSparse Linear Models (10/7/13)
STA56: Probabilistic machine learning Sparse Linear Models (0/7/) Lecturer: Barbara Engelhardt Scribes: Jiaji Huang, Xin Jiang, Albert Oh Sparsity Sparsity has been a hot topic in statistics and machine
More informationLinear Model Selection and Regularization
Linear Model Selection and Regularization Recall the linear model Y = β 0 + β 1 X 1 + + β p X p + ɛ. In the lectures that follow, we consider some approaches for extending the linear model framework. In
More informationParameter Norm Penalties. Sargur N. Srihari
Parameter Norm Penalties Sargur N. srihari@cedar.buffalo.edu 1 Regularization Strategies 1. Parameter Norm Penalties 2. Norm Penalties as Constrained Optimization 3. Regularization and Underconstrained
More informationSubgradient Descent. David S. Rosenberg. New York University. February 7, 2018
Subgradient Descent David S. Rosenberg New York University February 7, 2018 David S. Rosenberg (New York University) DS-GA 1003 / CSCI-GA 2567 February 7, 2018 1 / 43 Contents 1 Motivation and Review:
More informationCS 4491/CS 7990 SPECIAL TOPICS IN BIOINFORMATICS
CS 4491/CS 7990 SPECIAL TOPICS IN BIOINFORMATICS * Some contents are adapted from Dr. Hung Huang and Dr. Chengkai Li at UT Arlington Mingon Kang, Ph.D. Computer Science, Kennesaw State University Problems
More informationLinear Regression with Strongly Correlated Designs Using Ordered Weigthed l 1
Linear Regression with Strongly Correlated Designs Using Ordered Weigthed l 1 ( OWL ) Regularization Mário A. T. Figueiredo Instituto de Telecomunicações and Instituto Superior Técnico, Universidade de
More informationLinear Regression Linear Regression with Shrinkage
Linear Regression Linear Regression ith Shrinkage Introduction Regression means predicting a continuous (usually scalar) output y from a vector of continuous inputs (features) x. Example: Predicting vehicle
More informationConvex optimization COMS 4771
Convex optimization COMS 4771 1. Recap: learning via optimization Soft-margin SVMs Soft-margin SVM optimization problem defined by training data: w R d λ 2 w 2 2 + 1 n n [ ] 1 y ix T i w. + 1 / 15 Soft-margin
More informationSTAT 462-Computational Data Analysis
STAT 462-Computational Data Analysis Chapter 5- Part 2 Nasser Sadeghkhani a.sadeghkhani@queensu.ca October 2017 1 / 27 Outline Shrinkage Methods 1. Ridge Regression 2. Lasso Dimension Reduction Methods
More informationLinear Regression. Volker Tresp 2018
Linear Regression Volker Tresp 2018 1 Learning Machine: The Linear Model / ADALINE As with the Perceptron we start with an activation functions that is a linearly weighted sum of the inputs h = M j=0 w
More informationChapter 3. Linear Models for Regression
Chapter 3. Linear Models for Regression Wei Pan Division of Biostatistics, School of Public Health, University of Minnesota, Minneapolis, MN 55455 Email: weip@biostat.umn.edu PubH 7475/8475 c Wei Pan Linear
More informationMS&E 226. In-Class Midterm Examination Solutions Small Data October 20, 2015
MS&E 226 In-Class Midterm Examination Solutions Small Data October 20, 2015 PROBLEM 1. Alice uses ordinary least squares to fit a linear regression model on a dataset containing outcome data Y and covariates
More informationSparse regression. Optimization-Based Data Analysis. Carlos Fernandez-Granda
Sparse regression Optimization-Based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_spring16 Carlos Fernandez-Granda 3/28/2016 Regression Least-squares regression Example: Global warming Logistic
More informationGeneralization theory
Generalization theory Daniel Hsu Columbia TRIPODS Bootcamp 1 Motivation 2 Support vector machines X = R d, Y = { 1, +1}. Return solution ŵ R d to following optimization problem: λ min w R d 2 w 2 2 + 1
More informationSCMA292 Mathematical Modeling : Machine Learning. Krikamol Muandet. Department of Mathematics Faculty of Science, Mahidol University.
SCMA292 Mathematical Modeling : Machine Learning Krikamol Muandet Department of Mathematics Faculty of Science, Mahidol University February 9, 2016 Outline Quick Recap of Least Square Ridge Regression
More informationORIE 4741: Learning with Big Messy Data. Regularization
ORIE 4741: Learning with Big Messy Data Regularization Professor Udell Operations Research and Information Engineering Cornell October 26, 2017 1 / 24 Regularized empirical risk minimization choose model
More informationA New Combined Approach for Inference in High-Dimensional Regression Models with Correlated Variables
A New Combined Approach for Inference in High-Dimensional Regression Models with Correlated Variables Niharika Gauraha and Swapan Parui Indian Statistical Institute Abstract. We consider the problem of
More informationCOMS 4771 Lecture Fixed-design linear regression 2. Ridge and principal components regression 3. Sparse regression and Lasso
COMS 477 Lecture 6. Fixed-design linear regression 2. Ridge and principal components regression 3. Sparse regression and Lasso / 2 Fixed-design linear regression Fixed-design linear regression A simplified
More informationConvex relaxation for Combinatorial Penalties
Convex relaxation for Combinatorial Penalties Guillaume Obozinski Equipe Imagine Laboratoire d Informatique Gaspard Monge Ecole des Ponts - ParisTech Joint work with Francis Bach Fête Parisienne in Computation,
More informationESL Chap3. Some extensions of lasso
ESL Chap3 Some extensions of lasso 1 Outline Consistency of lasso for model selection Adaptive lasso Elastic net Group lasso 2 Consistency of lasso for model selection A number of authors have studied
More informationSimultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR
Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR Howard D. Bondell and Brian J. Reich Department of Statistics, North Carolina State University,
More informationMachine Learning. Regularization and Feature Selection. Fabio Vandin November 14, 2017
Machine Learning Regularization and Feature Selection Fabio Vandin November 14, 2017 1 Regularized Loss Minimization Assume h is defined by a vector w = (w 1,..., w d ) T R d (e.g., linear models) Regularization
More informationRegularization and Variable Selection via the Elastic Net
p. 1/1 Regularization and Variable Selection via the Elastic Net Hui Zou and Trevor Hastie Journal of Royal Statistical Society, B, 2005 Presenter: Minhua Chen, Nov. 07, 2008 p. 2/1 Agenda Introduction
More informationOptimization methods
Optimization methods Optimization-Based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_spring16 Carlos Fernandez-Granda /8/016 Introduction Aim: Overview of optimization methods that Tend to
More informationCOS513: FOUNDATIONS OF PROBABILISTIC MODELS LECTURE 10
COS53: FOUNDATIONS OF PROBABILISTIC MODELS LECTURE 0 MELISSA CARROLL, LINJIE LUO. BIAS-VARIANCE TRADE-OFF (CONTINUED FROM LAST LECTURE) If V = (X n, Y n )} are observed data, the linear regression problem
More informationSTAT 535 Lecture 5 November, 2018 Brief overview of Model Selection and Regularization c Marina Meilă
STAT 535 Lecture 5 November, 2018 Brief overview of Model Selection and Regularization c Marina Meilă mmp@stat.washington.edu Reading: Murphy: BIC, AIC 8.4.2 (pp 255), SRM 6.5 (pp 204) Hastie, Tibshirani
More informationLinear Regression Linear Regression with Shrinkage
Linear Regression Linear Regression ith Shrinkage Introduction Regression means predicting a continuous (usually scalar) output y from a vector of continuous inputs (features) x. Example: Predicting vehicle
More informationMS-C1620 Statistical inference
MS-C1620 Statistical inference 10 Linear regression III Joni Virta Department of Mathematics and Systems Analysis School of Science Aalto University Academic year 2018 2019 Period III - IV 1 / 32 Contents
More informationLecture 14: Variable Selection - Beyond LASSO
Fall, 2017 Extension of LASSO To achieve oracle properties, L q penalty with 0 < q < 1, SCAD penalty (Fan and Li 2001; Zhang et al. 2007). Adaptive LASSO (Zou 2006; Zhang and Lu 2007; Wang et al. 2007)
More informationPrediction & Feature Selection in GLM
Tarigan Statistical Consulting & Coaching statistical-coaching.ch Doctoral Program in Computer Science of the Universities of Fribourg, Geneva, Lausanne, Neuchâtel, Bern and the EPFL Hands-on Data Analysis
More informationLASSO Review, Fused LASSO, Parallel LASSO Solvers
Case Study 3: fmri Prediction LASSO Review, Fused LASSO, Parallel LASSO Solvers Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade May 3, 2016 Sham Kakade 2016 1 Variable
More informationOWL to the rescue of LASSO
OWL to the rescue of LASSO IISc IBM day 2018 Joint Work R. Sankaran and Francis Bach AISTATS 17 Chiranjib Bhattacharyya Professor, Department of Computer Science and Automation Indian Institute of Science,
More informationLecture 17 Intro to Lasso Regression
Lecture 17 Intro to Lasso Regression 11 November 2015 Taylor B. Arnold Yale Statistics STAT 312/612 Notes problem set 5 posted; due today Goals for today introduction to lasso regression the subdifferential
More informationBasic Surveying Week 3, Lesson 2 Semester 2017/18/2 Vectors, equation of line, circle, ellipse
Basic Surveying Week 3, Lesson Semester 017/18/ Vectors, equation of line, circle, ellipse 1. Introduction In surveying calculations, we use the two or three dimensional coordinates of points or objects
More information(Part 1) High-dimensional statistics May / 41
Theory for the Lasso Recall the linear model Y i = p j=1 β j X (j) i + ɛ i, i = 1,..., n, or, in matrix notation, Y = Xβ + ɛ, To simplify, we assume that the design X is fixed, and that ɛ is N (0, σ 2
More informationMachine Learning for Big Data CSE547/STAT548, University of Washington Emily Fox February 4 th, Emily Fox 2014
Case Study 3: fmri Prediction Fused LASSO LARS Parallel LASSO Solvers Machine Learning for Big Data CSE547/STAT548, University of Washington Emily Fox February 4 th, 2014 Emily Fox 2014 1 LASSO Regression
More informationLinear regression methods
Linear regression methods Most of our intuition about statistical methods stem from linear regression. For observations i = 1,..., n, the model is Y i = p X ij β j + ε i, j=1 where Y i is the response
More informationShrinkage Methods: Ridge and Lasso
Shrinkage Methods: Ridge and Lasso Jonathan Hersh 1 Chapman University, Argyros School of Business hersh@chapman.edu February 27, 2019 J.Hersh (Chapman) Ridge & Lasso February 27, 2019 1 / 43 1 Intro and
More informationIntroduction to Machine Learning
Introduction to Machine Learning Linear Regression Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574 1
More informationThe prediction of house price
000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050
More informationMIT 9.520/6.860, Fall 2018 Statistical Learning Theory and Applications. Class 08: Sparsity Based Regularization. Lorenzo Rosasco
MIT 9.520/6.860, Fall 2018 Statistical Learning Theory and Applications Class 08: Sparsity Based Regularization Lorenzo Rosasco Learning algorithms so far ERM + explicit l 2 penalty 1 min w R d n n l(y
More informationRecap from previous lecture
Recap from previous lecture Learning is using past experience to improve future performance. Different types of learning: supervised unsupervised reinforcement active online... For a machine, experience
More informationToday. Calculus. Linear Regression. Lagrange Multipliers
Today Calculus Lagrange Multipliers Linear Regression 1 Optimization with constraints What if I want to constrain the parameters of the model. The mean is less than 10 Find the best likelihood, subject
More informationMachine Learning for Biomedical Engineering. Enrico Grisan
Machine Learning for Biomedical Engineering Enrico Grisan enrico.grisan@dei.unipd.it Curse of dimensionality Why are more features bad? Redundant features (useless or confounding) Hard to interpret and
More informationThe Learning Problem and Regularization Class 03, 11 February 2004 Tomaso Poggio and Sayan Mukherjee
The Learning Problem and Regularization 9.520 Class 03, 11 February 2004 Tomaso Poggio and Sayan Mukherjee About this class Goal To introduce a particularly useful family of hypothesis spaces called Reproducing
More informationMark your answers ON THE EXAM ITSELF. If you are not sure of your answer you may wish to provide a brief explanation.
CS 189 Spring 2015 Introduction to Machine Learning Midterm You have 80 minutes for the exam. The exam is closed book, closed notes except your one-page crib sheet. No calculators or electronic items.
More informationDiscriminative Models
No.5 Discriminative Models Hui Jiang Department of Electrical Engineering and Computer Science Lassonde School of Engineering York University, Toronto, Canada Outline Generative vs. Discriminative models
More informationMachine Learning for Economists: Part 4 Shrinkage and Sparsity
Machine Learning for Economists: Part 4 Shrinkage and Sparsity Michal Andrle International Monetary Fund Washington, D.C., October, 2018 Disclaimer #1: The views expressed herein are those of the authors
More informationInstitute of Statistics Mimeo Series No Simultaneous regression shrinkage, variable selection and clustering of predictors with OSCAR
DEPARTMENT OF STATISTICS North Carolina State University 2501 Founders Drive, Campus Box 8203 Raleigh, NC 27695-8203 Institute of Statistics Mimeo Series No. 2583 Simultaneous regression shrinkage, variable
More informationClassification Logistic Regression
Announcements: Classification Logistic Regression Machine Learning CSE546 Sham Kakade University of Washington HW due on Friday. Today: Review: sub-gradients,lasso Logistic Regression October 3, 26 Sham
More informationMIT 9.520/6.860, Fall 2018 Statistical Learning Theory and Applications. Class 04: Features and Kernels. Lorenzo Rosasco
MIT 9.520/6.860, Fall 2018 Statistical Learning Theory and Applications Class 04: Features and Kernels Lorenzo Rosasco Linear functions Let H lin be the space of linear functions f(x) = w x. f w is one
More informationStatistical Methods for Data Mining
Statistical Methods for Data Mining Kuangnan Fang Xiamen University Email: xmufkn@xmu.edu.cn Support Vector Machines Here we approach the two-class classification problem in a direct way: We try and find
More informationDiscriminative Models
No.5 Discriminative Models Hui Jiang Department of Electrical Engineering and Computer Science Lassonde School of Engineering York University, Toronto, Canada Outline Generative vs. Discriminative models
More informationMachine Learning for OR & FE
Machine Learning for OR & FE Regression II: Regularization and Shrinkage Methods Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com
More informationOslo Class 6 Sparsity based regularization
RegML2017@SIMULA Oslo Class 6 Sparsity based regularization Lorenzo Rosasco UNIGE-MIT-IIT May 4, 2017 Learning from data Possible only under assumptions regularization min Ê(w) + λr(w) w Smoothness Sparsity
More informationCoordinate descent. Geoff Gordon & Ryan Tibshirani Optimization /
Coordinate descent Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Adding to the toolbox, with stats and ML in mind We ve seen several general and useful minimization tools First-order methods
More informationStability and the elastic net
Stability and the elastic net Patrick Breheny March 28 Patrick Breheny High-Dimensional Data Analysis (BIOS 7600) 1/32 Introduction Elastic Net Our last several lectures have concentrated on methods for
More informationBiostatistics Advanced Methods in Biostatistics IV
Biostatistics 140.754 Advanced Methods in Biostatistics IV Jeffrey Leek Assistant Professor Department of Biostatistics jleek@jhsph.edu Lecture 12 1 / 36 Tip + Paper Tip: As a statistician the results
More informationParcimonie en apprentissage statistique
Parcimonie en apprentissage statistique Guillaume Obozinski Ecole des Ponts - ParisTech Journée Parcimonie Fédération Charles Hermite, 23 Juin 2014 Parcimonie en apprentissage 1/44 Classical supervised
More informationSparse Composite Models of Galaxy Spectra (or Anything) using L1 Norm Regularization. Benjamin Weiner Steward Observatory University of Arizona
Sparse Composite Models of Galaxy Spectra (or Anything) using L1 Norm Regularization Benjamin Weiner Steward Observatory University of Arizona Inference of mass, redshift, etc. from physical models Many
More informationTutorial on Linear Regression
Tutorial on Linear Regression HY-539: Advanced Topics on Wireless Networks & Mobile Systems Prof. Maria Papadopouli Evripidis Tzamousis tzamusis@csd.uoc.gr Agenda 1. Simple linear regression 2. Multiple
More informationGeneralized Elastic Net Regression
Abstract Generalized Elastic Net Regression Geoffroy MOURET Jean-Jules BRAULT Vahid PARTOVINIA This work presents a variation of the elastic net penalization method. We propose applying a combined l 1
More informationA Modern Look at Classical Multivariate Techniques
A Modern Look at Classical Multivariate Techniques Yoonkyung Lee Department of Statistics The Ohio State University March 16-20, 2015 The 13th School of Probability and Statistics CIMAT, Guanajuato, Mexico
More information1 Sparsity and l 1 relaxation
6.883 Learning with Combinatorial Structure Note for Lecture 2 Author: Chiyuan Zhang Sparsity and l relaxation Last time we talked about sparsity and characterized when an l relaxation could recover the
More informationLecture 3: Introduction to Complexity Regularization
ECE90 Spring 2007 Statistical Learning Theory Instructor: R. Nowak Lecture 3: Introduction to Complexity Regularization We ended the previous lecture with a brief discussion of overfitting. Recall that,
More informationMachine Learning. Regularization and Feature Selection. Fabio Vandin November 13, 2017
Machine Learning Regularization and Feature Selection Fabio Vandin November 13, 2017 1 Learning Model A: learning algorithm for a machine learning task S: m i.i.d. pairs z i = (x i, y i ), i = 1,..., m,
More informationCOMS 4721: Machine Learning for Data Science Lecture 10, 2/21/2017
COMS 4721: Machine Learning for Data Science Lecture 10, 2/21/2017 Prof. John Paisley Department of Electrical Engineering & Data Science Institute Columbia University FEATURE EXPANSIONS FEATURE EXPANSIONS
More informationLecture 14: Shrinkage
Lecture 14: Shrinkage Reading: Section 6.2 STATS 202: Data mining and analysis October 27, 2017 1 / 19 Shrinkage methods The idea is to perform a linear regression, while regularizing or shrinking the
More informationISyE 691 Data mining and analytics
ISyE 691 Data mining and analytics Regression Instructor: Prof. Kaibo Liu Department of Industrial and Systems Engineering UW-Madison Email: kliu8@wisc.edu Office: Room 3017 (Mechanical Engineering Building)
More informationFunction Approximation
1 Function Approximation This is page i Printer: Opaque this 1.1 Introduction In this chapter we discuss approximating functional forms. Both in econometric and in numerical problems, the need for an approximating
More informationNeural Networks. David Rosenberg. July 26, New York University. David Rosenberg (New York University) DS-GA 1003 July 26, / 35
Neural Networks David Rosenberg New York University July 26, 2017 David Rosenberg (New York University) DS-GA 1003 July 26, 2017 1 / 35 Neural Networks Overview Objectives What are neural networks? How
More informationECE G: Special Topics in Signal Processing: Sparsity, Structure, and Inference
ECE 18-898G: Special Topics in Signal Processing: Sparsity, Structure, and Inference Sparse Recovery using L1 minimization - algorithms Yuejie Chi Department of Electrical and Computer Engineering Spring
More informationA robust hybrid of lasso and ridge regression
A robust hybrid of lasso and ridge regression Art B. Owen Stanford University October 2006 Abstract Ridge regression and the lasso are regularized versions of least squares regression using L 2 and L 1
More informationMSA220/MVE440 Statistical Learning for Big Data
MSA220/MVE440 Statistical Learning for Big Data Lecture 7/8 - High-dimensional modeling part 1 Rebecka Jörnsten Mathematical Sciences University of Gothenburg and Chalmers University of Technology Classification
More informationRidge Regression 1. to which some random noise is added. So that the training labels can be represented as:
CS 1: Machine Learning Spring 15 College of Computer and Information Science Northeastern University Lecture 3 February, 3 Instructor: Bilal Ahmed Scribe: Bilal Ahmed & Virgil Pavlu 1 Introduction Ridge
More informationIntroduction to conic sections. Author: Eduard Ortega
Introduction to conic sections Author: Eduard Ortega 1 Introduction A conic is a two-dimensional figure created by the intersection of a plane and a right circular cone. All conics can be written in terms
More informationLecture 24 May 30, 2018
Stats 3C: Theory of Statistics Spring 28 Lecture 24 May 3, 28 Prof. Emmanuel Candes Scribe: Martin J. Zhang, Jun Yan, Can Wang, and E. Candes Outline Agenda: High-dimensional Statistical Estimation. Lasso
More informationIEOR165 Discussion Week 5
IEOR165 Discussion Week 5 Sheng Liu University of California, Berkeley Feb 19, 2016 Outline 1 1st Homework 2 Revisit Maximum A Posterior 3 Regularization IEOR165 Discussion Sheng Liu 2 About 1st Homework
More information12. Structural Risk Minimization. ECE 830 & CS 761, Spring 2016
12. Structural Risk Minimization ECE 830 & CS 761, Spring 2016 1 / 23 General setup for statistical learning theory We observe training examples {x i, y i } n i=1 x i = features X y i = labels / responses
More informationMLCC 2018 Variable Selection and Sparsity. Lorenzo Rosasco UNIGE-MIT-IIT
MLCC 2018 Variable Selection and Sparsity Lorenzo Rosasco UNIGE-MIT-IIT Outline Variable Selection Subset Selection Greedy Methods: (Orthogonal) Matching Pursuit Convex Relaxation: LASSO & Elastic Net
More informationDirect Learning: Linear Regression. Donglin Zeng, Department of Biostatistics, University of North Carolina
Direct Learning: Linear Regression Parametric learning We consider the core function in the prediction rule to be a parametric function. The most commonly used function is a linear function: squared loss:
More informationHigh-dimensional regression
High-dimensional regression Advanced Methods for Data Analysis 36-402/36-608) Spring 2014 1 Back to linear regression 1.1 Shortcomings Suppose that we are given outcome measurements y 1,... y n R, and
More information6. Regularized linear regression
Foundations of Machine Learning École Centrale Paris Fall 2015 6. Regularized linear regression Chloé-Agathe Azencot Centre for Computational Biology, Mines ParisTech chloe agathe.azencott@mines paristech.fr
More informationLecture 3: More on regularization. Bayesian vs maximum likelihood learning
Lecture 3: More on regularization. Bayesian vs maximum likelihood learning L2 and L1 regularization for linear estimators A Bayesian interpretation of regularization Bayesian vs maximum likelihood fitting
More informationStatistical Computing (36-350)
Statistical Computing (36-350) Lecture 19: Optimization III: Constrained and Stochastic Optimization Cosma Shalizi 30 October 2013 Agenda Constraints and Penalties Constraints and penalties Stochastic
More informationMath 211, Fall 2014, Carleton College
A. Let v (, 2, ) (1,, ) 1, 2, and w (,, 3) (1,, ) 1,, 3. Then n v w 6, 3, 2 is perpendicular to the plane, with length 7. Thus n/ n 6/7, 3/7, 2/7 is a unit vector perpendicular to the plane. [The negation
More informationCS 195-5: Machine Learning Problem Set 1
CS 95-5: Machine Learning Problem Set Douglas Lanman dlanman@brown.edu 7 September Regression Problem Show that the prediction errors y f(x; ŵ) are necessarily uncorrelated with any linear function of
More informationOrdinary Least Squares Linear Regression
Ordinary Least Squares Linear Regression Ryan P. Adams COS 324 Elements of Machine Learning Princeton University Linear regression is one of the simplest and most fundamental modeling ideas in statistics
More information