Optimization for well-behaved problems
|
|
|
- Rodger Patrick
- 6 years ago
- Views:
Transcription
1 Optimization for well-behaved problems For statistical learning problems, well-behaved means: signal to noise ratio is decently high correlations between predictor variables are under control number of predictors p can be larger than number of observations n, but not absurdly so For well-behaved learning problems, people have observed that gradient or generalized gradient descent can converge extremely quickly (much more so than predicted by O(1/k) rate) Largely unexplained by theory, topic of current research. E.g., very recent work 4 shows that for some well-behaved problems, w.h.p.: x (k) x 2 c k x (0) x 2 + o(x x true 2 ) 4 Agarwal et al. (2012), Fast global convergence of gradient methods for high-dimensional statistical recovery
2 Administrivia HW2 out as of this past Tuesday due 10/9 Scribing Scribes 1 6 ready soon; handling errata missing days: 11/6, 12/4, and 12/6 Projects: you should expect to be contacted by TA mentor in next weeks project milestone: 10/30 2
3 Matrix differential calculus Optimization Geoff Gordon Ryan Tibshirani
4 Matrix calculus pain Take derivatives of fns involving matrices: write as huge multiple summations w/ lots of terms take derivative as usual, introducing more terms case statements for i = j vs. i j try to recognize that output is equivalent to some human-readable form hope for no indexing errors Is there a better way? 4
5 Differentials Assume f sufficiently nice Taylor: f(y) = f(x) + f (x) (y x) + r(y x) with r(y x) / y x 0 as y x Notation: 5
6 Definition Write dx = y x df = f(y) f(x) Suppose df = with and Then: 6
7 Matrix differentials For matrix X or matrix-valued function F(X): dx = df = where and Examples: 7
8 Working with differentials Linearity: d(f(x) + g(x)) = d(k f(x)) = Common linear operators: reshape(a, [m n k ]) vec(a) = A(:) = reshape(a, [], 1) tr(a) = A T 8
9 Reshape >> reshape(1:24, [2 3 4]) ans(:,:,1) = ans(:,:,2) = ans(:,:,3) = ans(:,:,4) =
10 Working with differentials Chain rule: L(x) = f(g(x)) want: have: 10
11 Working with differentials Product rule: L(x) = c(f(x), g(x)) where c is bilinear = linear in each argument (with other argument fixed) e.g., L(x) = f(x)g(x): f, g scalars, vectors, or matrices 11
12 Lots of products Cross product: d(a b) = Hadamard product A B = A.* B (A B)ij = d(a B) = Kronecker product d(a B) = Frobenius product A:B = Khatri-Rao product: d(a*b) = 12
13 Kronecker product >> A = reshape(1:6, 2, 3) A = >> B = 2*ones(2) B = >> kron(a, B) ans = >> kron(b, A) ans =
14 Hadamard product a, b vectors a b = diag(a) diag(b) = tr(diag(a) diag(b)) = tr(diag(b)) = 14
15 Some examples L = (Y XW) T (Y XW): differential wrt W dl = 15
16 Some examples L = dl = L = dl = 16
17 Trace tr(a) = Aii d tr(f(x)) = tr(x) = tr(x T ) = Frobenius product: A:B = 17
18 Trace rotation tr(ab) = tr(abc) = size(a): size(b): size(c): 18
19 More Identities: for a matrix X, d(x -1 ) = d(det X) = d(ln det X ) = 19
20 Example: linear regression Training examples: Input feature vectors: Target vectors: Weight matrix: minw L = as matrix: 20
21 Linear regression L = Y WX 2 = F dl = 21
22 Identification theorems Sometimes useful to go back and forth between differential & ordinary notations not always possible: e.g., d(x T X) = Six common cases (ID thms): ID for df(x) scalar x vector x matrix X scalar f df = a dx df = a T dx df = tr(a T dx) vector f matrix F df = a dx df = A dx df = A dx 22
23 10 Ex: Infomax ICA Training examples xi R d, i = 1:n Transformation yi = g(wxi) W R d d g(z) = Want: Wxi xi yi 23
24 10 Ex: Infomax ICA yi = g(wxi) dyi = xi 5 Method: maxw i ln(p(yi)) where P(yi) = 0 5 Wxi yi 24
25 Gradient L = ln det Ji yi = g(wxi) dyi = Ji dxi i 25
26 Gradient Ji = diag(ui) W dji = diag(ui) dw + diag(vi) diag(dw xi) W dl = 26
27 Natural gradient Matrix W, gradient dl = G:dW = tr(g T dw) step S = arg maxs L(S) = tr(g T S) SW -1 2 /2 scalar case: L = gs s 2 / 2w 2 L = dl = F 27
28 ICA natural gradient [W -T + C] W T W = yi Wxi start with W0 = I 28
![C] W T W = yi](/docs-images/78/78728559/images/29-2.jpg "Wxi start with")
29 ICA natural gradient [W -T + C] W T W = yi Wxi start with W0 = I 28
30 ICA on natural image patches 29
31 ICA on natural image patches 30
32 More info Minka s cheat sheet: papers/matrix/ Magnus & Neudecker. Matrix Differential Calculus. Wiley, nd ed. Applications-Statistics-Econometrics/dp/ X Bell & Sejnowski. An information-maximization approach to blind separation and blind deconvolution. Neural Computation, v7,
Matrix differential calculus Optimization Geoff Gordon Ryan Tibshirani
Matrix differential calculus 10-725 Optimization Geoff Gordon Ryan Tibshirani Review Matrix differentials: sol n to matrix calculus pain compact way of writing Taylor expansions, or definition: df = a(x;
Lecture 11: October 2
10-725: Optimization Fall 2012 Lecture 11: October 2 Lecturer: Geoff Gordon/Ryan Tibshirani Scribes: Tongbo Huang, Shoou-I Yu Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes
Linear Regression 1 / 25. Karl Stratos. June 18, 2018
Linear Regression Karl Stratos June 18, 2018 1 / 25 The Regression Problem Problem. Find a desired input-output mapping f : X R where the output is a real value. x = = y = 0.1 How much should I turn my
Computational Graphs, and Backpropagation. Michael Collins, Columbia University
Computational Graphs, and Backpropagation Michael Collins, Columbia University A Key Problem: Calculating Derivatives where and p(y x; θ, v) = exp (v(y) φ(x; θ) + γ y ) y Y exp (v(y ) φ(x; θ) + γ y ) φ(x;
Introduction to gradient descent
6-1: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction to gradient descent Derivation and intuitions Hessian 6-2: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction Our
. D Matrix Calculus D 1
D Matrix Calculus D 1 Appendix D: MATRIX CALCULUS D 2 In this Appendix we collect some useful formulas of matrix calculus that often appear in finite element derivations D1 THE DERIVATIVES OF VECTOR FUNCTIONS
CSC321 Lecture 2: Linear Regression
CSC32 Lecture 2: Linear Regression Roger Grosse Roger Grosse CSC32 Lecture 2: Linear Regression / 26 Overview First learning algorithm of the course: linear regression Task: predict scalar-valued targets,
Matrix Algebra, Class Notes (part 2) by Hrishikesh D. Vinod Copyright 1998 by Prof. H. D. Vinod, Fordham University, New York. All rights reserved.
Matrix Algebra, Class Notes (part 2) by Hrishikesh D. Vinod Copyright 1998 by Prof. H. D. Vinod, Fordham University, New York. All rights reserved. 1 Converting Matrices Into (Long) Vectors Convention:
Neural Network Training
Neural Network Training Sargur Srihari Topics in Network Training 0. Neural network parameters Probabilistic problem formulation Specifying the activation and error functions for Regression Binary classification
Least Squares Regression
E0 70 Machine Learning Lecture 4 Jan 7, 03) Least Squares Regression Lecturer: Shivani Agarwal Disclaimer: These notes are a brief summary of the topics covered in the lecture. They are not a substitute
Gradient Descent. Stephan Robert
Gradient Descent Stephan Robert Model Other inputs # of bathrooms Lot size Year built Location Lake view Price (CHF) 6 5 4 3 2 1 2 4 6 8 1 Size (m 2 ) Model How it works Linear regression with one variable
More on Neural Networks
More on Neural Networks Yujia Yan Fall 2018 Outline Linear Regression y = Wx + b (1) Linear Regression y = Wx + b (1) Polynomial Regression y = Wφ(x) + b (2) where φ(x) gives the polynomial basis, e.g.,
Announcements. Topics: Homework:
Announcements Topics: - sections 7.3 (the definite integral +area), 7.4 (FTC), 7.5 (additional techniques of integration) * Read these sections and study solved examples in your textbook! Homework: - review
CS598 Machine Learning in Computational Biology (Lecture 5: Matrix - part 2) Professor Jian Peng Teaching Assistant: Rongda Zhu
CS598 Machine Learning in Computational Biology (Lecture 5: Matrix - part 2) Professor Jian Peng Teaching Assistant: Rongda Zhu Feature engineering is hard 1. Extract informative features from domain knowledge
CIS519: Applied Machine Learning Fall Homework 5. Due: December 10 th, 2018, 11:59 PM
CIS59: Applied Machine Learning Fall 208 Homework 5 Handed Out: December 5 th, 208 Due: December 0 th, 208, :59 PM Feel free to talk to other members of the class in doing the homework. I am more concerned
8-1: Backpropagation Prof. J.C. Kao, UCLA. Backpropagation. Chain rule for the derivatives Backpropagation graphs Examples
8-1: Backpropagation Prof. J.C. Kao, UCLA Backpropagation Chain rule for the derivatives Backpropagation graphs Examples 8-2: Backpropagation Prof. J.C. Kao, UCLA Motivation for backpropagation To do gradient
Unit #10 : Graphs of Antiderivatives; Substitution Integrals
Unit #10 : Graphs of Antiderivatives; Substitution Integrals Goals: Relationship between the graph of f(x) and its anti-derivative F(x) The guess-and-check method for anti-differentiation. The substitution
CSC321 Lecture 6: Backpropagation
CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1 / 21 Overview We ve seen that multilayer neural networks are powerful. But how can we actually learn them?
Gradient Descent. Model
Gradient Descent Stephan Robert Model Other inputs # of bathrooms Lot size Year built Location Lake view Price (HF) 6 5 4 3 2 2 4 6 8 Size (m 2 ) 2 Model How it works Linear regression with one variable
BACKPROPAGATION. Neural network training optimization problem. Deriving backpropagation
BACKPROPAGATION Neural network training optimization problem min J(w) w The application of gradient descent to this problem is called backpropagation. Backpropagation is gradient descent applied to J(w)
Massoud BABAIE-ZADEH. Blind Source Separation (BSS) and Independent Componen Analysis (ICA) p.1/39
Blind Source Separation (BSS) and Independent Componen Analysis (ICA) Massoud BABAIE-ZADEH Blind Source Separation (BSS) and Independent Componen Analysis (ICA) p.1/39 Outline Part I Part II Introduction
Neural Networks. Intro to AI Bert Huang Virginia Tech
Neural Networks Intro to AI Bert Huang Virginia Tech Outline Biological inspiration for artificial neural networks Linear vs. nonlinear functions Learning with neural networks: back propagation https://en.wikipedia.org/wiki/neuron#/media/file:chemical_synapse_schema_cropped.jpg
Linear Regression. CSL603 - Fall 2017 Narayanan C Krishnan
Linear Regression CSL603 - Fall 2017 Narayanan C Krishnan ckn@iitrpr.ac.in Outline Univariate regression Multivariate regression Probabilistic view of regression Loss functions Bias-Variance analysis Regularization
Reminders. Thought questions should be submitted on eclass. Please list the section related to the thought question
Linear regression Reminders Thought questions should be submitted on eclass Please list the section related to the thought question If it is a more general, open-ended question not exactly related to a
Linear Regression. CSL465/603 - Fall 2016 Narayanan C Krishnan
Linear Regression CSL465/603 - Fall 2016 Narayanan C Krishnan ckn@iitrpr.ac.in Outline Univariate regression Multivariate regression Probabilistic view of regression Loss functions Bias-Variance analysis
Lecture 20: November 1st
10-725: Optimization Fall 2012 Lecture 20: November 1st Lecturer: Geoff Gordon Scribes: Xiaolong Shen, Alex Beutel Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not
A Derivation of the 3-entity-type model
A Derivation of the 3-entity-type model The 3-entity-type model consists of two relationship matrices, a m n matrix X and a n r matrix Y In the spirit of generalized linear models we define the low rank
Graphs of Antiderivatives, Substitution Integrals
Unit #10 : Graphs of Antiderivatives, Substitution Integrals Goals: Relationship between the graph of f(x) and its anti-derivative F (x) The guess-and-check method for anti-differentiation. The substitution
CPSC 540: Machine Learning
CPSC 540: Machine Learning Matrix Notation Mark Schmidt University of British Columbia Winter 2017 Admin Auditting/registration forms: Submit them at end of class, pick them up end of next class. I need
The Relation between the Integral and the Derivative Graphs. Unit #10 : Graphs of Antiderivatives, Substitution Integrals
Graphs of Antiderivatives - Unit #0 : Graphs of Antiderivatives, Substitution Integrals Goals: Relationship between the graph of f(x) and its anti-derivative F (x) The guess-and-check method for anti-differentiation.
Lecture 2: Linear regression
Lecture 2: Linear regression Roger Grosse 1 Introduction Let s ump right in and look at our first machine learning algorithm, linear regression. In regression, we are interested in predicting a scalar-valued
Lecture 9: September 28
0-725/36-725: Convex Optimization Fall 206 Lecturer: Ryan Tibshirani Lecture 9: September 28 Scribes: Yiming Wu, Ye Yuan, Zhihao Li Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These
Neural Networks, Convexity, Kernels and Curses
Neural Networks, Convexity, Kernels and Curses Yoshua Bengio Work done with Nicolas Le Roux, Olivier Delalleau and Hugo Larochelle August 26th 2005 Perspective Curse of Dimensionality Most common non-parametric
Lecture 4 Backpropagation
Lecture 4 Backpropagation CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 5, 2017 Things we will look at today More Backpropagation Still more backpropagation Quiz
Math 651 Introduction to Numerical Analysis I Fall SOLUTIONS: Homework Set 1
ath 651 Introduction to Numerical Analysis I Fall 2010 SOLUTIONS: Homework Set 1 1. Consider the polynomial f(x) = x 2 x 2. (a) Find P 1 (x), P 2 (x) and P 3 (x) for f(x) about x 0 = 0. What is the relation
8.1 Concentration inequality for Gaussian random matrix (cont d)
MGMT 69: Topics in High-dimensional Data Analysis Falll 26 Lecture 8: Spectral clustering and Laplacian matrices Lecturer: Jiaming Xu Scribe: Hyun-Ju Oh and Taotao He, October 4, 26 Outline Concentration
Optimization and (Under/over)fitting
Optimization and (Under/over)fitting EECS 442 Prof. David Fouhey Winter 2019, University of Michigan http://web.eecs.umich.edu/~fouhey/teaching/eecs442_w19/ Administrivia We re grading HW2 and will try
Stochastic Calculus. Kevin Sinclair. August 2, 2016
Stochastic Calculus Kevin Sinclair August, 16 1 Background Suppose we have a Brownian motion W. This is a process, and the value of W at a particular time T (which we write W T ) is a normally distributed
Announcements. Topics: Homework:
Announcements Topics: - sections 7.4 (FTC), 7.5 (additional techniques of integration), 7.6 (applications of integration) * Read these sections and study solved examples in your textbook! Homework: - review
Matrix Derivatives and Descent Optimization Methods
Matrix Derivatives and Descent Optimization Methods 1 Qiang Ning Department of Electrical and Computer Engineering Beckman Institute for Advanced Science and Techonology University of Illinois at Urbana-Champaign
Contents. 1 Introduction. 1.1 History of Optimization ALG-ML SEMINAR LISSA: LINEAR TIME SECOND-ORDER STOCHASTIC ALGORITHM FEBRUARY 23, 2016
ALG-ML SEMINAR LISSA: LINEAR TIME SECOND-ORDER STOCHASTIC ALGORITHM FEBRUARY 23, 2016 LECTURERS: NAMAN AGARWAL AND BRIAN BULLINS SCRIBE: KIRAN VODRAHALLI Contents 1 Introduction 1 1.1 History of Optimization.....................................
Vector, Matrix, and Tensor Derivatives
Vector, Matrix, and Tensor Derivatives Erik Learned-Miller The purpose of this document is to help you learn to take derivatives of vectors, matrices, and higher order tensors (arrays with three dimensions
M2PM1 Analysis II (2008) Dr M Ruzhansky List of definitions, statements and examples Preliminary version
M2PM1 Analysis II (2008) Dr M Ruzhansky List of definitions, statements and examples Preliminary version Chapter 0: Some revision of M1P1: Limits and continuity This chapter is mostly the revision of Chapter
Neural Networks: Backpropagation
Neural Networks: Backpropagation Seung-Hoon Na 1 1 Department of Computer Science Chonbuk National University 2018.10.25 eung-hoon Na (Chonbuk National University) Neural Networks: Backpropagation 2018.10.25
SEBASTIAN RASCHKA. Introduction to Artificial Neural Networks and Deep Learning. with Applications in Python
SEBASTIAN RASCHKA Introduction to Artificial Neural Networks and Deep Learning with Applications in Python Introduction to Artificial Neural Networks with Applications in Python Sebastian Raschka Last
Lecture 13: Simple Linear Regression in Matrix Format. 1 Expectations and Variances with Vectors and Matrices
Lecture 3: Simple Linear Regression in Matrix Format To move beyond simple regression we need to use matrix algebra We ll start by re-expressing simple linear regression in matrix form Linear algebra is
Computing Neural Network Gradients
Computing Neural Network Gradients Kevin Clark 1 Introduction The purpose of these notes is to demonstrate how to quickly compute neural network gradients in a completely vectorized way. It is complementary
1 Introduction Independent component analysis (ICA) [10] is a statistical technique whose main applications are blind source separation, blind deconvo
![1 Introduction Independent component analysis (ICA) [10] is a statistical technique whose main applications are blind source separation, blind deconvo](/thumbs/72/66692794.jpg "1 Introduction Independent component analysis (ICA) [10] is a statistical technique whose main applications are blind source separation, blind deconvo")
The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis Aapo Hyvarinen Helsinki University of Technology Laboratory of Computer and Information Science P.O.Box 5400,
ON SOME EXTENSIONS OF THE NATURAL GRADIENT ALGORITHM. Brain Science Institute, RIKEN, Wako-shi, Saitama , Japan
ON SOME EXTENSIONS OF THE NATURAL GRADIENT ALGORITHM Pando Georgiev a, Andrzej Cichocki b and Shun-ichi Amari c Brain Science Institute, RIKEN, Wako-shi, Saitama 351-01, Japan a On leave from the Sofia
Artificial Neural Networks
Artificial Neural Networks Threshold units Gradient descent Multilayer networks Backpropagation Hidden layer representations Example: Face Recognition Advanced topics 1 Connectionist Models Consider humans:
Optimization and Gradient Descent
Optimization and Gradient Descent INFO-4604, Applied Machine Learning University of Colorado Boulder September 12, 2017 Prof. Michael Paul Prediction Functions Remember: a prediction function is the function
Lecture 4: Types of errors. Bayesian regression models. Logistic regression
Lecture 4: Types of errors. Bayesian regression models. Logistic regression A Bayesian interpretation of regularization Bayesian vs maximum likelihood fitting more generally COMP-652 and ECSE-68, Lecture
Neural Networks in Structured Prediction. November 17, 2015
Neural Networks in Structured Prediction November 17, 2015 HWs and Paper Last homework is going to be posted soon Neural net NER tagging model This is a new structured model Paper - Thursday after Thanksgiving
OR MSc Maths Revision Course
OR MSc Maths Revision Course Tom Byrne School of Mathematics University of Edinburgh t.m.byrne@sms.ed.ac.uk 15 September 2017 General Information Today JCMB Lecture Theatre A, 09:30-12:30 Mathematics revision
Regression with Numerical Optimization. Logistic
CSG220 Machine Learning Fall 2008 Regression with Numerical Optimization. Logistic regression Regression with Numerical Optimization. Logistic regression based on a document by Andrew Ng October 3, 204
CPSC 540: Machine Learning
CPSC 540: Machine Learning Multivariate Gaussians Mark Schmidt University of British Columbia Winter 2019 Last Time: Multivariate Gaussian http://personal.kenyon.edu/hartlaub/mellonproject/bivariate2.html
10-725/ Optimization Midterm Exam
10-725/36-725 Optimization Midterm Exam November 6, 2012 NAME: ANDREW ID: Instructions: This exam is 1hr 20mins long Except for a single two-sided sheet of notes, no other material or discussion is permitted
Comparative Statics. Autumn 2018
Comparative Statics Autumn 2018 What is comparative statics? Contents 1 What is comparative statics? 2 One variable functions Multiple variable functions Vector valued functions Differential and total
Linear and non-linear programming
Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
CSC2515 Winter 2015 Introduction to Machine Learning. Lecture 2: Linear regression
CSC2515 Winter 2015 Introduction to Machine Learning Lecture 2: Linear regression All lecture slides will be available as.pdf on the course website: http://www.cs.toronto.edu/~urtasun/courses/csc2515/csc2515_winter15.html
APPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
Stochastic Gradient Descent: The Workhorse of Machine Learning. CS6787 Lecture 1 Fall 2017
Stochastic Gradient Descent: The Workhorse of Machine Learning CS6787 Lecture 1 Fall 2017 Fundamentals of Machine Learning? Machine Learning in Practice this course What s missing in the basic stuff? Efficiency!
Neural Networks Teaser
1/11 Neural Networks Teaser February 27, 2017 Deep Learning in the News 2/11 Go falls to computers. Learning 3/11 How to teach a robot to be able to recognize images as either a cat or a non-cat? This
CSC 411 Lecture 10: Neural Networks
CSC 411 Lecture 10: Neural Networks Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 10-Neural Networks 1 / 35 Inspiration: The Brain Our brain has 10 11
Logistic Regression Trained with Different Loss Functions. Discussion
Logistic Regression Trained with Different Loss Functions Discussion CS640 Notations We restrict our discussions to the binary case. g(z) = g (z) = g(z) z h w (x) = g(wx) = + e z = g(z)( g(z)) + e wx =
On the INFOMAX algorithm for blind signal separation
University of Wollongong Research Online Faculty of Informatics - Papers (Archive) Faculty of Engineering and Information Sciences 2000 On the INFOMAX algorithm for blind signal separation Jiangtao Xi
Least Squares Regression
CIS 50: Machine Learning Spring 08: Lecture 4 Least Squares Regression Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may not cover all the
g(t) = f(x 1 (t),..., x n (t)).
Reading: [Simon] p. 313-333, 833-836. 0.1 The Chain Rule Partial derivatives describe how a function changes in directions parallel to the coordinate axes. Now we shall demonstrate how the partial derivatives
Independent Components Analysis
CS229 Lecture notes Andrew Ng Part XII Independent Components Analysis Our next topic is Independent Components Analysis (ICA). Similar to PCA, this will find a new basis in which to represent our data.
Natural Gradient Learning for Over- and Under-Complete Bases in ICA
NOTE Communicated by Jean-François Cardoso Natural Gradient Learning for Over- and Under-Complete Bases in ICA Shun-ichi Amari RIKEN Brain Science Institute, Wako-shi, Hirosawa, Saitama 351-01, Japan Independent
Error Functions & Linear Regression (1)
Error Functions & Linear Regression (1) John Kelleher & Brian Mac Namee Machine Learning @ DIT Overview 1 Introduction Overview 2 Univariate Linear Regression Linear Regression Analytical Solution Gradient
Gradient Descent. Ryan Tibshirani Convex Optimization /36-725
Gradient Descent Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: canonical convex programs Linear program (LP): takes the form min x subject to c T x Gx h Ax = b Quadratic program (QP): like
STAT 135 Lab 13 (Review) Linear Regression, Multivariate Random Variables, Prediction, Logistic Regression and the δ-method.
STAT 135 Lab 13 (Review) Linear Regression, Multivariate Random Variables, Prediction, Logistic Regression and the δ-method. Rebecca Barter May 5, 2015 Linear Regression Review Linear Regression Review
Math Advanced Calculus II
Math 452 - Advanced Calculus II Manifolds and Lagrange Multipliers In this section, we will investigate the structure of critical points of differentiable functions. In practice, one often is trying to
11 a 12 a 21 a 11 a 22 a 12 a 21. (C.11) A = The determinant of a product of two matrices is given by AB = A B 1 1 = (C.13) and similarly.
C PROPERTIES OF MATRICES 697 to whether the permutation i 1 i 2 i N is even or odd, respectively Note that I =1 Thus, for a 2 2 matrix, the determinant takes the form A = a 11 a 12 = a a 21 a 11 a 22 a
g(2, 1) = cos(2π) + 1 = = 9
1. Let gx, y 2x 2 cos2πy 2 + y 2. You can use the fact that Dg2, 1 [8 2]. a Find an equation for the tangent plane to the graph z gx, y at the point 2, 1. There are two key parts to this problem. The first,
Sec 4.1 Limits, Informally. When we calculated f (x), we first started with the difference quotient. f(x + h) f(x) h
1 Sec 4.1 Limits, Informally When we calculated f (x), we first started with the difference quotient f(x + h) f(x) h and made h small. In other words, f (x) is the number f(x+h) f(x) approaches as h gets
Linear Regression, Neural Networks, etc.
Linear Regression, Neural Networks, etc. Gradient Descent Many machine learning problems can be cast as optimization problems Define a function that corresponds to learning error. (More on this later)
Hands-on Matrix Algebra Using R
Preface vii 1. R Preliminaries 1 1.1 Matrix Defined, Deeper Understanding Using Software.. 1 1.2 Introduction, Why R?.................... 2 1.3 Obtaining R.......................... 4 1.4 Reference Manuals
A special rule, the chain rule, exists for differentiating a function of another function. This unit illustrates this rule.
The Chain Rule A special rule, the chain rule, exists for differentiating a function of another function. This unit illustrates this rule. In order to master the techniques explained here it is vital that
MATH 680 Fall November 27, Homework 3
MATH 680 Fall 208 November 27, 208 Homework 3 This homework is due on December 9 at :59pm. Provide both pdf, R files. Make an individual R file with proper comments for each sub-problem. Subgradients and
Mathematics for Economics ECON MA/MSSc in Economics-2017/2018. Dr. W. M. Semasinghe Senior Lecturer Department of Economics
Mathematics for Economics ECON 53035 MA/MSSc in Economics-2017/2018 Dr. W. M. Semasinghe Senior Lecturer Department of Economics MATHEMATICS AND STATISTICS LERNING OUTCOMES: By the end of this course unit
COMS 4771 Introduction to Machine Learning. James McInerney Adapted from slides by Nakul Verma
COMS 4771 Introduction to Machine Learning James McInerney Adapted from slides by Nakul Verma Announcements HW1: Please submit as a group Watch out for zero variance features (Q5) HW2 will be released
Machine Learning. Lecture 2: Linear regression. Feng Li. https://funglee.github.io
Machine Learning Lecture 2: Linear regression Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2017 Supervised Learning Regression: Predict
Second-order approximation of dynamic models without the use of tensors
Second-order approximation of dynamic models without the use of tensors Paul Klein a, a University of Western Ontario First draft: May 17, 2005 This version: January 24, 2006 Abstract Several approaches
Lecture 4: Least Squares (LS) Estimation
ME 233, UC Berkeley, Spring 2014 Xu Chen Lecture 4: Least Squares (LS) Estimation Background and general solution Solution in the Gaussian case Properties Example Big picture general least squares estimation:
Lagrange Multipliers
Optimization with Constraints As long as algebra and geometry have been separated, their progress have been slow and their uses limited; but when these two sciences have been united, they have lent each
Vector Derivatives and the Gradient
ECE 275AB Lecture 10 Fall 2008 V1.1 c K. Kreutz-Delgado, UC San Diego p. 1/1 Lecture 10 ECE 275A Vector Derivatives and the Gradient ECE 275AB Lecture 10 Fall 2008 V1.1 c K. Kreutz-Delgado, UC San Diego
Unit #10 : Graphs of Antiderivatives, Substitution Integrals
Unit #10 : Graphs of Antiderivatives, Substitution Integrals Goals: Relationship between the graph of f(x) and its anti-derivative F(x) The guess-and-check method for anti-differentiation. The substitution
Computational Graphs, and Backpropagation
Chapter 1 Computational Graphs, and Backpropagation (Course notes for NLP by Michael Collins, Columbia University) 1.1 Introduction We now describe the backpropagation algorithm for calculation of derivatives
Meaning of the Hessian of a function in a critical point
Meaning of the Hessian of a function in a critical point Mircea Petrache February 1, 2012 We consider a function f : R n R and assume for it to be differentiable with continuity at least two times (that
Neural Networks (and Gradient Ascent Again)
Neural Networks (and Gradient Ascent Again) Frank Wood April 27, 2010 Generalized Regression Until now we have focused on linear regression techniques. We generalized linear regression to include nonlinear
Parameter estimation Conditional risk
Parameter estimation Conditional risk Formalizing the problem Specify random variables we care about e.g., Commute Time e.g., Heights of buildings in a city We might then pick a particular distribution
Lecture 22: Integration by parts and u-substitution
Lecture 22: Integration by parts and u-substitution Victoria LEBED, lebed@maths.tcd.ie MA1S11A: Calculus with Applications for Scientists December 1, 2017 1 Integration vs differentiation From our first
Gradient Descent. Sargur Srihari
Gradient Descent Sargur srihari@cedar.buffalo.edu 1 Topics Simple Gradient Descent/Ascent Difficulties with Simple Gradient Descent Line Search Brent s Method Conjugate Gradient Descent Weight vectors
Least Mean Squares Regression. Machine Learning Fall 2018
Least Mean Squares Regression Machine Learning Fall 2018 1 Where are we? Least Squares Method for regression Examples The LMS objective Gradient descent Incremental/stochastic gradient descent Exercises
Lecture 5 Multivariate Linear Regression
Lecture 5 Multivariate Linear Regression Dan Sheldon September 23, 2014 Topics Multivariate linear regression Model Cost function Normal equations Gradient descent Features Book Data 10 8 Weight (lbs.)
Logistic Regression. Some slides adapted from Dan Jurfasky and Brendan O Connor
Logistic Regression Some slides adapted from Dan Jurfasky and Brendan O Connor Naïve Bayes Recap Bag of words (order independent) Features are assumed independent given class P (x 1,...,x n c) =P (x 1
Topic 2: Logistic Regression
CS 4850/6850: Introduction to Machine Learning Fall 208 Topic 2: Logistic Regression Instructor: Daniel L. Pimentel-Alarcón c Copyright 208 2. Introduction Arguably the simplest task that we can teach
(0, 0), (1, ), (2, ), (3, ), (4, ), (5, ), (6, ).
1 Interpolation: The method of constructing new data points within the range of a finite set of known data points That is if (x i, y i ), i = 1, N are known, with y i the dependent variable and x i [x