A Short Introduction to the Theory of Statistical Learning. Kee Siong Ng, PhD
|
|
|
- Andrew Webb
- 5 years ago
- Views:
Transcription
1 A Short Introduction to the Theory of Statistical Learning Kee Siong Ng, PhD
2 Outline Introduction Bayesian Probability Theory Sequence Prediction and Data Compression Bayesian Networks Copyright 2015 Kee Siong Ng. All rights reserved. 1
3 Outline Introduction 1. What is machine learning? 2. When is learning (not) possible? Bayesian Probability Theory Sequence Prediction and Data Compression Bayesian Networks Copyright 2015 Kee Siong Ng. All rights reserved. 2
4 What is Machine Learning? From Wikipedia: Machine learning is the subfield of AI that is concerned with the design and development of algorithms that allow computers to improve their performance over time based on data. A major focus of machine learning research is to automatically produce (induce) models, such as rules and patterns, from data. In summary: It is about collecting data, analysing data, interpreting data, and acting on insights so obtained to achieve goals. Copyright 2015 Kee Siong Ng. All rights reserved. 3
5 Inductive Learning from Observations Machine learning research covers a wide spectrum of activities. Focus on classification/prediction problems in this course. Informally, given a sequence of data [(x 1,y 1 ),(x 2,y 2 ),...,(x n,y n )], x i 2 X,y i 2 Y generated by an (unknown) process, learn a function f : X! Y that, given any x 2 X, predicts the corresponding y value accurately. Copyright 2015 Kee Siong Ng. All rights reserved. 4
6 Example Problems Spam filtering Data: [( Congratulations, you have won the jackpot!...,+), ( Bankwest is conducting an account audit...,+), ( Your COMP3620 assignment is late..., ),...] Is Congratulations on your COMP3620 assignment... a spam? Hand-written character recognition Data: [(,3),(,1),(,3),(,4),(,7),...] What s the label of? Copyright 2015 Kee Siong Ng. All rights reserved. 5
7 Philosophical Concerns about Induction Hume (1748): How do you know the future will be like the past? Inductive processes are inherently unjustifiable. Popper (1935): Scientific theories/conjectures are only falsifiable; they can never be confirmed for sure. Competing theories cannot be compared. Need to define learning problems carefully. Copyright 2015 Kee Siong Ng. All rights reserved. 6
8 Learning (More Formally) Given a sequence of data D =[(x 1,y 1 ),(x 2,y 2 ),...,(x n,y n )], x i 2 X,y i 2 Y, where each (x i,y i ) is drawn independently at random from an unknown probability distribution P on X Y ; a set F of functions from X! Y ; Learn an f 2 F that minimises expected error E (x,y) P I(y 6= f (x)). Â P(x, y)i(y 6= f (x)), I : indicator function (x,y) Copyright 2015 Kee Siong Ng. All rights reserved. 7
9 Algorithmic Strategy We want f 2 F that minimises E (x,y) P I(y 6= f (x)). But we don t know P! Copyright 2015 Kee Siong Ng. All rights reserved. 8
10 Algorithmic Strategy We want f 2 F that minimises E (x,y) P I(y 6= f (x)). But we don t know P! Idea: Use data D as a proxy for P. Instead of seeking argmin f 2F E (x,y) P I(y 6= f (x)), we seek f that minimises empirical error 1 n  (x,y)2d I(y 6= f (x)). (1) Intuitive justification: Law of large numbers Fix f : (1)! E (x,y) P I(y 6= f (x)) as D! Copyright 2015 Kee Siong Ng. All rights reserved. 8
11 Laws of Large Numbers A Bernoulli trial is an experiment that returns 1 with probability p and 0 with probability 1 p. If we do n such experiments, how many times k are we likely to get the result 1? Coin experiment Copyright 2015 Kee Siong Ng. All rights reserved. 9
12 Laws of Large Numbers A Bernoulli trial is an experiment that returns 1 with probability p and 0 with probability 1 p. If we do n such experiments, how many times k are we likely to get the result 1? (Bernoulli s Theorem) For any small error e, any small difference d, there is a number of trials N such that for any n > N, Pr[(p e) apple k/n apple (p + e)] > 1 d Copyright 2015 Kee Siong Ng. All rights reserved. 10
13 Laws of Large Numbers Bernoulli s theorem doesn t tell us how fast k/n converges to p. Let b(k;n, p) be the probability of getting k occurrences of 1 in n Bernoulli trials. (Abraham De Moivre) A binomial distribution b(k;n, p) is approximated by a normal distribution with mean µ = pn and standard deviation s = p (1 p)pn. Thus, e.g., the probability that k is within 2s of pn is about Pr[(pn 2 p (1 p)pn) apple k apple (pn + 2 p (1 p)pn)] = 0.95 Copyright 2015 Kee Siong Ng. All rights reserved. 11
14 Laws of Large Numbers Bernoulli s theorem doesn t tell us how fast k/n converges to p. Let b(k;n, p) be the probability of getting k occurrences of 1 in n Bernoulli trials. (Abraham De Moivre) A binomial distribution b(k;n, p) is approximated by a normal distribution with mean µ = pn and standard deviation s = p (1 p)pn. Thus, e.g., the probability that k is within 2s of pn is about Pr[(pn 2 p (1 p)pn) apple k apple (pn + 2 p (1 p)pn)] = 0.95 r r (1 p)p (1 p)p = Pr[(p 2 ) apple k/n apple (p + 2 )] = 0.95 n n Copyright 2015 Kee Siong Ng. All rights reserved. 11
15 Minimisation Angst We still have a problem if the function class F is too large. Can have many functions that all minimise empirical error. Consider underlying distribution P defined as follows: P : Density ([0,1] {0,1}) 8 < 1 if (x 0.5 ^ y = 1) or (x < 0.5 ^ y = 0) P(x,y)= : 0 otherwise We can t learn if F is the set of all functions. Functions that remember the data achieve zero empirical error but incur large expected error! Copyright 2015 Kee Siong Ng. All rights reserved. 12
16 A Sufficient Condition for Successful Learning A sufficient (but far from necessary) condition for successful learning is F < and F X. Let s look at a simple proof. Further assumptions: There is an unknown distribution Q on X. There is an underlying true function t 2 F. Distribution P on X Y is defined by 8 < Q(x) if y = t (x) P(x,y)= : 0 otherwise Copyright 2015 Kee Siong Ng. All rights reserved. 13
17 Learnability Result Define F bad = {h 2 F E (x,y) P I(h(x) 6= y) E (x,y) P I(t (x) 6= y) e} = {h 2 F E (x,y) P I(h(x) 6= y) e}. Given n randomly drawn examples D = {(x i,y i )} n i=1 from P, Pr(9h 2 F bad consistent with D) = Pr(_ F bad i=1 h i consistent with D) F bad = Â i=1 Pr(h i consistent with D) = F bad (1 e) n apple F (1 e) n. Copyright 2015 Kee Siong Ng. All rights reserved. 14
18 Learnability Result We have Pr(9h 2 F bad consistent with D) apple F (1 e) n. We want F (1 e) n apple d for some small d. Solving for n yields 1 n ln 1d e + ln F. (2) This means if we have seen n examples satisfying (2) and pick a hypothesis h 2 F that is consistent with the examples, then with probability 1 d, E (x,y) P I(h (x) 6= y) < e. Copyright 2015 Kee Siong Ng. All rights reserved. 15
19 Learnability Result If we drop the assumption that t 2 F, picking the h 2 F that minimises error on data is still a good strategy. Examples needed: 1 n ln 1d 2e 2 + ln F Given data D = {(x i,y i )} n i=1, with probability 1 d, E (x,y) P I(h (x) 6= y) < 1 D n  i=1 I(h (x i ) 6= y i )+e. Copyright 2015 Kee Siong Ng. All rights reserved. 16
20 Problems in Practice We may not be able to get the number of examples required. We may not be able to efficiently find the function that minimises empirical error. There could be multiple empirical-error minimising functions. The first and third problems can be solved using Bayesian probability theory, which we look at next. Copyright 2015 Kee Siong Ng. All rights reserved. 17
21 Session Review Learning is not always possible When learning is possible, minimising empirical error is a good strategy justifiable using laws of large numbers Questions? Copyright 2015 Kee Siong Ng. All rights reserved. 18
22 Review of Session 1 Learning is not always possible When learning is possible, minimising empirical error is a good strategy justifiable using laws of large numbers Problems in practice We may not be able to get the number of examples required. We may not be able to efficiently find the function that minimises empirical error. There could be multiple empirical-error minimising functions. Copyright 2015 Kee Siong Ng. All rights reserved. 1
Outline. Introduction. Bayesian Probability Theory Bayes rule Bayes rule applied to learning Bayesian learning and the MDL principle
Outline Introduction Bayesian Probability Theory Bayes rule Bayes rule applied to learning Bayesian learning and the MDL principle Sequence Prediction and Data Compression Bayesian Networks Copyright 2015
Learning Theory. Piyush Rai. CS5350/6350: Machine Learning. September 27, (CS5350/6350) Learning Theory September 27, / 14
Learning Theory Piyush Rai CS5350/6350: Machine Learning September 27, 2011 (CS5350/6350) Learning Theory September 27, 2011 1 / 14 Why Learning Theory? We want to have theoretical guarantees about our
Introduction to Bayesian Learning. Machine Learning Fall 2018
Introduction to Bayesian Learning Machine Learning Fall 2018 1 What we have seen so far What does it mean to learn? Mistake-driven learning Learning by counting (and bounding) number of mistakes PAC learnability
The Naïve Bayes Classifier. Machine Learning Fall 2017
The Naïve Bayes Classifier Machine Learning Fall 2017 1 Today s lecture The naïve Bayes Classifier Learning the naïve Bayes Classifier Practical concerns 2 Today s lecture The naïve Bayes Classifier Learning
Discrete Probability Distributions
Discrete Probability Distributions EGR 260 R. Van Til Industrial & Systems Engineering Dept. Copyright 2013. Robert P. Van Til. All rights reserved. 1 What s It All About? The behavior of many random processes
With Question/Answer Animations. Chapter 7
With Question/Answer Animations Chapter 7 Chapter Summary Introduction to Discrete Probability Probability Theory Bayes Theorem Section 7.1 Section Summary Finite Probability Probabilities of Complements
Stephen Scott.
1 / 35 (Adapted from Ethem Alpaydin and Tom Mitchell) sscott@cse.unl.edu In Homework 1, you are (supposedly) 1 Choosing a data set 2 Extracting a test set of size > 30 3 Building a tree on the training
Probabilistic modeling. The slides are closely adapted from Subhransu Maji s slides
Probabilistic modeling The slides are closely adapted from Subhransu Maji s slides Overview So far the models and algorithms you have learned about are relatively disconnected Probabilistic modeling framework
Lecture 10: Bayes' Theorem, Expected Value and Variance Lecturer: Lale Özkahya
BBM 205 Discrete Mathematics Hacettepe University http://web.cs.hacettepe.edu.tr/ bbm205 Lecture 10: Bayes' Theorem, Expected Value and Variance Lecturer: Lale Özkahya Resources: Kenneth Rosen, Discrete
Introduction to Algorithms / Algorithms I Lecturer: Michael Dinitz Topic: Intro to Learning Theory Date: 12/8/16
600.463 Introduction to Algorithms / Algorithms I Lecturer: Michael Dinitz Topic: Intro to Learning Theory Date: 12/8/16 25.1 Introduction Today we re going to talk about machine learning, but from an
1 The Probably Approximately Correct (PAC) Model
COS 511: Theoretical Machine Learning Lecturer: Rob Schapire Lecture #3 Scribe: Yuhui Luo February 11, 2008 1 The Probably Approximately Correct (PAC) Model A target concept class C is PAC-learnable by
Hierarchical Concept Learning
COMS 6998-4 Fall 2017 Octorber 30, 2017 Hierarchical Concept Learning Presenter: Xuefeng Hu Scribe: Qinyao He 1 Introduction It has been shown that learning arbitrary polynomial-size circuits is computationally
Bayesian Learning (II)
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning (II) Niels Landwehr Overview Probabilities, expected values, variance Basic concepts of Bayesian learning MAP
COMPUTATIONAL LEARNING THEORY
COMPUTATIONAL LEARNING THEORY XIAOXU LI Abstract. This paper starts with basic concepts of computational learning theory and moves on with examples in monomials and learning rays to the final discussion
i=1 = H t 1 (x) + α t h t (x)
AdaBoost AdaBoost, which stands for ``Adaptive Boosting", is an ensemble learning algorithm that uses the boosting paradigm []. We will discuss AdaBoost for binary classification. That is, we assume that
An Introduction to No Free Lunch Theorems
February 2, 2012 Table of Contents Induction Learning without direct observation. Generalising from data. Modelling physical phenomena. The Problem of Induction David Hume (1748) How do we know an induced
Expectation is linear. So far we saw that E(X + Y ) = E(X) + E(Y ). Let α R. Then,
Expectation is linear So far we saw that E(X + Y ) = E(X) + E(Y ). Let α R. Then, E(αX) = ω = ω (αx)(ω) Pr(ω) αx(ω) Pr(ω) = α ω X(ω) Pr(ω) = αe(x). Corollary. For α, β R, E(αX + βy ) = αe(x) + βe(y ).
Bayesian Methods: Naïve Bayes
Bayesian Methods: aïve Bayes icholas Ruozzi University of Texas at Dallas based on the slides of Vibhav Gogate Last Time Parameter learning Learning the parameter of a simple coin flipping model Prior
Learning Theory. Machine Learning CSE546 Carlos Guestrin University of Washington. November 25, Carlos Guestrin
Learning Theory Machine Learning CSE546 Carlos Guestrin University of Washington November 25, 2013 Carlos Guestrin 2005-2013 1 What now n We have explored many ways of learning from data n But How good
Naïve Bayes classification
Naïve Bayes classification 1 Probability theory Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. Examples: A person s height, the outcome of a coin toss
2007 Winton. Empirical Distributions
1 Empirical Distributions 2 Distributions In the discrete case, a probability distribution is just a set of values, each with some probability of occurrence Probabilities don t change as values occur Example,
Probability Review and Naïve Bayes
Probability Review and Naïve Bayes Instructor: Alan Ritter Some slides adapted from Dan Jurfasky and Brendan O connor What is Probability? The probability the coin will land heads is 0.5 Q: what does this
Machine Learning Basics Lecture 7: Multiclass Classification. Princeton University COS 495 Instructor: Yingyu Liang
Machine Learning Basics Lecture 7: Multiclass Classification Princeton University COS 495 Instructor: Yingyu Liang Example: image classification indoor Indoor outdoor Example: image classification (multiclass)
Introduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 2. MLE, MAP, Bayes classification Barnabás Póczos & Aarti Singh 2014 Spring Administration http://www.cs.cmu.edu/~aarti/class/10701_spring14/index.html Blackboard
Skyrms s Higher Order Degrees of Belief
Skyrms s Higher Order Degrees of Belief Shivaram Lingamneni March 13, 2012 1 Logical structure Disclaimers: I don t agree with Skyrms, I don t understand many of his points, and I may not understand his
Statistical Learning Theory: Generalization Error Bounds
Statistical Learning Theory: Generalization Error Bounds CS6780 Advanced Machine Learning Spring 2015 Thorsten Joachims Cornell University Reading: Murphy 6.5.4 Schoelkopf/Smola Chapter 5 (beginning, rest
CISC 1100/1400 Structures of Comp. Sci./Discrete Structures Chapter 7 Probability. Outline. Terminology and background. Arthur G.
CISC 1100/1400 Structures of Comp. Sci./Discrete Structures Chapter 7 Probability Arthur G. Werschulz Fordham University Department of Computer and Information Sciences Copyright Arthur G. Werschulz, 2017.
http://imgs.xkcd.com/comics/electoral_precedent.png Statistical Learning Theory CS4780/5780 Machine Learning Fall 2012 Thorsten Joachims Cornell University Reading: Mitchell Chapter 7 (not 7.4.4 and 7.5)
ELEG 3143 Probability & Stochastic Process Ch. 2 Discrete Random Variables
Department of Electrical Engineering University of Arkansas ELEG 3143 Probability & Stochastic Process Ch. 2 Discrete Random Variables Dr. Jingxian Wu wuj@uark.edu OUTLINE 2 Random Variable Discrete Random
Computational Learning Theory (COLT)
Computational Learning Theory (COLT) Goals: Theoretical characterization of 1 Difficulty of machine learning problems Under what conditions is learning possible and impossible? 2 Capabilities of machine
Classification & Information Theory Lecture #8
Classification & Information Theory Lecture #8 Introduction to Natural Language Processing CMPSCI 585, Fall 2007 University of Massachusetts Amherst Andrew McCallum Today s Main Points Automatically categorizing
Computational Learning Theory
Computational Learning Theory Slides by and Nathalie Japkowicz (Reading: R&N AIMA 3 rd ed., Chapter 18.5) Computational Learning Theory Inductive learning: given the training set, a learning algorithm
An Introduction to Statistical Theory of Learning. Nakul Verma Janelia, HHMI
An Introduction to Statistical Theory of Learning Nakul Verma Janelia, HHMI Towards formalizing learning What does it mean to learn a concept? Gain knowledge or experience of the concept. The basic process
Introduction to Machine Learning
Introduction to Machine Learning PAC Learning and VC Dimension Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE
Maximum Likelihood, Logistic Regression, and Stochastic Gradient Training
Maximum Likelihood, Logistic Regression, and Stochastic Gradient Training Charles Elkan elkan@cs.ucsd.edu January 17, 2013 1 Principle of maximum likelihood Consider a family of probability distributions
COS 402 Machine Learning and Artificial Intelligence Fall Lecture 3: Learning Theory
COS 402 Machine Learning and Artificial Intelligence Fall 2016 Lecture 3: Learning Theory Sanjeev Arora Elad Hazan Admin Exercise 1 due next Tue, in class Enrolment Recap We have seen: AI by introspection
Naïve Bayes classification. p ij 11/15/16. Probability theory. Probability theory. Probability theory. X P (X = x i )=1 i. Marginal Probability
Probability theory Naïve Bayes classification Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. s: A person s height, the outcome of a coin toss Distinguish
Machine Learning Basics Lecture 2: Linear Classification. Princeton University COS 495 Instructor: Yingyu Liang
Machine Learning Basics Lecture 2: Linear Classification Princeton University COS 495 Instructor: Yingyu Liang Review: machine learning basics Math formulation Given training data x i, y i : 1 i n i.i.d.
Introduction to AI Learning Bayesian networks. Vibhav Gogate
Introduction to AI Learning Bayesian networks Vibhav Gogate Inductive Learning in a nutshell Given: Data Examples of a function (X, F(X)) Predict function F(X) for new examples X Discrete F(X): Classification
Computational Learning Theory
CS 446 Machine Learning Fall 2016 OCT 11, 2016 Computational Learning Theory Professor: Dan Roth Scribe: Ben Zhou, C. Cervantes 1 PAC Learning We want to develop a theory to relate the probability of successful
Web-Mining Agents Computational Learning Theory
Web-Mining Agents Computational Learning Theory Prof. Dr. Ralf Möller Dr. Özgür Özcep Universität zu Lübeck Institut für Informationssysteme Tanya Braun (Exercise Lab) Computational Learning Theory (Adapted)
CS 361: Probability & Statistics
February 26, 2018 CS 361: Probability & Statistics Random variables The discrete uniform distribution If every value of a discrete random variable has the same probability, then its distribution is called
Computational Learning Theory
Computational Learning Theory [read Chapter 7] [Suggested exercises: 7.1, 7.2, 7.5, 7.8] Computational learning theory Setting 1: learner poses queries to teacher Setting 2: teacher chooses examples Setting
Discrete Mathematics and Probability Theory Summer 2014 James Cook Midterm 1
CS 70 Discrete Mathematics and Probability Theory Summer 2014 James Cook Midterm 1 Thursday July 17, 2014, 12:40pm-2:00pm. Instructions: Do not turn over this page until the proctor tells you to. Don t
Machine Learning (CS 567) Lecture 2
Machine Learning (CS 567) Lecture 2 Time: T-Th 5:00pm - 6:20pm Location: GFS118 Instructor: Sofus A. Macskassy (macskass@usc.edu) Office: SAL 216 Office hours: by appointment Teaching assistant: Cheol
Computational Learning Theory
Computational Learning Theory Pardis Noorzad Department of Computer Engineering and IT Amirkabir University of Technology Ordibehesht 1390 Introduction For the analysis of data structures and algorithms
Bayesian Methods for Machine Learning
Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),
Bernoulli Trials and Binomial Distribution
Bernoulli Trials and Binomial Distribution Sec 4.4-4.5 Cathy Poliak, Ph.D. cathy@math.uh.edu Office in Fleming 11c Department of Mathematics University of Houston Lecture 9-3339 Cathy Poliak, Ph.D. cathy@math.uh.edu
Bayesian decision theory Introduction to Pattern Recognition. Lectures 4 and 5: Bayesian decision theory
Bayesian decision theory 8001652 Introduction to Pattern Recognition. Lectures 4 and 5: Bayesian decision theory Jussi Tohka jussi.tohka@tut.fi Institute of Signal Processing Tampere University of Technology
Machine Learning CMPT 726 Simon Fraser University. Binomial Parameter Estimation
Machine Learning CMPT 726 Simon Fraser University Binomial Parameter Estimation Outline Maximum Likelihood Estimation Smoothed Frequencies, Laplace Correction. Bayesian Approach. Conjugate Prior. Uniform
MATH MW Elementary Probability Course Notes Part I: Models and Counting
MATH 2030 3.00MW Elementary Probability Course Notes Part I: Models and Counting Tom Salisbury salt@yorku.ca York University Winter 2010 Introduction [Jan 5] Probability: the mathematics used for Statistics
Machine Learning
Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 13, 2011 Today: The Big Picture Overfitting Review: probability Readings: Decision trees, overfiting
Bayesian decision making
Bayesian decision making Václav Hlaváč Czech Technical University in Prague Czech Institute of Informatics, Robotics and Cybernetics 166 36 Prague 6, Jugoslávských partyzánů 1580/3, Czech Republic http://people.ciirc.cvut.cz/hlavac,
Machine Learning
Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University August 30, 2017 Today: Decision trees Overfitting The Big Picture Coming soon Probabilistic learning MLE,
Hypothesis Testing. Part I. James J. Heckman University of Chicago. Econ 312 This draft, April 20, 2006
Hypothesis Testing Part I James J. Heckman University of Chicago Econ 312 This draft, April 20, 2006 1 1 A Brief Review of Hypothesis Testing and Its Uses values and pure significance tests (R.A. Fisher)
Empirical Risk Minimization Algorithms
Empirical Risk Minimization Algorithms Tirgul 2 Part I November 2016 Reminder Domain set, X : the set of objects that we wish to label. Label set, Y : the set of possible labels. A prediction rule, h:
Introduction to probability and statistics
Introduction to probability and statistics Alireza Fotuhi Siahpirani & Brittany Baur sroy@biostat.wisc.edu Computational Network Biology Biostatistics & Medical Informatics 826 https://compnetbiocourse.discovery.wisc.edu
Tail Inequalities. The Chernoff bound works for random variables that are a sum of indicator variables with the same distribution (Bernoulli trials).
Tail Inequalities William Hunt Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV William.Hunt@mail.wvu.edu Introduction In this chapter, we are interested
Machine Learning 4771
Machine Learning 4771 Instructor: Tony Jebara Topic 11 Maximum Likelihood as Bayesian Inference Maximum A Posteriori Bayesian Gaussian Estimation Why Maximum Likelihood? So far, assumed max (log) likelihood
1 Direct Proofs Technique Outlines Example Implication Proofs Technique Outlines Examples...
CSE 311: Foundations of Computing I Proof Techniques What Is This? Each of the following is as close as we can get to giving you a template (and a completely worked out example) for every proof technique
Ordinary Least Squares Linear Regression
Ordinary Least Squares Linear Regression Ryan P. Adams COS 324 Elements of Machine Learning Princeton University Linear regression is one of the simplest and most fundamental modeling ideas in statistics
Introduction to Machine Learning Lecture 14. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 14 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Density Estimation Maxent Models 2 Entropy Definition: the entropy of a random variable
Machine Learning Theory (CS 6783)
Machine Learning Theory (CS 6783) Tu-Th 1:25 to 2:40 PM Hollister, 306 Instructor : Karthik Sridharan ABOUT THE COURSE No exams! 5 assignments that count towards your grades (55%) One term project (40%)
Machine Learning CSE546 Sham Kakade University of Washington. Oct 4, What about continuous variables?
Linear Regression Machine Learning CSE546 Sham Kakade University of Washington Oct 4, 2016 1 What about continuous variables? Billionaire says: If I am measuring a continuous variable, what can you do
EE5319R: Problem Set 3 Assigned: 24/08/16, Due: 31/08/16
EE539R: Problem Set 3 Assigned: 24/08/6, Due: 3/08/6. Cover and Thomas: Problem 2.30 (Maimum Entropy): Solution: We are required to maimize H(P X ) over all distributions P X on the non-negative integers
An Algorithms-based Intro to Machine Learning
CMU 15451 lecture 12/08/11 An Algorithmsbased Intro to Machine Learning Plan for today Machine Learning intro: models and basic issues An interesting algorithm for combining expert advice Avrim Blum [Based
Discriminative Models
No.5 Discriminative Models Hui Jiang Department of Electrical Engineering and Computer Science Lassonde School of Engineering York University, Toronto, Canada Outline Generative vs. Discriminative models
Active Learning and Optimized Information Gathering
Active Learning and Optimized Information Gathering Lecture 7 Learning Theory CS 101.2 Andreas Krause Announcements Project proposal: Due tomorrow 1/27 Homework 1: Due Thursday 1/29 Any time is ok. Office
Logistic Regression. Robot Image Credit: Viktoriya Sukhanova 123RF.com
Logistic Regression These slides were assembled by Eric Eaton, with grateful acknowledgement of the many others who made their course materials freely available online. Feel free to reuse or adapt these
Introduction to Machine Learning Fall 2017 Note 5. 1 Overview. 2 Metric
CS 189 Introduction to Machine Learning Fall 2017 Note 5 1 Overview Recall from our previous note that for a fixed input x, our measurement Y is a noisy measurement of the true underlying response f x):
ECE 5984: Introduction to Machine Learning
ECE 5984: Introduction to Machine Learning Topics: Ensemble Methods: Bagging, Boosting Readings: Murphy 16.4; Hastie 16 Dhruv Batra Virginia Tech Administrativia HW3 Due: April 14, 11:55pm You will implement
Bias-Variance Tradeoff
What s learning, revisited Overfitting Generative versus Discriminative Logistic Regression Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University September 19 th, 2007 Bias-Variance Tradeoff
CS 6375: Machine Learning Computational Learning Theory
CS 6375: Machine Learning Computational Learning Theory Vibhav Gogate The University of Texas at Dallas Many slides borrowed from Ray Mooney 1 Learning Theory Theoretical characterizations of Difficulty
DRAFT. Mathematical Methods 2017 Sample paper. Question Booklet 1
1 South Australian Certificate of Education Mathematical Methods 017 Sample paper Question Booklet 1 Part 1 Questions 1 to 10 Answer all questions in Part 1 Write your answers in this question booklet
COMP 551 Applied Machine Learning Lecture 2: Linear Regression
COMP 551 Applied Machine Learning Lecture 2: Linear Regression Instructor: Herke van Hoof (herke.vanhoof@mail.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise
MATH Notebook 5 Fall 2018/2019
MATH442601 2 Notebook 5 Fall 2018/2019 prepared by Professor Jenny Baglivo c Copyright 2004-2019 by Jenny A. Baglivo. All Rights Reserved. 5 MATH442601 2 Notebook 5 3 5.1 Sequences of IID Random Variables.............................
1 Review of The Learning Setting
COS 5: Theoretical Machine Learning Lecturer: Rob Schapire Lecture #8 Scribe: Changyan Wang February 28, 208 Review of The Learning Setting Last class, we moved beyond the PAC model: in the PAC model we
Empirical Risk Minimization, Model Selection, and Model Assessment
Empirical Risk Minimization, Model Selection, and Model Assessment CS6780 Advanced Machine Learning Spring 2015 Thorsten Joachims Cornell University Reading: Murphy 5.7-5.7.2.4, 6.5-6.5.3.1 Dietterich,
Stats notes Chapter 5 of Data Mining From Witten and Frank
Stats notes Chapter 5 of Data Mining From Witten and Frank 5 Credibility: Evaluating what s been learned Issues: training, testing, tuning Predicting performance: confidence limits Holdout, cross-validation,
Machine Learning. Computational Learning Theory. Eric Xing , Fall Lecture 9, October 5, 2016
Machine Learning 10-701, Fall 2016 Computational Learning Theory Eric Xing Lecture 9, October 5, 2016 Reading: Chap. 7 T.M book Eric Xing @ CMU, 2006-2016 1 Generalizability of Learning In machine learning
Probability and Independence Terri Bittner, Ph.D.
Probability and Independence Terri Bittner, Ph.D. The concept of independence is often confusing for students. This brief paper will cover the basics, and will explain the difference between independent
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Bayesian Learning. Tobias Scheffer, Niels Landwehr
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning Tobias Scheffer, Niels Landwehr Remember: Normal Distribution Distribution over x. Density function with parameters
CMPT 882 Machine Learning
CMPT 882 Machine Learning Lecture Notes Instructor: Dr. Oliver Schulte Scribe: Qidan Cheng and Yan Long Mar. 9, 2004 and Mar. 11, 2004-1 - Basic Definitions and Facts from Statistics 1. The Binomial Distribution
Discrete Mathematics and Probability Theory Spring 2014 Anant Sahai Note 10
EECS 70 Discrete Mathematics and Probability Theory Spring 2014 Anant Sahai Note 10 Introduction to Basic Discrete Probability In the last note we considered the probabilistic experiment where we flipped
Generative Learning algorithms
CS9 Lecture notes Andrew Ng Part IV Generative Learning algorithms So far, we ve mainly been talking about learning algorithms that model p(y x; θ), the conditional distribution of y given x. For instance,
Learning Linear Detectors
Learning Linear Detectors Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps Today Detection versus Classification Bayes Classifiers Linear Classifiers Examples of Detection 3 Learning: Detection
One sided tests. An example of a two sided alternative is what we ve been using for our two sample tests:
One sided tests So far all of our tests have been two sided. While this may be a bit easier to understand, this is often not the best way to do a hypothesis test. One simple thing that we can do to get
Computational Learning Theory: Probably Approximately Correct (PAC) Learning. Machine Learning. Spring The slides are mainly from Vivek Srikumar
Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning Spring 2018 The slides are mainly from Vivek Srikumar 1 This lecture: Computational Learning Theory The Theory
Generalization bounds
Advanced Course in Machine Learning pring 200 Generalization bounds Handouts are jointly prepared by hie Mannor and hai halev-hwartz he problem of characterizing learnability is the most basic question
PAC-learning, VC Dimension and Margin-based Bounds
More details: General: http://www.learning-with-kernels.org/ Example of more complex bounds: http://www.research.ibm.com/people/t/tzhang/papers/jmlr02_cover.ps.gz PAC-learning, VC Dimension and Margin-based
Slides for Data Mining by I. H. Witten and E. Frank
Slides for Data Mining by I. H. Witten and E. Frank Predicting performance Assume the estimated error rate is 5%. How close is this to the true error rate? Depends on the amount of test data Prediction
Probability and the Second Law of Thermodynamics
Probability and the Second Law of Thermodynamics Stephen R. Addison January 24, 200 Introduction Over the next several class periods we will be reviewing the basic results of probability and relating probability
Expectation Maximization (EM)
Expectation Maximization (EM) The EM algorithm is used to train models involving latent variables using training data in which the latent variables are not observed (unlabeled data). This is to be contrasted
PATTERN RECOGNITION AND MACHINE LEARNING
PATTERN RECOGNITION AND MACHINE LEARNING Chapter 1. Introduction Shuai Huang April 21, 2014 Outline 1 What is Machine Learning? 2 Curve Fitting 3 Probability Theory 4 Model Selection 5 The curse of dimensionality
Modeling Environment
Topic Model Modeling Environment What does it mean to understand/ your environment? Ability to predict Two approaches to ing environment of words and text Latent Semantic Analysis (LSA) Topic Model LSA
Module Contact: Dr Steven Hayward, CMP Copyright of the University of East Anglia Version 1
UNIVERSITY OF EAST ANGLIA School of Computing Sciences Main Series UG Examination 2016-17 MATHEMATICS FOR COMPUTING B CMP-4005Y Time allowed: 2 hours Answer ANY SIX questions out of SEVEN. Notes are not
Intro to Probability. Andrei Barbu
Intro to Probability Andrei Barbu Some problems Some problems A means to capture uncertainty Some problems A means to capture uncertainty You have data from two sources, are they different? Some problems
(Re)introduction to Statistics Dan Lizotte
(Re)introduction to Statistics Dan Lizotte 2017-01-17 Statistics The systematic collection and arrangement of numerical facts or data of any kind; (also) the branch of science or mathematics concerned
Probability. We will now begin to explore issues of uncertainty and randomness and how they affect our view of nature.
Probability We will now begin to explore issues of uncertainty and randomness and how they affect our view of nature. We will explore in lab the differences between accuracy and precision, and the role
Lecture 1: Probability Fundamentals
Lecture 1: Probability Fundamentals IB Paper 7: Probability and Statistics Carl Edward Rasmussen Department of Engineering, University of Cambridge January 22nd, 2008 Rasmussen (CUED) Lecture 1: Probability