Machine Learning 4771

Transcription

1 Machine Learning 4771 Instructor: Tony Jebara

2 Topic 7 Unsupervised Learning Statistical Perspective Probability Models Discrete & Continuous: Gaussian, Bernoulli, Multinomial Maimum Likelihood Logistic Regression Conditioning, Marginalizing, Bayes Rule, Epectations Classification, Regression, Detection Dependence/Independence Maimum Likelihood aïve Bayes

3 Unsupervised Learning Classification O O O O O O O O O Regression, f()=y Density/Structure Estimation Clustering Supervised Feature Selection Anomaly Detection Unsupervised (can help supervised)

4 Statistical Perspective Several problems with framework so far: Only have input-output approaches (SVM, eural et) Pulled non-linear squashing functions out of a hat Pulled loss functions (squared error, etc.) out of a hat Better approach for classification? What if we have multi-class classification? What if other problems, i.e. unobserved values of,y,etc Also, what if we don t have a true function? Eample of Projectile Cannon (c.f. Distal Learning) Would like to train a regression function to control a cannon s angle of fire (y) given target distance ()

5 Statistical Perspective Eample of Projectile Cannon (45 degree problem) = input target distance y = output cannon angle 2 = v 0 g sin( 2y ) + noise output angle input distance What does least squares do? Conditional statistical models address this problem

6 Probability Models Instead of deterministic functions, output is a probability Previously: our output was a scalar ŷ = f ow: our output is a probability e.g. a probability bump: p y p( y) = θ T +b p y subsumes or is a superset of Why is this representation for our answer more general?

7 Probability Models Instead of deterministic functions, output is a probability Previously: our output was a scalar ŷ = f ow: our output is a probability e.g. a probability bump: p y p( y) = θ T +b p y subsumes or is a superset of Why is this representation for our answer more general? A deterministic answer with complete confidence is like putting a probability p y where all the mass is at! ŷ p( y) = δ( y ŷ) p ( y )

8 Probability Models ow: our output is a probability density function (pdf) Probability Model: a family of pdf s with adjustable parameters which lets us select one of many p y E.g.: 1-dim Gaussian distribution given mean parameter µ: p y µ Want mean centered on f() s value ow, linear regression is: = 1 2π e 1 2 y f y f p( y Θ) = ( y µ ) = 1 y µ 2π e 1 2 ( ) 2 = 1 2π e 1 2( y θt b) 2 2 p y p( y) = y f ( ) p( y)

9 Probability Models To fit to data, we typically maimize likelihood of the probability model Log-likelihood = objective function (i.e. negative of cost) for probability models which we want to maimize Define (conditional) likelihood as or log-likelihood as l Θ = p( y i ) i L Θ = log L( Θ) = log p y i i For Gaussian p(y ), maimum likelihood is least squares! log p( y i ) i = log ( y i f ( )) i = log 1 2π = log( 1 2π) ( y f ( )) 2 2 i i ( ) 2 e 1 2 y i f i

10 Probability Models Can etend probability model to 2 bumps: p y Θ = 1 ( y µ ) + 1 ( y µ ) Each mean can be a linear regression fn. = 1 y f ( ) 2 1 p y,θ + 1 ( y f ( )) ( y θ T +b ) = 1 y θ T +b Therefore the (conditional) log-likelihood to maimize is: l ( Θ) = log( 1 ( y θ T 2 i 1 i +b ) ( y θ T +b )) 2 i 2 i 2 Maimize l(θ) using gradient ascent icely handles the cannon firing data

11 Probability Models ow classification: can also go beyond deterministic! Previously: wanted output to be binary ow: our output is a probability e.g. a probability table: p y y=0 y= This subsumes or is a superset again Consider probability over binary events (coin flips!): e.g. Bernoulli distribution (i.e 12 probability table) with parameter α p( y α) = α y ( 1 α) 1 y α 0,1 Linear classification can be done by setting α equal to f(): p( y ) = f ( ) y ( 1 f ( ) ) 1 y f ( ) 0,1 α

12 Probability Models ow linear classification is: p( y ) = f ( ) y ( 1 f ( ) ) 1 y f Log-likelihood is (negative of cost function): log p y i i = logf i y i = y i logf i 1 f i α 0,1 1 y i + 1 y i But, need a squashing function since f() in [0,1] Use sigmoid or logistic again f Called logistic regression new loss function Do gradient descent, similar to logistic output neural net! Can also handle multi-layer f() and do backprop again! log 1 f i = log f i + log 1 f i i class1 [0,1] = sigmoid θ T +b i class 0

13 Generative Probability Models Idea: Etend probability to describe both X and Y Find probability density function over both: p,y E.g. describe data with Multi-Dim. Gaussian (later ) Called a Generative Model because we can use it to synthesize or re-generate data similar to the training data we learned from Regression models & classification boundaries are not as fleible don t keep info about X don t model noise/uncertainty?

14 Properties of PDFs Let s review some basics of probability theory First, pdf is a function, multiple inputs, one output: p( 1,, ) p ( = 0.3,, = 1 ) n = n Function s output is always non-negative: p( 1,, ) n 0 Can have discrete or continuous or both inputs: p( 1 = 1, 2 = 0, 3 = 0, 4 = ) Summing over the domain of all inputs gives unity: p(,y) ddy = 1 p(,y) = y= = Continuous integral, Discrete sum y

15 Properties of PDFs Marginalizing: integrate/sum out a variable leaves a marginal distribution over the remaining ones p,y = p y Conditioning: if a variable y is given we get a conditional distribution over the remaining ones = p (,y ) p y p( y) Bayes Rule: mathematically just redo conditioning but has a deeper meaning (1764) if we have X being data and θ being a model posterior likelihood = p X θ p θ X p θ p X prior evidence

16 Properties of PDFs Epectation: can use pdf p() to compute averages and epected values for quantities, denoted by: E p ( ) Properties: Mean: epected value for E p ( ) { f ( ) } = p( ) f ( ) d or = p { } = p( ) d f ( ) eample: speeding ticket Fine=0$ Fine=20$ epected cost of speeding? f(=0)=0, f(=1)=20 p(=0)=0.8, p(=1)=0.2 Variance: epected value of (-mean) 2, how much varies Var E E { cf ( ) } = ce { f ( ) } { f ( ) +c} = E { f ( ) } +c { f ( ) } { } { } = E f E E 2 = E 2 2E = E { 2 } 2E { }E { } + E { } = E E { } { { } + E { } 2 } { } 2 = E 2 { } E { } 2

17 Properties of PDFs Covariance: how strongly and y vary together Cov {,y} = E E { } { } { } E { }E y Conditional Epectation: Sample Epectation: If we don t have pdf p(,y) can approimate epectations using samples of data f ( ) f ( ) i E p ( ) { } = p E E { y } { } 1 {( y E y )} = E y { } = 1 Sample Mean: E Sample Var: E ( E ( ) ) 2 1 Sample Cov: E ( E ( ) ) y E y { } = p y E y p y y i { ( )} 1 y y dy d y dy = E { y} ( i ) 2 { } ( i ) y i y

18 More Properties of PDFs Independence: probabilities of independent variables multiply. Denote with the following notation: y p(,y) = p( )p( y) y p y = p( ) also note in this case: { y} = p( )p( y)y d dy E p,y y = p( ) d p( y)y dy Conditional independence: when two variables become independent only if another is observed y z y z p y,z y = p( z) p( ) p y = E p ( ) { }E p ( y ) { y}

19 The IID Assumption Most of the time, we will assume that a dataset independent and identically distributed (IID) In many real situations, data is generated by some black bo phenomenon in an arbitrary order. Assume we are given a dataset: X = { 1,, } Independent means that (given the model θ) the probability of our data multiplies: = p ( i i Θ) p 1,, Θ Identically distributed means that each marginal probability is the same for each data point p( 1,, Θ) = p ( i i Θ) = p( i Θ)

20 The IID Assumption Bayes rule says likelihood is probability of data given model posterior X = { 1,, } p( X Θ) = p( 1,, Θ) = p ( i i Θ) The likelihood of Learn joint distribution Learn conditional p θ X under IID assumptions is: = p( i Θ) by maimum likelihood: Θ * = arg ma Θ p( i Θ) = arg ma Θ log p i Θ likelihood p θ = p X θ p( Θ) p y,θ p X evidence prior by ma conditional likelihood: Θ * = arg ma Θ p( y i i,θ) = arg ma Θ log p y i i,θ

21 Uses of PDFs Classification: have p(,y) and given. Asked for discrete y output, give most probable one p( y ) ŷ = arg ma m p( y = m ) p,y Regression: have p(,y) and given. Asked for a scalar y output, give most probable or epected one { } p,y i,y i p( y ) ŷ = Anomaly Detection: if have p(,y) and given both,y. Asked if it is similar threshold threshold { normal,anomaly } p,y { y} arg ma y p y E p y