Bayesian classification CISC 5800 Professor Daniel Leeds

Transcription

1 Bayesian classification CISC 5800 Professor Daniel Leeds Classifying with robabilities Examle goal: Determine is it cloudy out Available data: Light detector: x 0,25 Potential class (atmosheric states): Y={Cloudy, Non-Cloudy} Each class (atmosheric state) y has associated robability distribution P x Actually each y has a likelihood distribution 0.04 P x μ y, σ y 0 2 Classifying with robabilities Examle goal: Determine is it cloudy out Measure light: x Comute P x μ y, σ y y=non-cloudy for y=cloudy and Pick y which gives greatest likelihood P x μ y, σ y argmax y P x μ y, σ y This is Maximum Likelihood classification x=9 P(x=9 Cloudy)= P(x=9 Non-Cloud)= What if there s an eclise? Let s add a third otential class: Y={Cloudy, Non-Cloudy, Eclise} What is most likely class if x=9? Eclises are low robability! Or are they? (Aug 2017) x=9 0 P(x=9 Cloudy)= P(x=9 Non-Cloud)=0.02 P(x=9 Eclise)= 6 1

2 Incororating rior robability Define rior robabilities for each class P y = P(μ y, σ y ) Probability of class y same as robability of arameters μ y, σ y Posterior robability estimated as likelihood rior : P x μ y, σ y P μ y, σ y Classify as argmax y P x μ y, σ y P μ y, σ y Probability review: Bayes rule Recall: and: so: P A B = P(A,B) P(B) P(A, B) = P B A P(A) P(A B) = P B A P(A) P(B) The true osterior Terminology: μ y, σ y are arameters. In general use Here: = μ y, σ y. Posterior estimate is P x P 7 Equivalently: P y x = P x = P D = P D P( ) P(D) 8 The osterior estimate argmax P D P D P( ) Posterior Likelihood x Prior - means roortional We ignore the P(D) denominator because D stays same while comaring different classes (y reresented by ) Tyical classification aroaches MLE Maximum Likelihood: Determine arameters/class which maximize robability of the data argmax P D MAP Maximum A Posteriori: Determine arameters/class that has maximum robability argmax P D

3 Incororating a rior Three classes: Y={Cloudy, Non-Cloudy, Eclise} P(Cloudy)=0.4 P(Non-Cloudy)=0.4 P(Eclise)=0.2 x= P(x=9 Cloudy) P(Cloud) =x.4 =.048 P(x=9 Non-Cloud) P(Non-Cloud) = 0.02x.4 = P(x=9 Eclise) P(Eclise)=x.2 =.032 Bernoulli distribution coin flis We have three coins with known biases (favoring heads or tails) How can we determine our current coin? Fli K times to see which bias it has Data (D): {HHTH, TTHH, TTTT} P D = y Bias ( ): y robability of H for coin y 1 y T - # heads, T - # tails Bernoulli distribution reexamined T P D = y 1 y - # heads, T - # tails Multinomial examle 4-sided die 4 robabilities: side1, side2, side3, side4 3 (Note: side4 = 1 k=1 sidek ) More rigorously: in K trials, side k = P D = 0 if tails on fli k 1 if heads on fli k side k 1 side k y 1 y k P D = Define: δ x = 1 x = 0 0 otherwise δ side k 1 δ side k 2 δ side k 3 δ side k 4 side1 side2 side3 side4 k

4 log(x) ex(x) 2/1/2018 Otimization: finding the maximum likelihood arameter for a fixed class (fixed coin) argmax P(D ) = T argmax y 1 y Equivalently, maximize log P(D ) argmax y H log y + T log 1 y y - robability of Head The roerties of logarithms e a = b log b = a a < b log a < log b log ab = log a + log b log a n = n log a Convenient when dealing with small robabilities x = > = Otimization: finding zero sloe Finding the maximum a osteriori Location of maximum has sloe 0 maximize log P(D ) - robability of Head P D P D P( ) Incororating the Beta rior: argmax d d H log + T log 1 : H log + T log 1 = 0 T 1 = 0 21 P = α 1 (1 ) β 1 B(α,β) argmax P D P( ) = argmax log P D + log P( ) 23 4

5 MAP: estimating (estimating ) argmax log P D + log P() argmax H log + T log 1 + α 1 log + β 1 log 1 log(b α, β ) Intuition of the MAP result y = H + α 1 H + α 1 + T + β 1 T 1 + α 1 Set derivative to 0 β 1 1 = 0 Prior has strong influence when and T small Prior has weak influence when and T large 1 H T + 1 α 1 β 1 = 0 H + α 1 = ( H + T + α 1 + β 1 ) 24 α > β means exect to find coins biased to heads β > α means exect to find coins biased to tails 25 Multinomial distribution Classification What is mood of erson in current minute? M={Hay, Sad} Measure his/her actions every ten seconds: A={Cry, Jum, Laugh, Yell} Data (D): {LLJLCY, JJLYJL, CCLLLJ, JJJJJJ} Bias ( ): Probability table Hay Sad Cry Jum Laugh Yell P D = y Cry Cry y Jum Jum y Laugh Laugh y Yell Yell 26 Multinomial distribution reexamined P D = y Cry Cry y Jum Jum y Laugh Laugh y Yell Yell More rigorously: in K measures, δ trial k = Action = 0 if trial k Action 1 if trial k = Action P D = k i y Action i δ trial k =Action i Classification: Given known likelihoods for each action, find mood that maximizes likelihood of observed sequence of actions 27 (assuming each action is indeendent in the sequence) 5

6 Learning arameters MLE: P A = a i M = m j = j i = #D{A=a i M=m j } #D{M=m j } MAP: P A = a i M = m j = #D(A=a i M=m j )+(γ i 1) #D(M=m j )+ k (γ k 1) P Y = y j = #D(M=m j)+(β j 1) D + m (β m 1) β k is rior robability of each mood class m k γ k is rior robability of each action class a k 29 Multile multi-variate robabilities Mood based on Action, Tunes, Weather argmax P A, T, W How many entries in robability table? # arams = M x( A x T x W -1) Hay Sad Cry, Jazz, Sun Cry, Jazz, Rain Cry, Ra, Snow Laugh, Ra, Rain Yell, Oera, Wind Naïve bayes: Assuming indeendence of inut features argmax P A, T, W = argmax P A P T P W How many entries in robability tables? Hay # arams = M x(( A -1)+( T -1)+( W -1)) = 2x(3+2+3)=16 34 Sad Cry Jum Laugh Yell Hay Sad Jazz Ra Oera Hay Sad Sun Rain Snow Wind Benefits of Naïve Bayes Very fast learning and classifying: For multinomial roblem: Naïve indeendence: learn Y i X i 1 arameters Non-naïve: learn Y i X i 1 arameters Often works even if features are NOT indeendent Y is number of ossible classes X i is number of ossible values for i th feature 35 6

7 Tyical Naïve Bayes classification argmax P D argmax P D P P rior class robability P D = i P Xi where D = x 1 x n e.g., x 1 =Action, x 2 =Tunes is a list of feature values NB (Naïve Bayes): Find class y with to maximize P D 36 7