Bayesian Decision Theory Lecture 2

Transcription

1 Bayesian Decision Theory Lecture 2 Jason Corso SUNY at Buffalo 14 January 2009 J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

2 Overview and Plan Covering Chapter 2 of DHS (in two classes). Bayesian Decision Theory is a fundamental statistical approach to the problem of pattern classification. Quantifies the tradeoffs between various classifications using probability and the costs that accompany such classifications. Assumptions: Decision problem is posed in probabilistic terms. All relevant probability values are known. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

3 Recall the Fish! Recall our example from the first lecture on classifying two fish as salmon or sea bass. And recall our agreement that any given fish is either a salmon or a sea bass; DHS call this the state of nature of the fish. Let s define a (probabilistic) variable ω that describes the state of nature. ω = ω 1 for sea bass (1) ω = ω 2 for salmon (2) Salmon Sea Bass J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

4 Prior Probability Preliminaries The a priori or prior probability reflects our knowledge of how likely we expect a certain state of nature before we can actually observe said state of nature. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

5 Prior Probability Preliminaries The a priori or prior probability reflects our knowledge of how likely we expect a certain state of nature before we can actually observe said state of nature. In the fish example, it is the probability that we will see either a salmon or a sea bass next on the conveyor belt. Note: The prior may vary depending on the situation. If we get equal numbers of salmon and sea bass in a catch, then the priors are equal, or uniform. Depending on the season, we may get more salmon than sea bass, for example. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

6 Prior Probability Preliminaries The a priori or prior probability reflects our knowledge of how likely we expect a certain state of nature before we can actually observe said state of nature. In the fish example, it is the probability that we will see either a salmon or a sea bass next on the conveyor belt. Note: The prior may vary depending on the situation. If we get equal numbers of salmon and sea bass in a catch, then the priors are equal, or uniform. Depending on the season, we may get more salmon than sea bass, for example. We write P (ω = ω 1 ) or just P (ω 1 ) for the prior the next is a sea bass. The priors must exclusivity and exhaustivity. For c states of nature, or classes: c 1 = P (ω i ) (3) i=1 J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

7 Preliminaries Decision Rule From Only Priors Idea Check: What is a reasonable Decision Rule if The only available information is the prior. The cost of any incorrect classification is equal. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

8 Preliminaries Decision Rule From Only Priors Idea Check: What is a reasonable Decision Rule if The only available information is the prior. The cost of any incorrect classification is equal. Decide ω 1 if P (ω 1 ) > P (ω 2 ); otherwise decide ω 2. What can we say about this decision rule? J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

9 Preliminaries Decision Rule From Only Priors Idea Check: What is a reasonable Decision Rule if The only available information is the prior. The cost of any incorrect classification is equal. Decide ω 1 if P (ω 1 ) > P (ω 2 ); otherwise decide ω 2. What can we say about this decision rule? Seems reasonable, but it will always choose the same fish. If the priors are uniform, this rule will behave poorly. Under the given assumptions, no other rule can do better! (We will see this later on.) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

10 Preliminaries Features and Feature Spaces A feature is an observable variable. A feature space is a set from which we can sample or observe values. Features: Length Width Lightness Location of Dorsal Fin For simplicity, let s assume that our features are all continuous values. Denote a scalar feature as x and a vector feature as x. For a d-dimensional feature space, x R d. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

11 Preliminaries Features and Feature Spaces A feature is an observable variable. A feature space is a set from which we can sample or observe values. Features: Length Width Lightness Location of Dorsal Fin For simplicity, let s assume that our features are all continuous values. Denote a scalar feature as x and a vector feature as x. For a d-dimensional feature space, x R d. A note on the use of the term marginals as features (from first lecture): technically, a marginal is a distribution of one or more variables (e.g., p(x)). So, during modeling, when we say a feature is like a marginal, we are actually saying the distribution of a type of feature is like a marginal. This is only for conceptual reasoning. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

12 Preliminaries Class-Conditional Density or Likelihood The class-conditional probability density function is the probability density function for x, our feature, given that the state of nature is ω: p(x ω) (4) Here is the hypothetical class-conditional density p(x ω) for lightness values of sea bass and salmon. p(x ω i ) 0.4 ω ω x J. Corso (SUNY at Buffalo) 9 10Bayesian11Decision Theory 12 Lecture January / 58

13 Posterior Probability Bayes Formula Preliminaries If we know the prior distribution and the class-conditional density, how does this affect our decision rule? Posterior probability is the probability of a certain state of nature given our observables: P (ω x). Use Bayes Formula: P (ω, x) = P (ω x)p(x) = p(x ω)p (ω) (5) p(x ω)p (ω) P (ω x) = p(x) p(x ω)p (ω) = i p(x ω i)p (ω i ) (6) (7) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

14 Preliminaries FIGURE 2.2. Posterior probabilities for the particular priors P(ω 1 ) = 2/3 andp(ω 2 ) J. Corso = 1/3 (SUNY for at the Buffalo) class-conditional Bayesian probability Decision Theory densities Lecture shown 2 in Fig Thus Januaryin2009 this 9 / 58 Posterior Probability Notice the likelihood and the prior govern the posterior. The p(x) evidence term is a scale-factor to normalize the density. For the case of P (ω 1 ) = 2/3 and P (ω 2 ) = 1/3 the posterior is P(ω i x) ω ω x

15 Probability of Error Decision Theory For a given observation x, we would be inclined to let the posterior govern our decision: What is our probability of error? ω = arg max P (ω i x) (8) i J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

16 Probability of Error Decision Theory For a given observation x, we would be inclined to let the posterior govern our decision: What is our probability of error? ω = arg max P (ω i x) (8) i For the two class situation, we have { P (ω 1 x) if we decide ω 2 P (error x) = (9) P (ω 2 x) if we decide ω 1 J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

17 Probability of Error Decision Theory We can minimize the probability of error by following the posterior: Decide ω 1 if P (ω 1 x) > P (ω 2 x) (10) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

18 Probability of Error Decision Theory We can minimize the probability of error by following the posterior: Decide ω 1 if P (ω 1 x) > P (ω 2 x) (10) And, this minimizes the average probability of error too: P (error) = P (error x)p(x)dx (11) (Because the integral will be minimized when we can ensure each P (error x) is as small as possible.) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

19 Decision Theory Bayes Decision Rule (with Equal Costs) Decide ω 1 if P (ω 1 x) > P (ω 2 x); otherwise decide ω 2 Probability of error becomes P (error x) = min [P (ω 1 x), P (ω 2 x)] (12) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

20 Decision Theory Bayes Decision Rule (with Equal Costs) Decide ω 1 if P (ω 1 x) > P (ω 2 x); otherwise decide ω 2 Probability of error becomes P (error x) = min [P (ω 1 x), P (ω 2 x)] (12) Equivalently, Decide ω 1 if p(x ω 1 )P (ω 1 ) > p(x ω 2 )P (ω 2 ); otherwise decide ω 2 I.e., the evidence term is not used in decision making. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

21 Decision Theory Bayes Decision Rule (with Equal Costs) Decide ω 1 if P (ω 1 x) > P (ω 2 x); otherwise decide ω 2 Probability of error becomes P (error x) = min [P (ω 1 x), P (ω 2 x)] (12) Equivalently, Decide ω 1 if p(x ω 1 )P (ω 1 ) > p(x ω 2 )P (ω 2 ); otherwise decide ω 2 I.e., the evidence term is not used in decision making. If we have p(x ω 1 ) = p(x ω 2 ), then the decision will rely exclusively on the priors. Conversely, if we have uniform priors, then the decision will rely exclusively on the likelihoods. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

22 Decision Theory Bayes Decision Rule (with Equal Costs) Decide ω 1 if P (ω 1 x) > P (ω 2 x); otherwise decide ω 2 Probability of error becomes P (error x) = min [P (ω 1 x), P (ω 2 x)] (12) Equivalently, Decide ω 1 if p(x ω 1 )P (ω 1 ) > p(x ω 2 )P (ω 2 ); otherwise decide ω 2 I.e., the evidence term is not used in decision making. If we have p(x ω 1 ) = p(x ω 2 ), then the decision will rely exclusively on the priors. Conversely, if we have uniform priors, then the decision will rely exclusively on the likelihoods. Take Home Message: Decision making relies on both the priors and the likelihoods and Bayes Decision Rule combines them to achieve the minimum probability of error. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

23 Loss Functions Decision Theory A loss function states exactly how costly each action is. As earlier, we have c classes {ω 1,..., ω c }. We also have a possible actions {α 1,..., α a }. The loss function λ(α i ω j ) is the loss incurred for taking action α i when the class is ω j. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

24 Loss Functions Decision Theory A loss function states exactly how costly each action is. As earlier, we have c classes {ω 1,..., ω c }. We also have a possible actions {α 1,..., α a }. The loss function λ(α i ω j ) is the loss incurred for taking action α i when the class is ω j. The Zero-One Loss Function is a particularly common one: { 0 i = j λ(α i ω j ) = i, j = 1, 2,..., c (13) 1 i j It assigns no loss to a correct decision and uniform unit loss to an incorrect decision. (Similar to Dirac delta function...) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

25 Expected Loss a.k.a. Conditional Risk Decision Theory We can consider the loss that would be incurred from taking each possible action in our set. The expected loss is by definition c R(α i x) = λ(α i ω j )P (ω j x) (14) j=1 The zero-one conditional risk is R(α i x) = j i P (ω j x) (15) = 1 P (ω i x) (16) Hence, for an observation x, we can minimize the expected loss by selecting the action that minimizes the conditional risk. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

26 Expected Loss a.k.a. Conditional Risk Decision Theory We can consider the loss that would be incurred from taking each possible action in our set. The expected loss is by definition c R(α i x) = λ(α i ω j )P (ω j x) (14) j=1 The zero-one conditional risk is R(α i x) = j i P (ω j x) (15) = 1 P (ω i x) (16) Hence, for an observation x, we can minimize the expected loss by selecting the action that minimizes the conditional risk. (Teaser) You guessed it: this is what Bayes Decision Rule does! J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

27 Overall Risk Decision Theory Let α(x) denote a decision rule, a mapping from the input feature space to an action, R d {α 1,..., α a }. This is what we want to learn. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

28 Overall Risk Decision Theory Let α(x) denote a decision rule, a mapping from the input feature space to an action, R d {α 1,..., α a }. This is what we want to learn. The overall risk is the expected loss associated with a given decision rule. R = R (α(x) x) p (x) dx (17) Clearly, we want the rule α( ) that minimizes R(α(x) x) for all x. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

29 Bayes Risk The Minimum Overall Risk Decision Theory Bayes Decision Rule gives us a method for minimizing the overall risk. Select the action that minimizes the conditional risk: α = arg min R (α i x) (18) α i c = arg min λ(α i ω j )P (ω j x) (19) α i j=1 The Bayes Risk is the best we can do. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

30 Decision Theory Two-Category Classification Examples Consider two classes and two actions, α 1 when the true class is ω 1 and α 2 for ω 2. Writing out the conditional risks gives: R(α 1 x) = λ 11 P (ω 1 x) + λ 12 P (ω 2 x) (20) R(α 2 x) = λ 21 P (ω 1 x) + λ 22 P (ω 2 x). (21) Fundamental rule is decide ω 1 if In terms of posteriors, decide ω 1 if R(α 1 x) < R(α 2 x). (22) (λ 21 λ 11 )P (ω 1 x) > (λ 12 λ 22 )P (ω 2 x). (23) The more likely state of nature is scaled by the differences in loss (which are generally positive). J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

31 Decision Theory Two-Category Classification Examples Or, expanding via Bayes Rule, decide ω 1 if (λ 21 λ 11 )p(x ω 1 )P (ω 1 ) > (λ 12 λ 22 )p(x ω 2 )P (ω 2 ) (24) Or, assuming λ 21 > λ 11, decide ω 1 if p(x ω 1 ) p(x ω 2 ) = λ 12 λ 22 P (ω 2 ) λ 21 λ 11 P (ω 1 ) (25) LHS is called the likelihood ratio. Thus, we can say the Bayes Decision Rule says to decide ω 1 if the likelihood ratio exceeds a threshold that is independent of the observation x. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

32 Discriminants Pattern Classifiers Version 1: Discriminant Functions Discriminant Functions are a useful way of representing pattern classifiers. Let s say g i (x) is a discriminant function for the ith class. This classifier will assign a class ω i to the feature vector x if or, equivalently g i (x) > g j (x) j i, (26) i = arg max g i (x), decide ω i. i J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

33 FIGURE 2.5. The functional structure of a general statistical pattern classifier which includes d inputs and c discriminant functions g i (x). A subsequent step determines which J. Corso of (SUNY theatdiscriminant Buffalo) values Bayesian is Decision the maximum, Theory Lecture and 2 categorizes 14 the January input 2009 pattern 20 / 58 Discriminants Discriminants as a Network We can view the discriminant classifier as a network (for c classes and a d-dimensional input vector). action (e.g., classification) costs discriminant functions g 1 (x) g 2 (x)... g c (x) input x 1 x 2 x 3... x d

34 Discriminants Bayes Discriminants Minimum Conditional Risk Discriminant General case with risks g i (x) = R(α i x) (27) c = λ(α i ω j )P (ω j x) (28) j=1 Can we prove that this is correct? J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

35 Discriminants Bayes Discriminants Minimum Conditional Risk Discriminant General case with risks g i (x) = R(α i x) (27) c = λ(α i ω j )P (ω j x) (28) j=1 Can we prove that this is correct? Yes! The minimum conditional risk corresponds to the maximum discriminant. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

36 Discriminants Minimum Error-Rate Discriminant In the case of zero-one loss function, the Bayes Discriminant can be further simplified: g i (x) = P (ω i x). (29) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

37 Discriminants Uniqueness Of Discriminants Is the choice of discriminant functions unique? J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

38 Discriminants Uniqueness Of Discriminants Is the choice of discriminant functions unique? No! Multiply by some positive constant. Shift them by some additive constant. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

39 Discriminants Uniqueness Of Discriminants Is the choice of discriminant functions unique? No! Multiply by some positive constant. Shift them by some additive constant. For monotonically increasing function f( ), we can replace each g i (x) by f(g i (x)) without affecting our classification accuracy. These can help for ease of understanding or computability. The following all yield the same exact classification results for minimum-error-rate classification. g i (x) = P (ω i x) = p(x ω i)p (ω i ) j p(x ω j)p (ω j ) (30) g i (x) = p(x ω i )P (ω i ) (31) g i (x) = ln p(x ω i ) + ln P (ω i ) (32) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

40 Discriminants Visualizing Discriminants Decision Regions The effect of any decision rule is to divide the feature space into decision regions. Denote a decision region R i for ω i. One not necessarily connected region is created for each category and assignments is according to: If g i (x) > g j (x) j i, then x is in R i. (33) Decision boundaries separate the regions; they are ties among the discriminant functions. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

41 GURE J. Corso 2.6. (SUNY In this at Buffalo) two-dimensional Bayesian Decision two-category Theory Lecture classifier, 2 the probability 14 January 2009 densities 25 / 58 Discriminants Visualizing Discriminants Decision Regions p(x ω 1 )P(ω 1 ) p(x ω 2 )P(ω 2 ) R 1 R 2 R 2 5 decision boundary 0 0 5

42 Discriminants Two-Category Discriminants Dichotomizers In the two-category case, one considers single discriminant What is a suitable decision rule? g(x) = g 1 (x) g 2 (x). (34) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

43 Discriminants Two-Category Discriminants Dichotomizers In the two-category case, one considers single discriminant g(x) = g 1 (x) g 2 (x). (34) The following simple rule is then used: Decide ω 1 if g(x) > 0; otherwise decide ω 2. (35) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

44 Discriminants Two-Category Discriminants Dichotomizers In the two-category case, one considers single discriminant g(x) = g 1 (x) g 2 (x). (34) The following simple rule is then used: Decide ω 1 if g(x) > 0; otherwise decide ω 2. (35) Various manipulations of the discriminant: g(x) = P (ω 1 x) P (ω 2 x) (36) g(x) = ln p(x ω 1) p(x ω 2 ) + ln P (ω 1) P (ω 2 ) (37) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

45 The Normal Density Background on the Normal Density This next section is a slight digression to introduce the Normal Density (most of you will have had this already). The Normal density is very well studied. It easy to work with analytically. In many pattern recognition scenarios, an appropriate model seems to be where your data is assumed to be continuous-valued, randomly corrupted versions of a single typical value. Central Limit Theorem (Second Fundamental Theorem of Probability). The distribution of the sum of n random variables approaches the normal distribution when n is large. E.g., J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

46 Expectation The Normal Density Recall the definition of expected value of any scalar function f(x) in the continuous p(x) and discrete P (x) cases E[f(x)] = f(x)p(x)dx (38) E[f(x)] = f(x)p (x) (39) x where we have a set D over which the discrete expectation is computed. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

47 The Normal Density Univariate Normal Density Continuous univariate normal, or Gaussian, density: [ 1 p(x) = exp 1 ( ) ] x µ 2 2πσ 2 2 σ. (40) The mean is the expected value of x is µ E[x] = The variance is the expected squared deviation σ 2 E[(x µ) 2 ] = xp(x)dx. (41) (x µ) 2 p(x)dx. (42) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

48 The Normal Density Univariate Normal Density Sufficient Statistics Samples from the normal density tend to cluster around the mean and be spread-out based on the variance. p(x) σ 2.5% 2.5% µ - 2σ µ - σ µ µ + σ µ + 2σ FIGURE 2.7. A univariate normal distribution has roughly 95% of its area in the range x µ 2σ, as shown. The peak of the distribution has value p(µ) = 1/ 2πσ.From: Richard O. Duda, Peter E. Hart, and David G. Stork, Pattern Classification. Copyright c 2001 by John Wiley & Sons, Inc. x J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

49 The Normal Density Univariate Normal Density Sufficient Statistics Samples from the normal density tend to cluster around the mean and be spread-out based on the variance. p(x) σ 2.5% 2.5% µ - 2σ µ - σ µ µ + σ µ + 2σ FIGURE 2.7. A univariate normal distribution has roughly 95% of its area in the range x µ 2σ, as shown. The peak of the distribution has value p(µ) = 1/ 2πσ.From: Richard O. Duda, Peter E. Hart, and David G. Stork, Pattern Classification. Copyright c 2001 by John Wiley & Sons, Inc. The normal density is completely specified by the mean and the variance. These two are its sufficient statistics. We thus abbreviate the equation for the normal density as p(x) N(µ, σ 2 ) (43) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58 x

50 Entropy The Normal Density Entropy is the uncertainty in the random samples from a distribution. H(p(x)) = p(x) ln p(x)dx (44) The normal density has the maximum entropy for all distributions have a given mean and variance. What is the entropy of the uniform distribution? J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

51 Entropy The Normal Density Entropy is the uncertainty in the random samples from a distribution. H(p(x)) = p(x) ln p(x)dx (44) The normal density has the maximum entropy for all distributions have a given mean and variance. What is the entropy of the uniform distribution? The uniform distribution has maximum entropy (on a given interval). J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

52 The Normal Density Multivariate Normal Density And a test to see if your Linear Algebra is up to snuff. The multivariate Gaussian in d dimensions is written as [ 1 p(x) = (2π) d/2 exp 1 ] Σ 1/2 2 (x µ)t Σ 1 (x µ). (45) Again, we abbreviate this as p(x) N(µ, Σ). The sufficient statistics in d-dimensions: µ E[x] = xp(x)dx (46) Σ E[(x µ)(x µ) T ] = (x µ)(x µ) T p(x)dx (47) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

53 The Normal Density The Covariance Matrix Σ E[(x µ)(x µ) T ] = (x µ)(x µ) T p(x)dx Symmetric. Positive semi-definite (but DHS only considers positive definite so that the determinant is strictly positive). The diagonal elements σ ii are the variances of the respective coordinate x i. The off-diagonal elements σ ij are the covariances of x i and x j. What does a σ ij = 0 imply? J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

54 The Normal Density The Covariance Matrix Σ E[(x µ)(x µ) T ] = (x µ)(x µ) T p(x)dx Symmetric. Positive semi-definite (but DHS only considers positive definite so that the determinant is strictly positive). The diagonal elements σ ii are the variances of the respective coordinate x i. The off-diagonal elements σ ij are the covariances of x i and x j. What does a σ ij = 0 imply? That coordinates x i and x j are statistically independent. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

55 The Normal Density The Covariance Matrix Σ E[(x µ)(x µ) T ] = (x µ)(x µ) T p(x)dx Symmetric. Positive semi-definite (but DHS only considers positive definite so that the determinant is strictly positive). The diagonal elements σ ii are the variances of the respective coordinate x i. The off-diagonal elements σ ij are the covariances of x i and x j. What does a σ ij = 0 imply? That coordinates x i and x j are statistically independent. What does Σ reduce to if all off-diagonals are 0? J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

56 The Normal Density The Covariance Matrix Σ E[(x µ)(x µ) T ] = (x µ)(x µ) T p(x)dx Symmetric. Positive semi-definite (but DHS only considers positive definite so that the determinant is strictly positive). The diagonal elements σ ii are the variances of the respective coordinate x i. The off-diagonal elements σ ij are the covariances of x i and x j. What does a σ ij = 0 imply? That coordinates x i and x j are statistically independent. What does Σ reduce to if all off-diagonals are 0? The product of the d univariate densities. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

57 The Normal Density Linear Combinations of Normals Linear combinations of jointly normally distributed random variables, independent or not, are normally distributed. x 2 A t wµ N(A t wµ, I) A w N(µ,Σ) For p(x) N((µ), Σ) and A, a d-by-k matrix, define y = A T x. Then: A t µ A µ P p(y) N(A T µ, A T ΣA) (48) With the covariance matrix, we can calculate the dispersion of the data in any direction or in 0 N(A t µ, A t Σ A) FIGURE 2.8. The action of a linear transformation on the feature space vert an arbitrary normal distribution into another normal distribution. One tr any subspace. tion, A, takes the source distribution into distribution N(A t, A t A). Ano transformation a projection P onto a line defined by vector a leads to N(µ J. Corso (SUNY at Buffalo) Bayesian sureddecision along that Theory line. Lecture While2 the transforms yield 14 January distributions 2009 in a 34 different / 58 σ P t µ a x 1

58 The Normal Density Mahalanobis Distance The shape of the density is determined by the covariance Σ. Specifically, the eigenvectors of Σ give the principal axes of the hyperellipsoids and the eigenvalues determine the lengths of these axes. The loci of points of constant density are hyperellipsoids with constant Mahalonobis distance: x 2 µ x 1 (x µ) T Σ 1 (x µ) (49) FIGURE 2.9. Samples drawn from a two-dimensional Gaussian lie in on the mean. The ellipses show lines of equal probability density From: Richard O. Duda, Peter E. Hart, and David G. Stork, Pattern Clas right c 2001 by John Wiley & Sons, Inc. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

59 The Normal Density General Discriminant for Normal Densities Recall the minimum error rate discriminant, g i (x) = ln p(x ω i ) + ln P (ω i ). If we assume normal densities, i.e., if p(x ω i ) N(µ i, Σ i ), then the general discriminant is of the form g i (x) = 1 2 (x µ i) T Σ 1 i (x µ i ) d 2 ln 2π 1 2 ln Σ i + ln P (ω i ) (50) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

60 The Normal Density Simple Case: Statistically Independent Features with Same Variance What do the decision boundaries look like if we assume Σ i = σ 2 I? J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

61 The Normal Density Simple Case: Statistically Independent Features with Same Variance What do the decision boundaries look like if we assume Σ i = σ 2 I? They are hyperplanes ω 1 ω p(x ω i ) ω 1 ω P(ω 2 )=.5 ω ω R R 1 R 2 P(ω 1 )=.5 P(ω 2 )=.5 x -2 P(ω 1)= R 1 R 2 4 P(ω 2)= P(ω 1 )=.5 R FIGURE If the covariance matrices for two distributions are equal and proportional to the identity matrix, then the distributions are spherical in d dimensions, and the boundary is a generalized hyperplane of d 1 dimensions, perpendicular to the line separating the means. In these one-, two-, and three-dimensional examples, Let s see we indicate why... p(x ω i) and the boundaries for the case P(ω 1) = P(ω 2). In the three-dimensional case, the grid plane separates R 1 from R 2. From: Richard O. Duda, Peter E. Hart, and David G. Stork, Pattern Classification. Copyright c 2001 by John Wiley & Sons, Inc. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

62 The Normal Density Simple Case: Σ i = σ 2 I The discriminant functions take on a simple form: g i (x) = x µ i 2 2σ 2 + ln P (ω i ) (51) Think of this discriminant as a combination of two things 1 The distance of the sample to the mean vector (for each i). 2 A normalization by the variance and offset by the prior. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

63 The Normal Density Simple Case: Σ i = σ 2 I But, we don t need to actually compute the distances. Expanding the quadratic form (x µ) T (x µ) yields g i (x) = 1 2σ 2 [ x T x 2µ T i x + µ T i µ i ] + ln P (ω i ). (52) The quadratic term x T x is the same for all i and can thus be ignored. This yields the equivalent linear discriminant functions w i0 is called the bias. g i (x) = w T i x + w i0 (53) w i = 1 σ 2 µ i (54) w i0 = 1 2σ 2 µt i µ i + ln P (ω i ) (55) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

64 The Normal Density Simple Case: Σ i = σ 2 I Decision Boundary Equation The decision surfaces for a linear discriminant classifiers are hyperplanes defined by the linear equations g i (x) = g j (x). The equation can be written as w T (x x 0 ) = 0 (56) w = µ i µ j x 0 = 1 2 (µ i + µ j ) (57) σ 2 µ i µ j 2 ln P (ω i) P (ω j ) (µ i µ j ) (58) These equations define a hyperplane through point x 0 with a normal vector w. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

65 The Normal Density Simple Case: Σ i = σ 2 I Decision Boundary Equation The decision boundary changes with the prior. p(x ω i ) ω 1 ω p(x ω i ) ω 1 ω x x R 1 R 2 R 1 R 2 P(ω 1 )=.7 P(ω 2 )=.3 P(ω 1 )=.9 P(ω 2 )= ω 1 ω ω 1 ω P(ω 2)=.2 P(ω 2)=.01 P(ω 1)=.8 P(ω 1)=.99-2 R 1 R 2-2 R R 2 3 J. Corso (SUNY at Buffalo) 4 Bayesian Decision Theory Lecture 2 14 January / 58 2

66 The Normal Density General Case: Arbitrary Σ i The discriminant functions are quadratic (the only term we can drop is the ln 2π term): g i (x) = x T W i x + w T i x + w i0 (59) W i = 1 2 Σ 1 i (60) w i = Σ 1 i µ i (61) w i0 = 1 2 µt i Σ 1 i µ i 1 2 ln Σ i + ln P (ω i ) (62) The decision surface between two categories are hyperquadrics. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

67 The Normal Density General Case: Arbitrary Σ i FIGURE Arbitrary Gaussian distributions lead to Bayes decision boundaries that are general hyperquadrics. Conversely, given any hyperquadric, one can find two Gaussian distributions whose Bayesian decision Decision boundary Theory is that Lecture hyperquadric. 2 These variances J. Corso (SUNY at Buffalo) 14 January / 58

68 The Normal Density General Case: Arbitrary Σ i J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58 dec

69 The Normal Density General Case for Multiple Categories R 4 R 3 R 2 R 4 R 1 The decisionquite regions A Complicated for four normal Decisiondistributions. Surface! Even w J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

70 Receiver Operating Characteristics Signal Detection Theory p(x ω i ) ω 1 ω 2 A fundamental way of analyzing a classifier. Consider the following experimental setup: σ σ Suppose we are interested in detecting a single pulse. We can read an internal signal x. µ 1 x* µ 2 x FIGURE During any instant when no external pulse is present, the pr density for an internal signal is normal, that is, p(x ω 1) N(µ 1,σ 2 ); when the signal is present, the density is p(x ω 2) N(µ 2,σ 2 ). Any decision threshol determine the probability of a hit (the pink area under the ω 2 curve, above x ) false alarm (the black area under the ω 1 curve, above x ). From: Richard O. Du E. Hart, and David G. Stork, Pattern Classification. Copyright c 2001 by John Sons, Inc. The signal is distributed about mean µ 2 when an external signal is present and around mean µ 1 when no external signal is present. Assume the distributions have the same variances, p(x ω i ) N(µ i, σ 2 ). J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

71 Receiver Operating Characteristics Signal Detection Theory The detector uses x to decide if the external signal is present. Discriminability characterizes how difficult it will be to decide if the external signal is present without knowing x. d = µ 2 µ 1 σ (63) Even if we do not know µ 1, µ 2, σ, or x, we can find d by using a receiver operating characteristic or ROC curve. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

72 Receiver Operating Characteristics Receiver Operating Characteristics Definitions A Hit is the probability that the internal signal is above x given that the external signal is present P (x > x x ω 2 ) (64) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

73 Receiver Operating Characteristics Receiver Operating Characteristics Definitions A Hit is the probability that the internal signal is above x given that the external signal is present P (x > x x ω 2 ) (64) A Correct Rejection is the probability that the internal signal is below x given that the external signal is not present. P (x < x x ω 1 ) (65) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

74 Receiver Operating Characteristics Receiver Operating Characteristics Definitions A Hit is the probability that the internal signal is above x given that the external signal is present P (x > x x ω 2 ) (64) A Correct Rejection is the probability that the internal signal is below x given that the external signal is not present. P (x < x x ω 1 ) (65) A False Alarm is the probability that the internal signal is above x despite there being no external signal present. P (x > x x ω 1 ) (66) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

75 Receiver Operating Characteristics Receiver Operating Characteristics Definitions A Hit is the probability that the internal signal is above x given that the external signal is present P (x > x x ω 2 ) (64) A Correct Rejection is the probability that the internal signal is below x given that the external signal is not present. P (x < x x ω 1 ) (65) A False Alarm is the probability that the internal signal is above x despite there being no external signal present. P (x > x x ω 1 ) (66) A Miss is the probability that the internal signal is below x given that the external signal is present. P (x < x x ω 2 ) (67) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

76 Receiver Operating Characteristics Receiver Operating Characteristics We can experimentally determine the rates, in particular the Hit-Rate and the False-Alarm-Rate. Basic idea is to assume our densities are fixed (reasonable) but vary our threshold x, which will thus change the rates. The receiver operating characteristic plots the hit rate against the false alarm rate. What shape curve do we want? P(x > x* x ω 2 ) hit 1 d'=3 d'=2 d'=1 d'=0 P(x < x* x ω 2 ) false alarm FIGURE In a receiver operating characteristic (ROC) curve, the abs probability of false alarm, P(x > x x ω 1 ), and the ordinate is the proba P(x > x x ω 2 ). From the measured hit and false alarm rates (here corres x in Fig and shown as the red dot), we can deduce that d = 3. From: J. Corso (SUNY at Buffalo) Bayesian Duda, Decision Peter E. Theory Hart, Lecture and David 2 G. Stork, Pattern 14 January Classification Copyright 49 / 58 1

77 Missing Features Missing or Bad Features Suppose we have built a classifier on multiple features, for example the lightness and width. What do we do if one of the features is not measurable for a particular case? For example the lightness can be measured but the width cannot because of occlusion. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

78 Missing Features Missing or Bad Features Suppose we have built a classifier on multiple features, for example the lightness and width. What do we do if one of the features is not measurable for a particular case? For example the lightness can be measured but the width cannot because of occlusion. Marginalize! Let x be our full feature feature and x g be the subset that are measurable (or good) and let x b be the subset that are missing (or bad/noisy). We seek an estimate of the posterior given just the good features x g. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

79 Missing Features Missing or Bad Features P (ω i x g ) = p(ω i, x g ) p(x g ) (68) = = p(ωi, x g, x b )dx b p(x g ) p(ωi x)p(x)dx b p(x g ) (69) (70) = gi (x)p(x)dx b p(x)dxb (71) We will cover the Expectation-Maximization algorithm later. This is normally quite expensive to evaluate unless the densities are special (like Gaussians). J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

80 Bayesian Belief Networks Statistical Independence Two variables x i and x j are independent if p(x i, x j ) = p(x i )p(x j ) (72) x 3 x 1 x 2 FIGURE A three-dimensional distribution which obeys p(x1, x3) = p(x1)p(x3); thus here x1 and x3 are statistically independent but the other feature pairs are not. From: Richard O. Duda, Peter E. Hart, and David G. Stork, Pattern Classification. Copyright c 2001 by John Wiley & Sons, Inc. J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

81 Bayesian Belief Networks An Early Graphical Model We represent these statistical dependencies graphically. Bayesian Belief Networks, or Bayes Nets, are directed acyclic graphs. Each link is directional. No loops. The Bayes Net factorizes the distribution into independent parts (making for more easily learned and computed terms). J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

82 Bayesian Belief Networks Simple Example of Conditional Independence From Russell and Norvig Consider a simple example consisting of four variables: the weather, the presence of a cavity, the presence of a toothache, and the presence of other mouth-related variables such as dry mouth. The weather is clearly independent of the other three variables. And the toothache and catch are conditionally independent given the cavity (one as no effect on the other given the information about the cavity). Weather Cavity Toothache Catch J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

83 Bayesian Belief Networks Bayes Nets Components Each node represents one variable (assume discrete for simplicity). A link joining two nodes is directional and it represents conditional probabilities. The intuitive meaning of a link is that the source has a direct influence on the sink. Since we typically work with discrete distributions, we evaluate the conditional probability at each node given its parents and store it in a lookup table called a conditional probability table. P(c a) C P(e c) P(f e) F P(a) A P(c d) E P(g f) G D P(g e) P(b) B P(d b) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

84 Key: given knowledge of the values of some nodes in the network, we can apply Bayesian inference to determine the maximum posterior values of the unknown variables! J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58 Bayesian Belief Networks A More Complex Example From Russell and Norvig Burglary P(B).001 Earthquake P(E).002 Alarm B T T F F E T F T F P(A) JohnCalls A T F P(J) MaryCalls A T F P(M).70.01

85 Bayesian Belief Networks Full Joint Distribution on a Bayes Net Consider a Bayes network with n variables x 1,..., x n. Denote the parents of a node x i as P(x i ). Then, we can decompose the joint distribution into the product of conditionals P (x 1,..., x n ) = n P (x i P(x i )) (73) i=1 J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58

86 Bayesian Belief Networks Belief at a Single Node What is the distribution at a single node, given the rest of the network and the evidence e? Parents of X, the set P are the nodes on which X is conditioned. Children of X, the set C are the nodes conditioned on X. Use the Bayes Rule, for the case on the right: or more generally, A P(x a) P(c x) C X B P(x b) P(d x) D parents of X children of X FIGURE A portion of a belief network, consisting of a node X values (x 1, x 2,...), its parents (A and B), and its children (C and D). Duda, Peter E. Hart, and David G. Stork, Pattern Classification. Cop John Wiley & Sons, Inc. P (a, b, x, c, d) = P (a, b, x c, d)p (c, d) (74) = P (a, b x)p (x c, d)p (c, d) (75) P (C(x), x, P(x) e) = P (C(x) x, e)p (x P(x), e)p (P(x), e) (76) J. Corso (SUNY at Buffalo) Bayesian Decision Theory Lecture 2 14 January / 58