Deep Learning for Computer Vision

Size: px

Start display at page:

Download "Deep Learning for Computer Vision"

Avice Daniels
5 years ago
Views:

1 Deep Learning for Computer Vision Lecture 3: Probability, Bayes Theorem, and Bayes Classification Peter Belhumeur Computer Science Columbia University

2 Probability

3 Should you play this game? Game: A fair die is rolled. If the result is 2, 3, or 4, you win $1; if it is 5, you win $2; but if it is 1 or 6, you lose $3.

4 Random Experiment a random experiment is a process whose outcome is uncertain. Examples: Tossing a coin once or several times Picking a card or cards from a deck Measuring temperature of patients...

5 Events & Sample Spaces Sample Space The sample space is the set of all possible outcomes. Simple Events The individual outcomes are called simple events. Event An event is any collection of one or more simple events.

6 Example Experiment: Toss a coin 3 times. Sample space Ω Ω = {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT}. Examples of events include A = {HHH, HHT, HTH, THH} = {at least two heads} B = {HTT, THT, TTH} = {exactly two tails}

7 Basic Concepts (from Set Theory) The union of two events A and B, A B, is the event consisting of all outcomes that are either in A or in B or in both events. The complement of an event A, A c, is the set of all outcomes in Ω that are not in A. The intersection of two events A and B, A B, is the event consisting of all outcomes that are in both events. When two events A and B have no outcomes in common, they are said to be mutually exclusive, or disjoint, events.

8 Example Experiment: toss a coin 10 times and the number of heads is observed. Let A = { 0, 2, 4, 6, 8, 10}. B = { 1, 3, 5, 7, 9}, C = {0, 1, 2, 3, 4, 5}. A B= {0, 1,, 10} = Ω. A B contains no outcomes. So A and B are mutually exclusive. C c = {6, 7, 8, 9, 10}, A C = {0, 2, 4}.

9 Rules Commutative Laws: A B = B A, A B = B A Associative Laws: (A B) C = A (B C ) (A B) C = A (B C) Distributive Laws: (A B) C = (A C) (B C) (A B) C = (A C) (B C) DeMorgan s Laws: n c n n c n c Ai = Ai, Ai = i= 1 i= 1 i= 1 i = 1 A c i.

10 Venn Diagram Ω A A B B

11 A probability is a number assigned to each subset (events) of a sample space Ω that satifies the following rules.

12 Axioms of Probability For any event A, 0 P(A) 1. P(Ω) =1. If A 1, A 2, A n is a partition of A, then P(A) = P(A 1 ) + P(A 2 ) P(A n ) (A 1, A 2, A n is called a partition of A if A 1 A 2 A n = A and A 1, A 2, A n are mutually exclusive.)

13 Properties of Probability For any event A, P(A c ) = 1 - P(A). If A B, then P(A) P(B). For any two events A and B, P(A B) = P(A) + P(B) - P(A B). For three events, A, B, and C, P(A B C) = P(A) + P(B) + P(C) - P(A B) - P(A C) - P(B C) + P(A B C).

14 Frequentist View of Probability The probability of an event a could be defined as: P (a) =lim n!1 N(a) P (a) = lim n!1 n N(a) n Where N(a) is the number that event a happens in n trials

15 Here We Go Again: Not So Basic Probability

16 Bring on the Notation Let Ω be the sample space, ω in Ω be a single outcome, A in Ω a set of outcomes of interest, then 1. P(a) 0 8A 2 2. P( ) =1 3. A i \ A j = ; i, j =) P ([ n i=1a i )= 4. P(;) =0 nx i=1 P (A i )

17 Independence The probability of independent events A, B and C is given by: P (A, B, C) =P (A)P (B)P (C) A and B are independent, if knowing that A has happened does not say anything about B happening

18 Conditional Probability We say probabilty of A given B to mean the probability of event A given that event B occurs. Ω B A

19 Conditional Probability So probabilty of A given B is the probability that both event A and B occur normalized by the probability of event B. P (A B) = P (A, B) P (B)

20 Bayes Theorem Provides a way to convert a-priori probabilities to a- posteriori probabilities: P (A B) = P (B A)P (A) P (B)

21 Random Variables A (scalar) random variable X is a function that maps the outcome of a random event into real scalar values Ω X(ω) ω

22 Random Variable s Distributions Cumulative Probability Distribution (CDF): F X (x) =P (X apple x) Probability Density Function (PDF): p X (x) = df X(x) dx

23 The PDF integrates to 1 So as you would expect: Z 1 1 p X (x)dx =1.0

24 Uniform Distribution A R.V. X that is uniformly distributed between x 1 and x 2 has density function: 1 p X (x) = x 1 apple x apple x 2 x 2 x 1 =0 otherwise X 1 X 2

25 Gaussian (Normal) Distribution A R.V. X that is normally distributed has density function: p(x) = 1 p 2 2 (x µ)2 e 2 2 µ

26 Simple Statistics Expectation (Mean or First Moment): Z 1 E(X) = 1 xp(x) dx Second Moment: E(X 2 )= Z 1 1 x 2 p(x) dx

27 Simple Statistics Variance of X: Var(X) =E[(X E[X]) 2 ] Z 1 = (x E[X]) 2 p(x) dx 1 = E[X 2 ] (E[X]) 2 Standard Deviation of X: Std(X) = p Var(X)

28 Sample Mean Given a set of N samples from a distribution, we can estimate the mean of the distribution by: NX µ = 1 N i=1 x i

29 Sample Variance Given a set of N samples from a distribution, we can estimate the variance of the distribution by: 2 = 1 N 1 NX (x i µ) 2 i=1

30 Bayesian Classifiers

31 Classification: An Example Classify fish species at an Alaskan Canning Factory Species Sea bass Salmon

33 Priors The sea bass/salmon example: Let ω 1 be the state or class that the fish is a salmon Let ω 2 be the state or class that the fish is a sea bass Let P(ω 1 ) be the prior probability that a fish is salmon Let P(ω 2 ) be the prior probability that a fish is sea bass P(ω 1 ) + P( ω 2 ) = 1 (no other species are possible) Pattern Classification, Chapter 2 (Part 1)

34 Dumb Classifier Decision rule with only the prior information: Decide ω 1 if P(ω 1 ) > P(ω 2 ) otherwise decide ω 2 This does not use any of the class conditional information or features Pattern Classification, Chapter 2 (Part 1)

35 Our features are lightness and the width of the fish Fish x T = [x 1, x 2 ] Lightness Width

36 How should we use our features?

37 Minimum Error Rate Classifier Probability of error given x: P(error x) = min [P(ω 1 x), P(ω 2 x)] Minimizing the probability of error: Decide ω 1 if P(ω 1 x) > P(ω 2 x); otherwise decide ω 2 Pattern Classification, Chapter 2 (Part 1)

38 How do we compute P(ω i x)? Pattern Classification, Chapter 2 (Part 1)

39 Bayes Theorem P (! i x) = (x! i)p (! i ) P (x) = (x! i)p (! i ) P i (x! i)p (! i ) = likelihood prior evidence Pattern Classification, Chapter 2 (Part 1)

40 Likelihood (Class-conditional Density) Need the class conditional information: p(x ω 1 ) and p(x ω 2 ) describe the difference in lightness between populations of sea-bass and salmon These are also known as likelihood functions. Pattern Classification, Chapter 2 (Part 1)

41 Pattern Classification, Chapter 2 (Part 1)

42 Pattern Classification, Chapter 2 (Part 1)

43 If our feature space is one dimensional then the boundary that separates the area assigned to one class vs. another class is a point.

44 But what happens as the dimensionality of our feature space increases?

45 x Let s think a classifier as set of scalar functions g i (x) one for each class i that assigns a score to the vector of feature values and then choses the class i with the highest score.

46 So a classifier uses the following decision rule: Choose class i if g i (x) >g j (x) 8 j, i 6= j

47 So our Bayesian classifier assigns a score based on the a posteriori probabilities: g i (x) =P (! i x).

48 So if our feature space is n-dimensional, i.e.,, then the boundaries separating regions that our classifier assigns to the same class is n-1 dimensional surface. x 2 R n

STAT 430/510 Probability

STAT 430/510 Probability Hui Nie Lecture 3 May 28th, 2009 Review We have discussed counting techniques in Chapter 1. Introduce the concept of the probability of an event. Compute probabilities in certain