Bayes Theorem. Jan Kracík. Department of Applied Mathematics FEECS, VŠB - TU Ostrava

Transcription

1 Jan Kracík Department of Applied Mathematics FEECS, VŠB - TU Ostrava

2 Introduction Bayes theorem fundamental theorem in probability theory named after reverend Thomas Bayes ( ) discovered in Bayes work and published by Richard Price ( ) further developed by Pierre-Simon Laplace ( ) forms a cornerstone of statistical learning

3

4 - motivation Area of a plane region Volume of a solid Weight of a solid Count of elements Price of goods

5 - motivation Area of a plane region Volume of a solid Weight of a solid Count of elements Price of goods measures in a common sense

6 - area Area of a plane region A(R)... area of a region R (set) A(R) 0 for any region R A( ) = 0 If R 1 R 2 = then A(R 1 R 2 ) = A(R 1 ) + A(R 2 )

7 - volume Volume of a solid V (S)... volume of a solid S (set) V (S) 0 for any solid S V ( ) = 0 If S 1 S 2 = then V (S 1 S 2 ) = V (S 1 ) + V (S 2 )

8 - count Count of elements of a set C(S)... count of elements of a set S C(S) 0 for any set S C( ) = 0 If S 1 S 2 = then C(S 1 S 2 ) = C(S 1 ) + C(S 2 )

9 What are the common properties of the measures?

10 What are the common properties of the measures? Let Ω be an arbitrary set and F be a set of (all) subsets of Ω.

11 What are the common properties of the measures? Let Ω be an arbitrary set and F be a set of (all) subsets of Ω. is a function M : F [0, + ) {+ } such that for all S F: M(S) 0, M( ) = 0, if S 1, S 2 F and S 1 S 2 = then M(S 1 S 2 ) = M(S 1 ) + M(S 2 )

12 From mathematical point of view, a probability is a measure.

13 Random experiment - outcome of the experiment cannot be completely determined beforehand.

14 Random experiment - outcome of the experiment cannot be completely determined beforehand. Sample space Ω... set of all possible outcomes Random event A Ω A occurs if and only if the outcome ω A

15 Random experiment - outcome of the experiment cannot be completely determined beforehand. Sample space Ω... set of all possible outcomes Random event A Ω A occurs if and only if the outcome ω A A occurs

16 Random experiment - outcome of the experiment cannot be completely determined beforehand. Sample space Ω... set of all possible outcomes Random event A Ω A occurs if and only if the outcome ω A A does not occur

17 Random experiment - example Random experiment: rolling a dice Sample space Ω = {1, 2, 3, 4, 5, 6} Random event A... prime number is rolled Random event B... odd number is rolled

18 Random experiment - example Random experiment: rolling a dice Sample space Ω = {1, 2, 3, 4, 5, 6} Random event A... prime number is rolled Random event B... odd number is rolled Result: ω = 2. A occurred; B did not occur.

19 Let A, B be random events. A B occurs if and only if A occurs and B occurs (intersection) A B occurs if and only if A occurs or B occurs (union) A occurs if and only if A does not occur (complement)

20 Let A, B be random events. A B occurs if and only if A occurs and B occurs (intersection) A B occurs if and only if A occurs or B occurs (union) A occurs if and only if A does not occur (complement) A B

21 Let A, B be random events. A B occurs if and only if A occurs and B occurs (intersection) A B occurs if and only if A occurs or B occurs (union) A occurs if and only if A does not occur (complement) A B

22 Let A, B be random events. A B occurs if and only if A occurs and B occurs (intersection) A B occurs if and only if A occurs or B occurs (union) A occurs if and only if A does not occur (complement) A

23 Let F be the set of all random events in a sample space Ω.

24 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1.

25 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur.

26 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur. From the properties of measure it follows: P( ) = 0 P(A) = 1 P(A)...

27 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur. From the properties of measure it follows: P( ) = 0 P(A) = 1 P(A)......

28 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur. From the properties of measure it follows: P( ) = 0 P(A) = 1 P(A)

29 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur. From the properties of measure it follows: P( ) = 0 P(A) = 1 P(A)

30 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur. From the properties of measure it follows: P( ) = 0 P(A) = 1 P(A)

31 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur. From the properties of measure it follows: P( ) = 0 P(A) = 1 P(A)

32 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur. From the properties of measure it follows: P( ) = 0 P(A) = 1 P(A)

33 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur. From the properties of measure it follows: P( ) = 0 P(A) = 1 P(A)

34 Let F be the set of all random events in a sample space Ω. is a measure on F such that P(Ω) = 1. P(A) is interpreted as a measure of belief that A occur. From the properties of measure it follows: P( ) = 0 P(A) = 1 P(A)

35 Conditional probability Let A, B be random events and P(B) > 0. Conditional probability of A given B P(A B) = P(A B) P(B) P(A B) is a measure of belief that A occurs if B occurred.

36 Conditional probability Let A, B be random events and P(B) > 0. Conditional probability of A given B P(A B) = P(A B) P(B) P(A B) is a measure of belief that A occurs if B occurred. P(A)... size of A (relative to the size of Ω)

37 Conditional probability Let A, B be random events and P(B) > 0. Conditional probability of A given B P(A B) = P(A B) P(B) P(A B) is a measure of belief that A occurs if B occurred. P(A B)... size of A B relative to the size of B

38 Let A, B be random events and P(A) > 0, P(B) > 0. P(A B) = P(B A)P(A) P(B).

39 Let A, B be random events and P(A) > 0, P(B) > 0. P(A B) = P(B A)P(A) P(B) = P(B A)P(A) P(B A)P(A) + P(B A)P(A).

40 Let A, B be random events and P(A) > 0, P(B) > 0. P(A B) = P(B A)P(A) P(B) = P(B A)P(A) P(B A)P(A) + P(B A)P(A) To evaluate P(A B) we need P(A), P(B A), and P(B A).

41 Let A, B be random events and P(A) > 0, P(B) > 0. P(A B) = P(B A)P(A) P(B) = P(B A)P(A) P(B A)P(A) + P(B A)P(A) To evaluate P(A B) we need P(A), P(B A), and P(B A). Prior and posterior probability P(A)... prior belief that A occur P(A B)... posterior belief that A occur if B occurred Occurrence of B (partial information) brings an information about A.

42 P(A B) = P(B A)P(A) P(B) represents a fundamental principle of statistical learning.

43 Example 2: Image Processing - Finding Charlie

44 Example 2: Image Processing - Finding Charlie Charlie was kidnapped last night. Please help!

45 Example: Image Processing Consider a binary digital image (black and white pixels) of a scene. White areas... foreground objects Black areas... background of the scene. Real-world scene

46 Example: Image Processing Consider a binary digital image (black and white pixels) of a scene. White areas... foreground objects Black areas... background of the scene. Real-world scene: small number of large objects is more likely than large number of small objects For example...

47 Original binary image (unobserved): pixels Observed gray-scale image: pixels

48 Original binary image (unobserved): pixels Random events: A 1, A 2,..., A A k occurs if and only if the original image is the k-th one. Observed gray-scale image: pixels

49 Original binary image (unobserved): pixels Random events: A 1, A 2,..., A A k occurs if and only if the original image is the k-th one. Observed gray-scale image: pixels Random events: B 1, B 2,..., B B n occurs if and only if the observed image is the n-th one.

50 Observed image... B O

51 Select observation model: P(B O A k ) for k = 1, 2,..., relates the observation to the original image (different Gaussian noise for black and white pixels)

52 Naive approach: Uniform prior P(A 1 ) = P(A 2 ) = = P(A ) = The image with highest posterior probability P(A E B O ) = max P(A k B O ) k

53 Naive approach: Uniform prior P(A 1 ) = P(A 2 ) = = P(A ) = The image with highest posterior probability P(A E B O ) = max P(A k B O ) k

54 Naive prior: P(A 1 ) = P(A 2 ) = = P(A ) = original result

55 Naive prior: P(A 1 ) = P(A 2 ) = = P(A ) = original result :-(

56 Sophisticated approach: select prior probabilities so that they reflect the prior knowledge: Small number of large objects is more likely than large number of small objects.

57 Sophisticated approach: select prior probabilities so that they reflect the prior knowledge: Small number of large objects is more likely than large number of small objects. c... constant P(A k ) = c exp( H(A k )) H(A k ) is the number of adjacent pixels with different color in image A k H(A k ) equals to the length of the border between black and white areas

58 Visualized posterior probability (of individual pixels being white) 3

59 Visualized posterior probability (of individual pixels being white) 2

60 Visualized posterior probability (of individual pixels being white) 1

61 Visualized posterior probability (of individual pixels being white)

62 Visualized estimated picture

63 Compare the results

64 Compare the results :-)

65 Compare the results Thank you for your attention!

66

67 The original image