Review: mostly probability and some statistics

Transcription

1 Review: mostly probability and some statistics C2 1

2 Content robability (should know already) Axioms and properties Conditional probability and independence Law of Total probability and Bayes theorem Random Variables Discrete Continuous airs of Random Variables Random Vectors Gaussian Random Variable 2

3 Basics We are performing a random experiment (catching one fish from the sea) Sample space S: the S: all fish in the sea set of all possible outcomes event A An event A: a set of possible outcomes of total number of events: 2 12 experiment, i.e. a subset of S all events in S robability law:a rule that assigns probabilities to events in an experiment A (A) probability events

4 Axioms of robability (A) (S) 0 = 1 3. If AIB = then ( AU B) = ( A) + ( B) 4

5 roperties of robability ( ) = 0 (A) 1 ( A c ) = 1 ( A) A B ( A) < ( B) ( AUB) = ( A) + ( B) ( AIB) N U k= 1 { A } ( ) i I Aj =, i, j A k = Ak N k= 1 5

6 Conditional robability If A and B are two events, and we know that event B has occurred, then (if (B)>0) (A B) (A B)= (B) S U A A B U B B occurred A B U B the new sample space is B, the new A is old A U B multiplication rule (A B)= U (A B) (B) 6

7 Independence A and B are independent events if (A B) = (A) (B) U By the law of conditional probability, if A and B are independent (A) (B) (A B) = = (A) (B) If two events are not independent, then they are said to be dependent 7

8 Law of Total robability B, B,,B partition S 1 2 n B 1 Consider an event A Thus B 2 A = U U U A U B 1 A U B 2 A U B 3 A A U B 3 B 4 B 4 ( ) ( ) ( ) ( ) ( ) A = AI B + Or using multiplication rule: 1 + AIB 2 + AIB 3 AIB 4 ( ) ( ) ( ) ( ) ( ) A = A B K+ 1 B1 + A B4 B4 n ( ) = ( A B ) ( ) k B k A k= 1 sample spaces

9 Bayes Theorem Let B, B,, B, be a partition of the 1 2 n sample space S. Suppose event A occurs. What is the probability of event B? Answer: Bayes Rule i ( B A) i from conditional probability ( Bi I A) ( A) = = n k = 1 ( A B ) ( B ) i ( A B ) ( B ) from the law of total probability k i k One of the most useful tools we are going to use 9

10 Random Variables A random variable X is a function from sample space S to a real number. X: S R S X X X X R (# of fins) X is random due to randomness of its argument ( X = a) = ( X ( ω ) = a) = ( ω X ( ω) = a)

11 Two Types of Random Variables Discrete random variable has countable number of values number of fish fins (0,1,2,.,30) Continuous random variable has continuous number of values fish weight (any real number between 0 and 100)

12 Cumulative Distribution Function Given a random variable X, CDF is defined as F ( a ) = ( X a ) CDF for discrete rv CDF for continuous rv 1 F(30) F(20) 1 # fins fish weight

13 roperties of CDF 1. F(a) is non decreasing lim b lim b F F ( b) = 1 ( b) = 0 F ( ) ( ) a = F(30) F(20) 1 X a CDF for continuous rv fish weight Questions about X can be asked in terms of CDF ( a < X b) = F( b) F( a) Example: (fish weights between 20 and 30)=F(30)-F(20)

14 Discrete RV: robability Mass Function Given a discrete random variable X, we define the probability mass function as ( ) p ( a) = X = Satisfies all axioms of probability CDF in discrete case satisfies ( X a) = ( X = a = p( a) F ( a) = ) x a a x a 14

15 Continuous RV: robability Density Function Given a continuous RV X, we say f(x) is its probability density function if a F( a) = ( X a) = f ( x)dx and, more generally ( X b) f ( x)dx a b = a 15

16 roperties of robability Density Function d dx F ( ) ( ) a ( X = a) = f ( x) dx = 0 a ( X ) = f ( x) dx = 1 f x = f ( x) 0 x 16

17 probability mass probability density pmf pdf # fins fish weight true probability (fish has 2 or 3 fins)= =p(2)+p(3)= take sums density, not probability (fish weights 30kg) 0.6 (fish weights 30kg)=0 (fish weights between 29 and 31kg)= 31 f ( x) dx integrate 29

18 Expected Value Useful characterization of a r.v. Also known as mean, expectation, or first moment discrete case: continuous case: µ = E ( ) ( ) = Expectation can be thought of as the average over many experiments X ( ) x x p x µ = E X = x f( x) dx 18

19 Expected Value for Functions of X Let g(x) be a function of the r.v. X. Then discrete case: continuous case: [ g( X) ] g( x p( x) E = x ) [ g( X) ] g( x) E = f( x) dx An important function of X: [X-E(X)] Variance E[[X-E(X)] E(X)] ] = var(x)= )=σ Variance measures the spread around the mean Standard deviation = [var(x)] 1/2, has the same units as the r.v. X 19

20 roperties of Expectation If X is constant r.v. X=c, then E(X) = c If a and b are constants, E(aX+b)=aE(X)+b More generally, E ( ) n ( ) n a X + c = ( a E( X ) + c ) i = 1 i i i i = 1 i i i If a and b are constants, then 2 var(ax+b)= a var(x) 20

21 airs of Random Variables Say we have 2 random variables: Fish weight X Fish lightness Y Can define joint CDF F a,b = X a,y b = ω S X ω a,y ω b ( ) ( ) ( ( ) ( ) ) Similar to single variable case, can define discrete: joint probability mass function ( ) ( ) p a, b = X = a, Y = continuous: joint density function ( ) ( ) a X b, c Y d = f x, y b a x c y b d ( ) f x, y dxdy 21

22 Marginal Distributions given joint mass function p X,Y (x,y), marginal, i.e. probability mass function for r.v. X can be obtained from p X,Y (x,y) X ( x) = p ( ) X, Y x y p ( ) = ( ) Y y px, Y x, y p, y marginal densities f X (x) and f Y (y) are obtained from joint density f X,Y (x,y) by integrating x f X y = y= x = x= ( x) = f ( X, Y x, y)dy f ( y) = f ( x, y)dx Y X, Y 22

23 Independence of Random Variables r.v. X and Y are independent if ( ) ( ) ( ) X x, Y y = X x Y y Theorem: r.v. X and Y are independent if and only if p ( ) ( ) ( ), y x, y py y px x (discrete) x = ( ) ( ) ( ) fx, y x, y = fy y fx x (continuous) 23

24 More on Independent RV s If X and Y are independent, then E(XY)=E(X)E(Y) Var(X+Y)=Var(X)+Var(Y) G(X) and H(Y) are independent 24

25 Covariance Given r.v. X and Y, covariance is defined as: cov ( ) [( ( ))( ( ))] ( ) ( ) ( ) X, Y = E X E X Y E Y = E XY E X E Y Covariance is useful for checking if features X and Y give similar information Covariance (from co-vary) indicates tendency of X and Y to vary together If X and Y tend to increase together, Cov(X,Y) > 0 If X tends to decrease when Y increases, Cov(X,Y) < 0 If decrease (increase) in X does not predict behavior of Y, Cov(X,Y) is close to 0 25

26 Covariance Correlation If cov(x,y) = 0, then X and Y are said to be uncorrelated (think unrelated). However X and Y are not necessarily independent. If X and Y are independent, cov(x,y) = 0 Can normalize covariance to get correlation ( X, Y) ( ) ( ) cov 1 cor( X, Y) = 1 var X var Y 26

27 Random Vectors Generalize from pairs of r.v. to vector of r.v. X= [X X X ] (think multiple features) Joint CDF, DF, MF are defined similarly to the case of pair of r.v. s Example: F ( ) ( ) x 1, x2,..., xn X1 x1, X2 x2 =,..., X All the properties of expectation, variance, covariance transfer with suitable modifications n x n 27

28 Covariance Matrix characteristics summary of random vector T cov(x)=cov[x X X ] = Σ =E[(X- µ)(x- µ) ]= 1 2 n E(X µ )(X µ ) E(X µ )(X µ ) E(X µ )(X µ ) n n 1 1 E(X µ )(X µ ) n n 1 1 E(X µ )(X µ ) n n 2 2 E(X µ )(X µ ) n n n n variances σ 2 1 c 21 c 12 c 13 σ 2 2 c 31 c 23 c 32 σ 2 3 covariances 28

29 Normal or Gaussian Random Variable Has density f ( ) x = σ 1 x µ 1 2 σ e 2π 2 Mean µ, and variance σ 2 29

30 Multivariate Gaussian has density f ( ) x = ( ) 2π 1 n / 2 1/ 2 e 1 2 [( ) ( )] t 1 x µ x µ mean vector covariance matrix [ µ ] µ =,K 1, µ n 30

31 Conditional Mass Function: Bayes Rule Define conditional mass function of X given Y=y by The law of Total robability: ( x) = ( x,y ) = ( x y ) ( y ) y ( x y ) = y is fixed y ( x, y ) ( y ) The Bayes Rule: ( y x ) = ( y, x ) ( x ) = y ( x y ) ( y ) ( x y ) ( y )

32 Conditional Density Function: Continuous RV Does it make sense to talk about conditional density p(x y) if Y is a continuous random variable? After all, r[y=y]=0, so we will never see Y=y in practice Measurements have limited accuracy. Can interpret observation y as observation in interval [y-ε, y+ε], and observation x as observation in interval [x-ε, x+ε] y-ε y+ε x-ε x+ε y x

33 Conditional Density Function: Continuous RV Let B(x) denote interval [x-ε,x+ε] r x + ε [ X B( x) ] = p( x) dx 2ε p( x) x ε Similarly r[ Y B( y )] 2ε p( y ) 2 r[ X B( x) IY B( y )] 4ε p( x,y ) Thus we should have ( x y ) p ( x y ) Which can be simplified to: p r r [ X B( x) IY B( y )] 2ε r[ Y B( y )] p(x) x-ε x x+ε [ X B( x) Y B( y )] 2ε ( x,y ) ( y ) p p

34 Conditional Density Function: Continuous RV Define conditional density function of X given Y=y by p ( ) ( x, y ) p x y = p( y ) This is a probability density function because: y is fixed ( x, y ) p( y ) ( x, y ) p dx p p p ( x y ) dx = dx = = = p p The law of Total robability: ( y ) ( x) p( x, y) dy = p( x y ) p( y ) p = dy ( y ) ( y ) 1

35 Conditional Density Function: Bayes Rule The Bayes Rule: p ( y x ) = p p ( y, x ) ( x ) = p p ( x y ) p( y ) ( x y ) p( y ) dy