Review of Probability

Transcription

1 Review of robabilit

2 robabilit Theor: Man techniques in speech processing require the manipulation of probabilities and statistics. The two principal application areas we will encounter are: Statistical pattern recognition. Modeling of linear sstems.

3 Events: It is customar to refer to the probabilit of an event. An event is a certain set of possible outcomes of an eperiment or trial. Outcomes are assumed to be mutuall eclusive and, taken together, to cover all possibilities. 3

4 Aioms of robabilit: To an event A we can assign a number, A, which satisfies the following aioms: A 0. S=. If A and B are mutuall eclusive, then A+B=A+B. The number A is called the probabilit of A. 4

5 Aioms of robabilit some consequence: Some immediate consequence: If A is the complement of A, then A A S A A 0,the probabilit of the impossible event, is 0. A. If two event A and B are not mutuall eclusive, we can show that A+B=A+B-AB. 5

6 6 Conditional robabilit: The conditional probabilit of an event A, given that event B has occurred, is defined as: We can infer B A b means of Baes theorem: B AB B A A B B A A B

7 7 Independence: Events A and B ma have nothing to do with each other and the are said to be independent. Two events are independent if AB=AB. From the definition of conditional probabilit: A B A B A B B A B A B A

8 8 Independence: Three events A,B and C are independent onl if: C B A ABC C B BC C A AC B A AB

9 Random Variables: A random variable is a number chosen at random as the outcome of an eperiment. Random variable ma be real or comple and ma be discrete or continuous. In S..,the random variable encounter are most often real and discrete. We can characterize a random variable b its probabilit distribution or b its probabilit densit function pdf. 9

10 Random Variables distribution function: The distribution function for a random variable is the probabilit that does not eceed some value u, F u u and u v F v F u 0

11 Random Variables probabilit densit function: The probabilit densit function is the derivative of the distribution: d f u F u du and, u v f d F v u f d

12 Random Variables epected value: We can also characterize a random variable b its statistics. The epected value of g is written E{g} or <g> and defined as Continuous random variable: g g f d Discrete random variable: g g p

13 Random Variables moments: The statistics of greatest interest are the moment of X. The kth moment of X is the epected value of. X k For a discrete random variable: m X k k p k 3

14 Random Variables mean & variance: The first moment,,is the mean of. Continuous: Discrete: m X f d X X p The second central moment, also known as the variance of p, is given b m X p 4

15 Random Variables : To estimate the statistics of a random variable, we repeat the eperiment which generates the variable a large number of times. If the eperiment is run N times, then each value will occur Np times, thus N k mˆ k i N ˆ i N i N i 5

16 Random Variables Uniform densit: A random variable has a uniform densit on the interval a, b if : 0, F X, a / b a, a a b b f X / b 0, b a a, a b otherwise 6

17 Random Variables Gaussian densit: The Gaussian, or normal densit function is given b: / n ;, e 7

18 Random Variables Gaussian densit: The distribution function of a normal variable is: N ;, n u;, du If we define error function as Thus, erf N ;, e u erf / du 8

19 Two Random Variables: If two random variables and are to be considered together, the can be described in terms of their joint probabilit densit f, or, for discrete variables, p,. Two random variable are independent if p, p p 9

20 Two Random Variables Continue: Given a function g,, its epected value is defined as: Continuous: Discrete: And joint moment for two discrete random variable is: g, g, f, dd g,, g, p, m ij, i j p, 0

21 Two Random Variables Continue: Moments are estimated in practice b averaging repeated measurements: mˆ ij A measure of the dependence of two random variables is their correlation and the correlation of two variables is their joint second moment: N m p, N, i j

22 Two Random Variables Continue: The joint second central moment of, is their covariance: m If and are independent then their covariance is zero. The correlation coefficient of and is their covariance normalized to their standard deviations: r

23 3 Two Random Variables Gaussian Random Variable: Two random variables and are jointl Gaussian if their densit function is : Where r ep, r r r n

24 Two Random Variables Sum of Random Variables: The epected value of the sum of two random variables is : This is true whether and are independent or not And also we have : c c i i i i 4

25 Two Random Variables Sum of Random Variable: The variance of the sum of the two independent random variable is : If two random variable are independent, the probabilit densit of their sum is the convolution of the densities of the individual variables : Continuous: Discrete: f z f u u f z u p z p u p z u du 5

26 Central Limit Theorem Central Limit Theorem informal paraphrase: If man independent random variables are summed, the probabilit densit function pdf of the sum tends toward the Gaussian densit, no matter what their individual densities are. 6

27 Multivariate Normal Densit The normal densit function can be generalized to an number of random variables. Let X be the random vector, N Where Q n / R / Col X, X ep Q T R [ n,..., X ] The matri R is the covariance matri of X R is ositive-definite R T 7

28 Random Functions : A random function is one arising as the outcome of an eperiment. Random functions do not need to be functions of time, but in all cases of interest to us the will be. A discrete stochastic process is characterized b man probabilit densit functions of the form, p,, 3,..., n, t, t, t3,..., t n 8

29 Random Functions : If the individual values of the random signal are independent, then p,,..., n, t, t,..., tn p, t p, t... p n n If these individual probabilit densities are all the same, then we have a sequence of independent, identicall distributed samples i.i.d.., t 9

30 30 mean & autocorrelation The mean is the epected value of t : The autocorrelation function is the epected value of the product : t p t t,,,,,, t t p t t t t r t t

31 ensemble & time average Mean and autocorrelation can be determined in two was: The eperiment can be repeated man times and the average taken over all these functions. Such an average is called ensemble average. Take an one of these function as being representative of the ensemble and find the average from a number of samples of this one function. This is called a time average. 3

32 ergodicit & stationarit If the time average and ensemble average of a random function are the same, it is said to be ergodic. A random function is said to be stationar if its statistics do not change as a function of time. This is also called strict sense stationarit vs. wide sense stationarit. An ergodic function is also stationar. 3

33 33 ergodicit & stationarit For a stationar signal we have: Stationarit is defined as: Where And the autocorrelation function is : t,,,,, p t t p t t,,, p r

34 ergodicit & stationarit When t is ergodic, its mean and autocorrelation are : N lim t N N t N r t t lim t t N N N t N 34

35 cross-correlation The cross-correlation of two ergodic random functions is : N r t t lim t t N N t N The subscript indicates a cross-correlation. 35

36 Random Functions power & cross spectral densit: The Fourier transform of the autocorrelation function of an ergodic random function is called the power spectral densit of t : S r e The cross-spectral densit of two ergodic random functions is : S r e r j j 36

37 37 Random Functions power densit: For an ergodic signal t, can be written as: Then from elementar Fourier transform properties, X X X X X S r r Assuming t is real

38 Random Functions White Noise: If all values of a random signal are uncorrelated, r Then this random function is called white noise The power spectrum of white noise is constant, S White noise is a miture of all frequencies. 38

39 Random Signal in Linear Sstems : Let T[ ] represent the linear operation; then T[ t] T[ t ] Given a sstem with impulse response hn, n n h n n h n A stationar signal applied to a linear sstem ields a stationar output, r S r h h S H 39