1 Review of Probability and Distributions

Transcription

1 Random variables. A numerically valued function X of an outcome ω from a sample space Ω X : Ω R : ω X(ω) is called a random variable (r.v.), and usually determined by an experiment. We conventionally denote random variables by uppercase letters X,Y,Z,U,V,... from the end of the alphabet. In particular, a discrete random variable is a random variable that can take values on a finite set {a,a 2,...,a n } of real numbers (usually integers), or on a countably infinite set {a,a 2,a 3,...}. The statement such as X = a i is an event since {ω : X(ω) = a i } is a subsetof the sample space Ω. Thus, we can consider the probability of the event {X = a i }, denoted by P(X = a i ). The function p(a i ) := P(X = a i ) over thepossiblevaluesofx, say a,a 2,..., iscalled afrequency function, oraprobability mass function. Continuous random variable. A continuous random variable is a random variable whose possible values are real values such as 78.6, 5.7, 0.24, and so on. Examples of continuous random variables include temperature, height, diameter of metal cylinder, etc. In what follows, a random variable always means a continuous random variable, unless it is particularly said to be discrete. The probability distribution of a random variable X specifies how its values are distributed over the real numbers. This is completely characterized by the cumulative distribution function (cdf). The cdf F(t) := P(X t). represents the probability that the random variable X is less than or equal to t. Then we often say that the random variable X is distributed as F. Probability distributions. It is often the case that the probability of the random variable X being in any particular range of values is given by the area under a curve over that range of values. This curve is called the probability density function (pdf) of the random variable X, denoted by f(x). Thus, the probability that a X b can be expressed as P(a X b) = b a f(x)dx. Furthermore, we can find the following relationship between cdf and pdf: F(t) = P(X t) = t f(x)dx. Joint distributions. When a pair (X, Y) of random variables is considered, a joint density function f(x, y) is used to compute probabilities constructed from the random variables X and Y simultaneously. Given the joint density function f(x,y), the distribution for each of X and Y is called the marginal distribution. The marginal density functions of X and Y, denoted by f X (x) and f Y (y), are given respectively by f X (x) = f(x,y)dy and f Y (y) = f(x,y)dx. Lecture Note Page

2 Conditional probability distributions. Suppose that two random variables X and Y has a joint density function f(x,y). If f X (x) > 0, then we can define the conditional density function f Y X (y x) given X = x by f Y X (y x) = f(x,y) f X (x). Similarly we can define the conditional density function f X Y (x y) given Y = y by f X Y (x y) = f(x,y) f Y (y) if f Y (y) > 0. Then, clearly we have the following relation f(x,y) = f Y X (y x)f X (x) = f X Y (x y)f Y (y). Independence of random variables. If the joint density function f(x, y) for continuous random variables X and Y is expressed in the form then X and Y are said to be independent. f(x,y) = f X (x)f Y (y) for all x,y, Expectation. Let X be a random variable, and let f is the pdf of X. Then the expectation (or expected value, or mean) of the random variable X is given by E[X] = xf(x)dx. () We denote the expected value () by E(X), µ, or µ X. For a function g, we define the expectation E[g(X)] of the function g(x) of the random variable X by E[g(X)] = g(x)f(x)dx. One can think of the expectation E(X) as an operation on a random variable X which returns the average value for X. It is also useful to observe that this is a linear operator satisfying the following properties: [ ] n n E a+ b i X i = a+ b i E[X i ] with random variables X,...,X n and constants a and b,...,b n. Variance and covariance. The variance of a random variable X, denoted by Var(X), is the expected value of the squared difference between the random variable and its expected value E(X). Thus, it is given by Var(X) := E[(X E(X)) 2 ] = E[X 2 ] (E[X]) 2. Thesquare-root Var(X) is called the standard error (SE) (or standard deviation (SD)) of the random variable X. Now suppose that we have two random variables X and Y. Then the covariance of two random variables X and Y can be defined as Cov(X,Y) := E((X µ x )(Y µ y )) = E(XY) E(X) E(Y), where µ x = E(X) and µ y = E(Y). The covariance Cov(X,Y) measures the strength of the dependence of the two random variables. Lecture Note Page 2

3 In contrast to the expectation, the variance is not a linear operator: The variance of ax + b with constants a and b is given by Var(aX +b) = a 2 Var(X). For two random variables X and Y, we have Var(X +Y) = Var(X)+Var(Y)+2Cov(X,Y). (2) However, if X and Y are independent, then Cov(X,Y) = 0, and consequently Var(X+Y) = Var(X)+ Var(Y). In general, when we have a sequence of independent random variables X,...,X n, we have Var(X + +X n ) = Var(X )+ +Var(X n ). Normal distributions. A normal distribution is represented by a family of distributions which have the same general shape, sometimes described as bell shaped. The normal distribution has the pdf f(x) := ] [ σ 2π exp (x µ)2 2σ 2, (3) which depends upon two parameters µ and σ 2. Here π = is the famous pi (the ratio of the circumference of a circle to its diameter), and exp(u) is the exponential function e u with the base e = of the natural logarithm. Much celebrated integrations are the following: σ 2π σ 2π σ 2π e (x µ)2 2σ 2 dx =, (4) xe (x µ)2 2σ 2 dx = µ, (x µ) 2 e (x µ)2 2σ 2 dx = σ 2. The integration (4) guarantees that the density function (3) always represents a probability density no matter what values the parameters µ and σ 2 are chosen. We say that a random variable X is normally distributed with parameter (µ,σ 2 ) when X has the density function (3). The parameter µ, called mean (or, location parameter), provides the center of the density, and the density function f(x) is symmetric around µ. The parameter σ is a standard deviation (or, a scale parameter); small values of σ lead to high peaks but sharp drops. Larger values of σ lead to flatter densities. The shorthand notation X N(µ,σ 2 ) is often used to express that X is a normal random variable with parameter (µ,σ 2 ). Linear transform of normal random variable. One of the important properties of normal distribution is that if X is a normal random variable with parameter (µ,σ 2 ) [that is, the pdf of X is given by the density function (3)]. then Y = ax + b is also a normal random variable having parameter (aµ+b,(aσ) 2 ). In particular, X µ (5) σ becomes a normal random variable with parameter (0, ), called the z-score. Then the normal density φ(x) with parameter (0, ) becomes φ(x) := e x2 2, 2π Lecture Note Page 3

4 which is known as the standard normal density. We define the cdf Φ by Φ(t) := t φ(x) dx. Sample statistics. Let X,...,X n be independent and identically distributed random variables ( iid random variables for short). Then the random variable X = n is called the sample mean, and the random variable S 2 = n (X i n X) 2 is called the sample variance. n Sample statistics under normal assumption. Assume that X,...,X n are iid normal random variables with parameter (µ,σ 2 ). Then the sample statistics X and S 2 have the following properties: (a) E[ X] = µ and Var( X) = σ2 n. (b) The sample variance S 2 can be rewritten as ( n ) ( n ) S 2 = Xi 2 n n X 2 = (X i µ) 2 n( X µ) 2. n In particular, we can obtain ( n ) E[S 2 ] = E[(X i µ) 2 ] ne[( n X µ) 2 ] X i ( n ) = Var(X i ) nvar( n X) = σ 2. (c) X has the normal distribution with parameter (µ,σ 2 /n), and X and S 2 are independent. Furthermore, W n = (n )S2 σ 2 = n σ 2 (X i X) 2 has the chi-square distribution with (n ) degrees of freedom. Transformation of one random variable. Let X be a random variable with pdf f X (x), and let g(x) be a differentiable and strictly monotonic function (that is, either strictly increasing or strictly decreasing) on real line. Then the inverse function g exists, and the random variable Y = g(x) has the pdf f Y (y) = f X (g (y)) d dy g (y). (6) Linear transform of random variable. Let X be a random variable with pdf f X (x). Then we can define the linear transform Y = ax + b with real values a and b. To compute the density function f Y (y) of the random variable Y, we can apply (6) to obtain f Y (y) = f X ( y b a ) a. Lecture Note Page 4

5 For example, suppose that X is a normal random variable with parameter (µ,σ 2 ). Then the density function f Y (y) for the linear transform Y = ax +b becomes ] f Y (y) = [ ( a σ) 2π exp (y (aµ+b))2 2(aσ) 2, which implies that Y is a normal random variable with parameter (aµ+b,(aσ) 2 ). Change of variables in double integral. For a rectangle A = {(x,y) : a x b, a 2 y b 2 } on a plane, the integration of a function f(x,y) over A is formally written as A f(x,y)dxdy = b2 b a 2 a f(x,y)dxdy (7) Suppose that a transformation (g (x,y),g 2 (x,y)) is differentiable and has the inverse transformation (h (u,v),h 2 (u,v)) satisfying { { x = h (u,v) u = g (x,y) if and only if. (8) y = h 2 (u,v) v = g 2 (x,y) Then we can state the change of variable in the double integral (7) as follows: (i) define the inverse image of A by h (A) = g(a) = {(g (x,y),g 2 (x,y)) : (x,y) A}, (ii) calculate the Jacobian [ J h (u,v) = det u h (u,v) v h ] (u,v) u h 2(u,v) v h, 2(u,v) (iii) finally, if it does not vanish for all x,y, then we can obtain f(x,y)dxdy = f(h (u,v),h 2 (u,v)) J h (u,v) dudv. A g(a) Transformation of two random variables. Suppose that (a) random variables X and Y has a joint density function f X,Y (x,y), (b) a transformation (g (x,y),g 2 (x,y)) is differentiable, and (c) its inverse transformation (h (u,v),h 2 (u,v)) satisfies (8). Then the random variables U = g (X,Y) and V = g 2 (X,Y) has the joint density function f U,V (u,v) = f X,Y (h (u,v),h 2 (u,v)) J h (u,v). Order statistics. Let X,...,X n be independent and identically distributed random variables (iid random variables) with the common cdf F(x) and the pdf f(x). Then we can define the random variable V = min(x,...,x n ) of the minimum of X,...,X n. To find the distribution of V, consider the survival function P(V > x) of V and calculate as follows. P(V > x) = P(X > x,...,x n > x) = P(X > x) P(X n > x) = [ F(x)] n. Thus, we can obtain the cdf F V (x) and the pdf f V (x) of V F V (x) = [ F(x)] n and f V (x) = d dx F V(x) = nf(x)[ F(x)] n. Lecture Note Page 5

6 In particular, if X,...,X n be independent exponential random variables with the common parameter λ, then the minimum min(x,...,x n ) has the density (nλ)e (nλ)x which is again the exponential density function with parameter (nλ). Let X,...,X n be iid random variables with the common cdf F(x) and the pdf f(x). When we sort X,...,X n as X () < X (2) < < X (n), the random variable X (k) is called the k-th order statistic. Then we have the following theorem Theorem. The pdf of the k-th order statistic X (k) is given by f k (x) = n! (k )!(n k)! f(x)fk (x)[ F(x)] n k. Beta distribution. When X,...,X n be iid uniform random variables on [0,θ], the pdf of X (k) /θ becomes n! f k (x) = (k )!(n k)! xk ( x) n k, 0 x, which is known as the beta distribution with parameters α = k and β = n k+. Uniform on [0,θ] Beta(k,n+ k) f(x) = θ, 0 x θ E[X] = θ/2, Var(X) = θ 2 /2 X (k) /θ f(x) = Γ(n+) Γ(k)Γ(n+ k) xk ( x) n k, 0 < x < E[X] = k k(n+ k), Var(X) = n+ (n+) 2 (n+2) Assignments.. Suppose that X has the pdf f(x) = (x+2)/8, 2 x 4. Calculate E[X], E[(X +2) 3 ], and E[6X 2(X +2) 3 ]. 2. Let f(x) = 6x( x),0 x, be a pdf for X. Find the mean and the variance. 3. Complete the following calculations using Interactive Toolbox at (a) If X N(75,00), calculate P(X 60) and P(70 X 00). (b) If X N(µ,σ 2 ), find b so that P( b < (X µ)/σ < b) = Suppose that X and Y have the joint density f(x,y) = 2e x y, 0 < x < y <. Find the joint density for U = 2X and V = Y X. 5. Suppose that X and Y have the joint density f(x,y) = 8xy, 0 < x < y <. Find the joint density for U = X/Y and V = Y. 6. LetX (k), k =,2,3,4, betheorderstatisticsofarandomsampleofsize4fromthepdff(x) = e x, 0 < x <. Find P(X (4) > 3). Lecture Note Page 6

7 Summary of discrete random variables. (a) X is the number of Bernoulli trials until the first success occurs. (b) X is the number of Bernoulli trials until the rth success occurs. If X,,X r are independent geometric random variables with parameter p, then X = r k= X k becomes a negative binomial random variable with parameter (r, p). (c) r =. (d) X is the number of successes in n Bernoulli trials. If X,,X n are independent Bernoulli trials, then X = n k= X k becomes a binomial random variable with (n,p). (e) If X and X 2 are independent binomial random variables with respective parameters (n,p) and (n 2,p), then X = X +X 2 is also a binomial random variable with (n +n 2,p). (f) Poisson approximation is used for binomial distribution by letting n and p 0 while λ = np. (g) If X and X 2 are independent Poisson random variables with respective parameters λ and λ 2, then X = X +X 2 is also a Poisson random variable with λ +λ 2. Bernoulli trial (a) P(X = ) = p and P(X = 0) = q := p E[X] = p, Var(X) = pq The probability p of success and the probability q := p of failure (d) (e) Geometric(p) P(X = i) = q i p, i =,2,... E[X] = p, Var(X) = q p 2 Binomial(n, p) ( ) n P(X = i) = p i q n i, i = 0,...,n i E[X] = np, Var(X) = npq (b) (c) (f) Negative binomial(r, p) ( ) i P(X = i) = p r q i r, i = r,r+,... r E[X] = r rq, Var(X) = p p 2 Poisson(λ) P(X = i) = e λλi i!, i = 0,,... E[X] = λ, Var(X) = λ (g) Lecture Note Page 7

8 Summary of continuous random variables. (a) If X and X 2 are independent normal random variables with respective parameters (µ,σ 2) and (µ 2,σ2 2), then X = X +X 2 is a normal random variable with (µ +µ 2,σ 2 +σ2 2 ). m ( ) (b) If X,,X m are iid normal random variables with parameter (µ,σ 2 Xk µ 2 ), then X = σ is a chi-square (χ 2 ) random variable with m degrees of freedom. (c) n = m 2 and λ = 2. (d) IfX,...,X n areindependentexponentialrandomvariableswithrespectiveparametersλ,,λ n, then X = min{x,...,x n } is an exponential random variable with parameter n λ i. (e) If X,,X n are iid exponential random variables with parameter λ, then X = gamma random variable with (n, λ). (f) n =. k= n X k is a (g) If X and X 2 are independent random variables having gamma distributions with respective parameters (n,λ) and (n 2,λ), then X = X +X 2 is a gamma random variable with parameter (n +n 2,λ). k= (d) Normal(µ,σ 2 ) f(x) = 2πσ e (x µ)2 2σ 2, < x < E[X] = µ, Var(X) = σ 2 Exponential(λ) f(x) = λe λx x 0 E[X] = λ, Var(X) = λ 2 (a) (b) (c) (e) (f) Gamma(n, λ) Chi-square (with m degrees of freedom) f(x) = 2 e x 2( x 2 )m 2 Γ( m 2 ), x 0 E[X] = m, Var(X) = 2m f(x) = λe λx (λx) n, x 0 Γ(n) E[X] = n λ, Var(X) = n λ 2 (g) Γ(n) = (n )!, when n is a positive integer. Lecture Note Page 8