1 EE 4TM4: Digital Commuicatios II Probability Theory I. RANDOM VARIABLES A radom variable is a real-valued fuctio defied o the sample space. Example: Suppose that our experimet cosists of tossig two fair cois. Lettig Y deote the umber of heads appearig, the Y is a radom variable takig o oe of the values,1,2 with respective probabilities P {Y } P {(T, T )} 1 4, P {Y 1} P {(T, H), (H, T )} 1 2, P {Y 2} P {(H, H)} 1 4. Example: Suppose that we toss a coi havig a probability p of comig up heads, util the first head appears. Lettig N deote the umber of flips required, the assumig that the outcome of successive flips are idepedet, N is a radom variable takig o oe of the values 1, 2, 3,, with respective probabilities Note that P {N 1} P {H} p, P {N 2} P {(T, H)} (1 p)p, P {N 3} P {(T, T, H)} (1 p) 2 p,. P {N } P {(T, T,, T, H)} (1 p) 1 p, 1. P {N } p (1 p) 1 p 1 (1 p) 1. 1 1 Radom variables that take o either a fiite or a coutable umber of possible values are called discrete. Radom variables that take o a cotiuum of possible values are kow as cotiuous radom variables. The cumulative distributio fuctio (cdf) (or more simply the distributio fuctio) F ( ) of the radom variable X is defied for ay real umber b, < b <, by F (b) P {X b}. I words, F (b) deotes the probability that the radom variable X takes o a value that is less tha or equal to b. Some properties of the cdf F are
2 1) F (b) is a odecreasig fuctio of b, 2) lim b F (b) F ( ) 1, 3) lim b F (b) F (). Note that P {a < X b} F (b) F (a) for all a < b. Also ote that P {X < b} lim h + P {X b h} lim h + F (b h), which does ot ecessarily equal F (b) sice F (b) also icludes the probability that X equals b. A. Discrete Radom Variables For a discrete radom variable X, we defie the probability mass fuctio p(a) of X by p(a) P {X a}. The probability mass fuctio p(a) is positive for at most a coutable umber of values of a. That is, if X must assume oe of the values x 1, x 2,, the p(x i ) >, i 1, 2,, p(x) for all other values of x. Sice X must take o oe of the values x i, we have i1 p(x i) 1. The cumulative distributio fuctio F ca be expressed i terms of p(a) by F (a) x i a p(x i). For istace, suppose X has a probability mass fuctio give by p(1) 1 2, p(2) 1 3, p(3) 1 6, the, the cumulative distributio fuctio of F of X is give by F (a) for a < 1, F (a) 1 2 for 1 a < 2, F (a) 5 6 for 2 a < 3, F (a) 1 for 3 a. The Beroulli Radom Variable Suppose that a trial whose outcome ca be classified as either a success or as a failure is performed. If we let X equal 1 if the outcome is a success ad if it is failure, the the probability mass fuctio of X is give by p() P {X } 1 p, p(1) P {X 1} p, where p, p 1, is the probability that the trial is a success. The radom variable X is said to be a Beroulli radom variable. The Biomial Radom Variable Suppose that idepedet trials, each of which results i a success with probability p ad i a failure with probability 1 p, are to be performed. If X represets the umber of successes that occur i the trials, the X is said to be a biomial radom variable with parameters (, p). The probability mass fuctio of a biomial radom variable havig parameters (, p) is give by ( ) p(i) p i (1 p) i, i, 1,,, i where ( ) i! ( i)!i!. Note that, by the biomial theorem, the probabilities sum to oe, that is, ( ) p(i) p i (1 p) i (p + (1 p)) 1. i i The Geometric Radom Variable i Suppose that idepedet trials, each havig probability p of beig a success, are performed util a success occurs. If we let X be the umber of trials required util the first success, the X is said to be a geometric
3 radom variable with parameter p. Its probability mass fuctio is give by Note that The Poisso Radom Variable p() P {X } (1 p) 1 p, 1, 2,. p() p (1 p) 1 1. 1 1 A radom variable X, takig o oe of the values, 1, 2,, is said to be a Poisso radom variable with parameter λ, if for some λ >, Note that p(i) P {X i} e p(i) e λ i i λ λi, i, 1,. i! λ i i! e λ e λ 1. A importat property of the Poisso radom variable is that it may be used to approximate a biomial radom variable whe the biomial parameter is large ad p is small. To see this, suppose that X is a biomial radom variable with parameters (, p), ad let λ p. The Now, for large ad p small! P {X i} ( i)!i! pi (1 p) i! ( i)!i! ( λ )i (1 λ ) i (1 λ ) e λ, Hece, for large ad p small, B. Cotiuous Radom Variables ( 1) ( i + 1) λ i (1 λ/) i i! (1 λ/) i. ( 1) ( i + 1) i 1, (1 λ )i 1. λ λi P {X i} e i!. We say that X is a cotiuous radom variable if there exists a oegative fuctio f(x), defied for all real x (, ), havig the property that for ay set B of real umbers P {X B} f(x)dx. The fuctio f(x) is called the probability desity fuctio of the radom variable X. Note that B 1 P {X (, )} P {a X b} P {X a} b a b a f(x)dx, f(x)dx. f(x)dx,
4 Somethig is of probability zero does ot mea it s impossible. The relatioship betwee the cumulative distributio F ( ) ad the probability desity f( ) is expressed by Differetiatig both sides of the precedig yields The Uiform Radom Variable F (a) P {X (, a]} d F (a) f(a). da a f(x)dx. A radom variable is said to be uiformly distributed over the iterval (, 1) if its probability desity fuctio is give by f(x) 1 for < x < 1 ad f(x) otherwise. Note that P {a X b} b a f(x)dx b a. I geeral, we say that X is a uiform radom variable o the iterval (α, β) if its probability desity fuctio is give by f(x) 1 β α Expoetial Radom Variables if α < x < β ad f(x) otherwise. A cotiuous radom variable whose probability desity fuctio is give, for some λ >, by f(x) λe λx if x ad f(x) if x < is said to be a expoetial radom variable with parameter λ. Note that the cumulative distributio fuctio F is give by F (a) I particular, F ( ) λe λx dx 1. Normal Radom Variables a λe λx dx 1 e λa, a. We say that X is a ormal radom variable (or simply that X is ormally distributed) with parameters µ ad σ 2 if the desity of X is give by f(x) 1 2πσ e (x µ)2 /2σ 2, < x <. A importat fact about ormal radom variables is that if X is ormally distributed with parameters µ ad σ 2 the Y αx + β is ormally distributed with parameters αµ + β ad α 2 σ 2. To prove this, suppose first that α > ad ote that F Y ( ), the cumulative distributio fuctio of the radom variable Y, is give by F Y (a) P {Y a} P {αx + β a} P {X a β α } F X ( a β α ) (a β)/α a 1 2πσ e (x µ)2 /2σ 2 dx 1 2πασ exp( (y (αµ + β))2 2α 2 σ 2 )dy,
5 where the last equality is obtaied by the chage i variables y αx + β. However, sice F Y (a) a f Y (y)dv, it follows that the probability desity fuctio f Y ( ) is give by f Y (y) 1 (y (αµ + β))2 exp( 2πασ 2α 2 σ 2 ), < y <. Hece, Y is ormally distributed with parameters αµ + β ad (ασ) 2. A similar result is also true whe α <. Oe implicatio of the precedig result is that if X is ormally distributed with parameters µ ad σ 2 the Y (X µ)/σ is ormally distributed with parameters ad 1. Such a radom variable Y is said to be have the stadard or uit ormal distributio. Fially, we shall verify that Note that ( ) 2 e x2 /2 dx 1 2π e x2 /2 dx 1. e x2 /2 dx e y2 /2 dy e (x2 +y 2 )/2 dxdy. Let x r cos θ ad y r si θ. The Jacobia determiat is give by x x J r θ cos θ r si θ si θ r cos θ r cos2 θ + r si 2 θ r. Therefore, y r y θ e (x2 +y 2 )/2 dxdy C. Expectatio ad Variace of a Radom Variable 2π e r2 /2 rdrdθ 2π. Expectatio of a radom variable X: E[X] x xp(x) (discrete) ad E[X] xf(x)dx (cotiuous). Expectatio of a Beroulli Radom Variable Expectatio of a Biomial Radom Variable E[X] ip(i) E[X] (1 p) + 1(p) p. i ( i i i i1 i1 ) p i (1 p) i i! ( i)!i! pi (1 p) i! ( i)!(i 1)! pi (1 p) i
6 ( 1)! p ( i)!(i 1)! pi 1 (1 p) i i1 1 ( ) 1 p p k (1 p) 1 k k k p[p + (1 p)] 1 p, where the secod from the last equality follows by lettig k i 1. Note that this result ca also be obtaied by writig the biomial radom variable as the sum of idepedet Beroulli radom variables. Expectatio of a Geometric Radom Variable where q 1 p. Expectatio of a Poisso Radom Variable E[X] p p p(1 p) 1 1 1 1 q 1 d dq (q ) p d ( q ) dq 1 p d ( q ) dq 1 q p (1 q) 2 1 p, E[X] ie λ λ i i i1 i! e λ λ i (i 1)! λe λ i1 λe λ k λe λ e λ λ. λ i 1 (i 1)! λ k k!
7 Expectatio of a Uiform Radom Variable E[X] Expectatio of a Expoetial Radom Variable β α x β α dx β2 α 2 2(β α) β + α. 2 E[X] Expectatio of a Normal Radom Variable xλe λx dx xe λx + e λx dx e λx λ 1 λ. E[X] 1 2πσ xe (x µ)2 /2σ 2 dx 1 (x µ)e (x µ)2 /2σ 2 dx + µ 2πσ 1 2πσ µ. ye y2 /2σ 2 dy + µ Propositio (Expectatio of a Fuctio of a Radom Variable): (a) If X is a discrete radom variable with probability mass fuctio p(x), the for ay real-valued fuctio g, E[g(X)] x g(x)p(x). (b) If X is a cotiuous radom variable with probability desity fuctio f(x), the for ay real-valued fuctio g, E[g(X)] g(x)f(x)dx. Corollary: If a ad b are costats, the E[aX + b] ae[x] + b. Variace of a radom variable X: Var(X) E[(X E[X]) 2 ]. A useful idetity: Var(X) E[X 2 ] (E[X]) 2.
8 Variace of a Beroulli Radom Variable Variace of a Normal Radom Variable Var(X) E[X 2 ] (E[X]) 2 Var(X) E[(X µ) 2 ] p p 2. 1 (x µ) 2 e (x µ)2 /2σ 2 dx 2πσ σ2 2π σ2 y 2 e y2 /2 dy ( ye y2 /2 + 2π σ2 2π σ 2. e y2 /2 dy ) e y2 /2 dy D. Joitly Distributed Radom Variables We defie, for ay two radom variables X ad Y, the joit cumulative probability distributio fuctio X ad Y by F (a, b) P {X a, Y b}, < a, b <. I the case where X ad Y are both discrete radom variables, it is coveiet to defie the joit probability mass fuctio of X ad Y by p(x, y) P {X x, Y y}. Let f(x, y) be the joit probability desity fuctio of X ad Y. We have P {X A, Y B} f(x, y)dxdy. Because differetiatio yields A importat idetity: F (a, b) P {X a, Y b} B A a b d 2 F (a, b) f(a, b). dadb f(x, y)dydx, E[a 1 X 1 + a 2 X 2 + + a X ] a 1 E[X 1 ] + a 2 E[X 2 ] + + a E[X ].
9 The radom variables X ad Y are said to be idepedet if, for all a, b, P {X a, Y b} P {X a}p {Y b}. I terms of the joit distributio fuctio F of X ad Y, we have that X ad Y are idepedet if F (a, b) F X (a)f Y (b) for all a, b. Whe X ad Y are discrete, the coditio of idepedece reduces to p(x, y) p X (x)p Y (y) while if X ad Y are joitly cotiuous, idepedece reduces to f(x, y) f X (x)f Y (y). Oe ca exted the defiitio of idepedece to multiple radom variables. For example, X, Y, Z are idepedet if P {X a, Y b, Z c} P {X a}p {Y b}p {Z c} for a, b, c. Propositio: If X ad Y are idepedet, the for ay fuctios h ad g E[g(X)h(Y )] E[g(X)]E[h(Y )]. The covariace of ay two radom variables X ad Y, deoted by Cov(X, Y ), is defied by Cov(X, Y ) E[(X E[X])(Y E[Y ])] E[XY ] E[Y ]E[X] E[X]E[Y ] + E[X]E[Y ] E[XY ] E[X]E[Y ]. Note that if X ad Y are idepedet, the Cov(X, Y ). However, Cov(X, Y ) does ot X ad Y are idepedet. For example, P {X 1} P {X 1} 1/2, P {Y 1 X 1} P {Y 1 X 1} P {Y 2 X 1} P {Y 2 X 1}. It is clear that X ad Y are ot idepedet, but it ca be verified that Cov(X, Y ). E. Momet Geeratig Fuctios The momet geeratig fuctio ϕ(t) of the radom variable X is defied for all values t by ϕ(t) E[e tx ] e tx p(x), x etx f(x)dx, if X is discrete if X is cotiuous We call ϕ(t) the momet geeratig fuctio because all of the momets of X ca be obtaied by successively differetiatig ϕ(t). For example, ϕ (t) d dt E[etX ] [ d E dt etx] E[Xe tx ].
1 Hece, ϕ () E[X]. Similarly, ϕ (t) d dt ϕ (t) d dt E[XetX ] d ] E[ dt (XetX ) E[X 2 e tx ]. Ad so ϕ () E[X 2 ]. I geeral, the th derivative of ϕ(t) evaluated at t equals E[X ], that is, ϕ () E[X ], 1. Note that if ϕ(t) is a aalytic fuctio, the ϕ(t) ϕ() + ϕ ()t + ϕ ()t 2 /2 +, where ϕ() 1, ad ϕ () () E[X ] for 1. Therefore, kowig the momets of X, we ca recostruct ϕ(t) ad further recostruct the distributio of X. The biomial Distributio with Parameters ad p ϕ(t) E[e tx ] ( ) e tk p k (1 p) k k k ( ) (pe t ) k (1 p) k k k (pe t + 1 p). Hece, ϕ (t) (pe t + 1 p) 1 pe t ad so E[X] ϕ () p. Differetiatig a secod time yields ϕ (t) ( 1)(pe t + 1 p) 2 (pe t ) 2 + (pe t + 1 p) 1 pe t ad so E[X 2 ] ϕ () ( 1)p 2 + p. Thus, the variace of X is give by The Poisso Distributio with Mea λ Var(X) E[X 2 ] (E[X]) 2 ( 1)p 2 + p 2 p 2 p(1 p).
11 ϕ(t) E[e tx ] e t e λ λ! e λ (λe t )! e λ e λet exp{λ(e t 1)}. Differetiatig yields ϕ (t) λe t exp{λ(e t 1)}, ϕ (t) (λe t ) 2 exp{λ(e t 1)} + λe t exp{λ(e t 1)}. ad so E[X] ϕ () λ, E[X 2 ] ϕ () λ 2 + λ, Var(X) E[X 2 ] (E[X]) 2 λ. The Expoetial Distributio with Parameter λ Differetiatig of ϕ(t) yields Hece ϕ(t) E[e tx ] λ λ λ t e tx e λx dx e (λ t)x dx for t < λ. ϕ λ (t) (λ t) 2 ϕ (t) 2λ (λ t) 3. The variace of X is thus give by E[X] ϕ () 1 λ, E[X2 ] ϕ () 2 λ 2. Var(X) E[X 2 ] (E[X]) 2 1 λ 2.
12 The Normal Distributio with Parameters µ ad σ 2 The momet geeratig fuctio of a stadard ormal radom variable Z is obtaied as follows. E[e tz ] 1 2π 1 2π e t2 /2 1 2π e t2 /2. e tx e x2 /2 dx e (x2 2tx)/2 dx e (x t)2 /2 dx If Z is a stadard ormal, the X σz + µ is ormal with parameters µ ad σ 2 ; therefore, By differetiatig we obtai ad so implyig that ϕ(t) E[e tx ] E[e t(σz+µ) ] e tµ E[e tσz ] exp( σ2 t 2 + µt). 2 ϕ (t) (µ + tσ 2 ) exp( σ2 t 2 + µt), 2 ϕ (t) (µ + tσ 2 ) 2 exp( σ2 t 2 + µt) + σ 2 exp( σ2 t 2 + µt), 2 2 E[X] ϕ () µ, E[X 2 ] ϕ () µ 2 + σ 2, Var(X) E[X 2 ] (E[X]) 2 σ 2. A importat property of momet geeratig fuctios is that the momet geeratig fuctio of the sum of idepedet radom variables is just the product of the idividual momet geeratig fuctios. To see this, suppose that X ad Y are idepedet ad have momet geeratig fuctio ϕ X (t) ad ϕ Y (t), respectively. The ϕ X+Y (t), the momet geeratig fuctio of X + Y, is give by ϕ X+Y (t) E[e t(x+y ) ] E[e tx e ty ] E[e tx ]E[e ty ] ϕ X (t)ϕ Y (t). Aother importat result is that the momet geeratig fuctio uiquely determies the distributio. That is, there exists a oe-to-oe correspodece betwee the momet geeratig fuctio ad the distributio fuctio of a radom variable.
13 F. Limit Theorems Markov Iequality: If X is a radom variable that takes oly oegative values, the for ay value a > Proof: Note that ad the result is prove. As a corollary, we obtai the followig. P {X a} E[X] a. E[X] E[X1 {X A} ] E[a1 {X A} ] ap {X a}, Chebyshev s Iequality: If X is a radom variable with mea µ ad variace σ 2, the, for ay value k >, P { X µ k} σ2 k 2. Proof: Sice (X µ) 2 is a oegative radom variable, we ca apply Markov s iequality (with a k 2 ) to obtai P {(X µ) 2 k 2 } E[(X µ)2 ] k 2. But sice (X µ) 2 k 2 if ad oly if X µ k, the precedig is equivalet to ad the proof is complete. P { X µ k} E[(X µ)2 ] k 2 σ2 k 2, Weak Law of Large Numbers: Let X 1, X 2, be a sequece of idepedet radom variables havig a commo distributio, ad let E[X i ] µ. The for ay ϵ > { P X 1 + X 2 + + X } µ > ϵ as. Let Y X1+X2+ +X Chebyshev s iequality µ. Note that E[Y ] ad Var(Y ) σ2, where σ2 Var(X i ). Therefore, by P { Y > ϵ} σ2 as. ϵ2 Note that i this proof we actually oly eed X 1, X 2, to be ucorrelated. Strog Law of Large Numbers: Let X 1, X 2, be a sequece of idepedet radom variables havig a commo distributio, ad let E[X i ] µ. The, with probability 1, X 1 + X 2 + + X µ as. Cetral Limit Theorem: Let X 1, X 2, be a sequece of idepedet, idetically distributed radom variables, each with mea µ ad variace σ 2. The the distributio of X 1 + X 2 + + X µ σ
14 teds to the stadard ormal as. That is, { X1 + X 2 + + X µ P σ as. } a 1 a e x2 /2 dx 2π We ow preset a heuristic proof of the cetral limit theorem. Suppose first that the X i have mea ad variace 1, ad let E[e tx ] deote their commo momet geeratig fuctio. The, the momet geeratig fuctio of X 1 + +X is [ { ( X1 + + X )}] E exp t E[e tx1/ e tx2/ e tx/ ] (E[e tx/ ]). Now, for large, we obtai from the Taylor series expasio of e y that Takig expectatios shows that whe is large e tx/ 1 + tx + t2 X 2 2. E[e tx/ ] 1 + te[x] + t2 E[X 2 ] 2 1 + t2 2 because E[X], E[X2 ] 1. Therefore, we obtai that whe is large [ { ( X1 + + X )}] E exp t (1 + t2 2 ). Whe goes to the approximatio ca be show to become exact ad we have [ { ( lim E X1 + + X )}] exp t e t2 /2. Thus, the momet geeratig fuctio of X 1+ +X coverges to the momet geeratig fuctio of a (stadard) ormal radom variable with mea ad variace 1. Usig this, it ca be prove that the distributio fuctio of the radom variable X 1+ +X coverges to the stadard ormal distributio fuctio Φ. Whe the X i have mea µ ad variace σ 2, the radom variables X i µ σ precedig shows that which proves the cetral limit theorem. { X1 µ + X 2 µ + + X µ } P σ a Φ(a) have mea ad variace 1. Thus, the REFERENCES [1] S. M. Ross, Itroductio to Probablity Models. Teth Editio. Academic Press, 29.