PROBABILITY AND RANDOM VARIABLES

Transcription

1 CHAPTER PROBABILIT AND RANDOM VARIABLES. Introduction.. Basics of Probability.3.. Terminology.3.. Probability of an event.5.3 Random Variables.3.3. Distribution function.3.3. Probability density function Joint distribution and density function Conditional density..3.5 Statistical independence.3.4 Transformation of Variables.7.4. Functions of one random variable.8.4. Functions of two random variables.37.5 Statistical Averages Variance Covariance.53.6 Some Useful Probability Models Discrete random variables Continuous random variables.63 Appendix A.: Proof of Eq Appendix A.: Q Function Table.78 Appendix A.3: Proof that ( σ, ) N m is a valid PDF.80 References.8

2 CHAPTER Probability and Random Variables. Introduction At the start of Sec..., we had indicated that one of the possible ways of classifying the signals is: deterministic or random. By random we mean unpredictable; that is, in the case of a random signal, we cannot with certainty predict its future value, even if the entire past history of the signal is known. If the signal is of the deterministic type, no such uncertainty exists. Consider the signal x( t) A cos( f t ) = π + θ. If A, θ and f are known, then (we are assuming them to be constants) we know the value of x () t for all t. ( A, θ and f can be calculated by observing the signal over a short period of time). Now, assume that x ( t ) is the output of an oscillator with very poor frequency stability and calibration. Though, it was set to produce a sinusoid of frequency f = f, frequency actually put out maybe f ' ' where f ( f f ) ±. Even this value may not remain constant and could vary with time. Then, observing the output of such a source over a long period of time would not be of much use in predicting the future values. We say that the source output varies in a random manner. Another example of a random signal is the voltage at the terminals of a receiving antenna of a radio communication scheme. Even if the transmitted.

3 (radio) signal is from a highly stable source, the voltage at the terminals of a receiving antenna varies in an unpredictable fashion. This is because the conditions of propagation of the radio waves are not under our control. But randomness is the essence of communication. Communication theory involves the assumption that the transmitter is connected to a source, whose output, the receiver is not able to predict with certainty. If the students know ahead of time what is the teacher (source + transmitter) is going to say (and what jokes he is going to crack), then there is no need for the students (the receivers) to attend the class! Although less obvious, it is also true that there is no communication problem unless the transmitted signal is disturbed during propagation or reception by unwanted (random) signals, usually termed as noise and interference. (We shall take up the statistical characterization of noise in Chapter 3.) However, quite a few random signals, though their exact behavior is unpredictable, do exhibit statistical regularity. Consider again the reception of radio signals propagating through the atmosphere. Though it would be difficult to know the exact value of the voltage at the terminals of the receiving antenna at any given instant, we do find that the average values of the antenna output over two successive one minute intervals do not differ significantly. If the conditions of propagation do not change very much, it would be true of any two averages (over one minute) even if they are well spaced out in time. Consider even a simpler experiment, namely, that of tossing an unbiased coin (by a person without any magical powers). It is true that we do not know in advance whether the outcome on a particular toss would be a head or tail (otherwise, we stop tossing the coin at the start of a cricket match!). But, we know for sure that in a long sequence of tosses, about half of the outcomes would be heads (If this does not happen, we suspect either the coin or tosser (or both!))..

4 Statistical regularity of averages is an experimentally verifiable phenomenon in many cases involving random quantities. Hence, we are tempted to develop mathematical tools for the analysis and quantitative characterization of random signals. To be able to analyze random signals, we need to understand random variables. The resulting mathematical topics are: probability theory, random variables and random (stochastic) processes. In this chapter, we shall develop the probabilistic characterization of random variables. In chapter 3, we shall extend these concepts to the characterization of random processes.. Basics of Probability We shall introduce some of the basic concepts of probability theory by defining some terminology relating to random experiments (i.e., experiments whose outcomes are not predictable).... Terminology Def..: Outcome The end result of an experiment. For example, if the experiment consists of throwing a die, the outcome would be anyone of the six faces, F,..., F 6 Def..: Random experiment An experiment whose outcomes are not known in advance. (e.g. tossing a coin, throwing a die, measuring the noise voltage at the terminals of a resistor etc.) Def..3: Random event A random event is an outcome or set of outcomes of a random experiment that share a common attribute. For example, considering the experiment of throwing a die, an event could be the 'face F ' or 'even indexed faces' ( F, F 4, F 6 ). We denote the events by upper case letters such as A, B or A, A.3

5 Def..4: Sample space The sample space of a random experiment is a mathematical abstraction used to represent all possible outcomes of the experiment. We denote the sample space by S. Each outcome of the experiment is represented by a point in S and is called a sample point. We use s (with or without a subscript), to denote a sample point. An event on the sample space is represented by an appropriate collection of sample point(s). Def..5: Mutually exclusive (disjoint) events Two events A and B are said to be mutually exclusive if they have no common elements (or outcomes).hence if A and B are mutually exclusive, they cannot occur together. Def..6: Union of events The union of two events A and B, denoted A ( A B) B, {also written as + or ( A or B )} is the set of all outcomes which belong to A or B or both. This concept can be generalized to the union of more than two events. Def..7: Intersection of events The intersection of two events, A and B, is the set of all outcomes which belong to A as well as B. The intersection of A and B is denoted by ( A B) or simply ( AB ). The intersection of A and B is also referred to as a joint event A and B. This concept can be generalized to the case of intersection of three or more events. Def..8: Occurrence of an event An event A of a random experiment is said to have occurred if the experiment terminates in an outcome that belongs to A..4

6 Def..9: Complement of an event The complement of an event A, denoted by A is the event containing all points in S but not in A. Def..0: Null event The null event, denoted φ, is an event with no sample points. Thus φ =S (note that if A and B are disjoint events, then AB = φ and vice versa)... Probability of an Event The probability of an event has been defined in several ways. Two of the most popular definitions are: i) the relative frequency definition, and ii) the classical definition. Def..: The relative frequency definition: Suppose that a random experiment is repeated n times. If the event A occurs n A times, then the probability of A, denoted by P( A ), is defined as n = n n lim A P A (.) n A n represents the fraction of occurrence of A in n trials. For small values of n, it is likely that na n will fluctuate quite badly. But na as n becomes larger and larger, we expect, n to tend to a definite limiting value. For example, let the experiment be that of tossing a coin and A the event na 'outcome of a toss is Head'. If n is the order of 00, n may not deviate from.5

7 expect by more than, say ten percent and as n becomes larger and larger, we n A n to converge to. Def..: The classical definition: The relative frequency definition given above has empirical flavor. In the classical approach, the probability of the event A is found without experimentation. This is done by counting the total number N of the possible outcomes of the experiment. If occurrence of the event A, then A P A N A of those outcomes are favorable to the N = (.) N where it is assumed that all outcomes are equally likely! Whatever may the definition of probability, we require the probability measure (to the various events on the sample space) to obey the following postulates or axioms: P) P( A) 0 (.3a) P) P ( S ) = (.3b) P3) ( AB ) =φ, then P( A B) P( A) P( B) + = + (.3c) (Note that in Eq..3(c), the symbol + is used to mean two different things; namely, to denote the union of A and B and to denote the addition of two real numbers). Using Eq..3, it is possible for us to derive some additional relationships: i) If AB φ, then P( A + B) = P( A) + P( B) P( AB) (.4) ii) Let A, A,..., A n be random events such that: a) AA i j =φ, for i j and (.5a).6

8 b) A + A An = S. (.5b) Then, P( A) P( AA ) P( AA ) P( AA ) = n (.6) where A is any event on the sample space. Note: A, A,, A are said to be mutually exclusive (Eq..5a) and exhaustive (Eq..5b). n iii) P( A) P( A) = (.7) The derivation of Eq..4,.6 and.7 is left as an exercise. A very useful concept in probability theory is that of conditional probability, denoted P( B A ); it represents the probability of B occurring, given that A has occurred. In a real world random experiment, it is quite likely that the occurrence of the event B is very much influenced by the occurrence of the event A. To give a simple example, let a bowl contain 3 resistors and capacitor. The occurrence of the event 'the capacitor on the second draw' is very much dependent on what has been drawn at the first instant. Such dependencies between the events is brought out using the notion of conditional probability. The conditional probability P( B A ) can be written in terms of the joint probability P( AB ) and the probability of the event P( A ). This relation can be arrived at by using either the relative frequency definition of probability or the classical definition. Using the former, we have nab = n n lim P AB n = n n lim A P A.7

9 where n AB is the number of times AB occurs in n repetitions of the experiment. As P( B A ) refers to the probability of B occurring, given that A has occurred, we have Def.3: Conditional Probability n P( B A) = lim AB n n A nab P( AB) = lim n =, P( A) 0 n na P( A) n or P( AB) = P( B A) P( A) (.8a) Interchanging the role of A and B, we have P AB P( A B) =, P( B) 0 (.8b) P B Eq..8(a) and.8(b) can be written as ( ) ( ) P AB = P B A P A = P B P A B (.9) In view of Eq..9, we can also write Eq..8(a) as Similarly ( ) P B A P( B) P( A B) =, P( A) P( A) P( B A) ( ) =, P( B) P A B P A 0 (.0a) P B 0 (.0b) Eq..0(a) or.0(b) is one form of Bayes rule or Bayes theorem. Eq..9 expresses the probability of joint event AB in terms of conditional probability, say P( B A ) and the (unconditional) probability P( A ). Similar relation can be derived for the joint probability of a joint event involving the intersection of three or more events. For example P( ABC ) can be written as.8

10 = P( AB) P( C AB) P ABC ( ) ( ) = P A P B A P C AB (.) Exercise. Let A, A,..., A n be n mutually exclusive and exhaustive events and B is another event defined on the same space. Show that ( j B) P A = P( B Aj) P( Aj) n P( B Aj ) P( Aj ) i = (.) Eq.. represents another form of Bayes theorem. Another useful probabilistic concept is that of statistical independence. Suppose the events A and B are such that ( ) P( B) P B A = (.3) That is, knowledge of occurrence of A tells no more about the probability of occurrence B than we knew without that knowledge. Then, the events A and B are said to be statistically independent. Alternatively, if A and B satisfy the Eq..3, then P( A) P( B) P AB = (.4) Either Eq..3 or.4 can be used to define the statistical independence of two events. Note that if A and B are independent, then = P( A) P( B), whereas if they are disjoint, then 0 P AB P AB =. The notion of statistical independence can be generalized to the case of more than two events. A set of k events A, A,..., A k are said to be statistically independent if and only if (iff) the probability of every intersection of k or fewer events equal the product of the probabilities of its constituents. Thus three events A, B, C are independent when.9

11 = P( A) P( B) P AB = P( A) P( C) P AC = P( B) P( C) P BC and P( ABC) = P( A) P( B) P( C) We shall illustrate some of the concepts introduced above with the help of two examples. Example. Priya (P) and Prasanna (P), after seeing each other for some time (and after a few tiffs) decide to get married, much against the wishes of the parents on both the sides. They agree to meet at the office of registrar of marriages at :30 a.m. on the ensuing Friday (looks like they are not aware of Rahu Kalam or they don t care about it). However, both are somewhat lacking in punctuality and their arrival times are equally likely to be anywhere in the interval to hrs on that day. Also arrival of one person is independent of the other. Unfortunately, both are also very short tempered and will wait only 0 min. before leaving in a huff never to meet again. a) Picture the sample space b) Let the event A stand for P and P meet. Mark this event on the sample space. c) Find the probability that the lovebirds will get married and (hopefully) will live happily ever after. a) The sample space is the rectangle, shown in Fig..(a)..0

12 Fig..(a): S of Example. b) The diagonal OP represents the simultaneous arrival of Priya and Prasanna. Assuming that P arrives at : x, meeting between P and P would take place if P arrives within the interval a to b, as shown in the figure. The event A, indicating the possibility of P and P meeting, is shown in Fig..(b). Fig..(b): The event A of Example..

13 c) Probability of marriage Shaded area = = Total area 36 Example.: Let two honest coins, marked and, be tossed together. The four possible outcomes are TT, TH, HT, HH. ( T indicates toss of coin resulting in tails; similarly T etc.) We shall treat that all these outcomes are equally likely; that is the probability of occurrence of any of these four outcomes is. (Treating each of these outcomes as an event, we find that these events 4 are mutually exclusive and exhaustive). Let the event A be 'not HH' and B be the event 'match'. (Match comprises the two outcomes TT, HH). Find P( B A ). Are A and B independent? We know that P( B A) P AB =. P A AB is the event 'not HH' and 'match'; i.e., it represents the outcome TT. Hence P( AB ) =. The event A comprises of the outcomes TT, TH and 4 HT ; therefore, 3 P( A ) = 4 P( B A ) = 4 = Intuitively, the result P( B A ) = is satisfying because, given 'not HH the 3 toss would have resulted in anyone of the three other outcomes which can be.

14 treated to be equally likely, namely 3. This implies that the outcome TT given 'not HH', has a probability of 3. As P( B ) = and ( ) P B A =, A and B are dependent events. 3.3 Random Variables Let us introduce a transformation or function, say, whose domain is the sample space (of a random experiment) and whose range is in the real line; that is, to each s i S, assigns a real number, ( s ), as shown in Fig... i Fig..: A mapping from S to the real line. The figure shows the transformation of a few individual sample points as well as the transformation of the event A, which falls on the real line segment [, ] a a..3. Distribution function: Taking a specific case, let the random experiment be that of throwing a die. The six faces of the die can be treated as the six sample points in S ; that is,.3

15 F s, i,,..., 6 i = =. Let ( s ) i i = i. Once the transformation is induced, then the events on the sample space will become transformed into appropriate segments of the real line. Then we can enquire into the probabilities such as or { : < } { : < } P s s a P s b s b { : = } P s s c These and other probabilities can be arrived at, provided we know the Distribution Function of, denoted by F which is given by { } : F x = P s s x That is, F (.5) x is the probability of the event, comprising all those sample points which are transformed by into real numbers less than or equal to x. (Note that, for convenience, we use x as the argument of F. But, it could be any other symbol and we may use F ( α ), F ( a ) etc.) Evidently, F is a function whose domain is the real line and whose range is the interval [ 0, ] ). As an example of a distribution function (also called Cumulative Distribution Function CDF), let S consist of four sample points, s to s 4, with each with sample point representing an event with the probabilities P( s ) =, P( s ) = and P( s ) =. If ( s ) i.5, i,, 3, the distribution function F ( x ), will be as shown in Fig i P s =, 4 = =, then.4

16 Fig..3: An example of a distribution function F satisfies the following properties: i) 0, F x < x < ii) F ( ) = 0 iii) F ( ) = iv) If a b v) If a b { } >, then : F a F b = P s b < s a >, then F ( a) F ( b) The first three properties follow from the fact that F represents the probability and P ( S ) =. Properties iv) and v) follow from the fact { s: ( s) b} { s: b < ( s) a} = { s: ( s) a} F ( 0.5) Referring to the Fig..3, note that F ( x ) = 0 for x < 0.5 whereas =. In other words, there is a discontinuity of 4 4 at the point x = 0.5. In general, there is a discontinuity in F of magnitude P a at a point x = a, if and only if { } P s s a = = P : a (.6).5

17 The properties of the distribution function are summarized by saying that F is monotonically non-decreasing, is continuous to the right at each point x x, and has a step of size P a at point a if and if Eq..6 is satisfied. Functions such as for which distribution functions exist are called Random Variables (RV). In other words, for any real x, { s: ( s) x} should be an event in the sample space with some assigned probability. (The term random variable is somewhat misleading because an RV is a well defined function from S into the real line.) However, every transformation from S into the real line need not be a random variable. For example, let S consist of six sample points, s to s 6. The only events that have been identified on the sample space are: A { s, s }, B { s, s, s } and C { s } = = = and their probabilities are P( A), P( B) and P( C) = = =. We see that the probabilities for the 6 6 various unions and intersections of A, B and C can be obtained. Let the transformation be ( si ) fails to exist because [ : ] 4 = i. Then the distribution function P s < x = P s is not known as s 4 is not an event on the sample space. Let x = a. Consider, with > 0, lim lim P a < s a + = F a + F a 0 0 We intuitively feel that as 0 can be proved to be so. Hence, { s a s a }, the limit of the set : F ( a+ ) F ( a) = 0, where a = lim ( a + ) That is, F x is continuous to the right. + 0 < + is the null set and.6

18 Exercise. Let S be a sample space with six sample points, s to s 6. The events identified on S are the same as above, namely, A { s, s } = {,, } and = { } with P( A), P( B) and P( C) B s s s C s 6 Let be the transformation, s i, i =, =, i = 3,4,5 3, i = 6 =, = = =. 3 6 Show that is a random variable by finding F ( y ). Sketch F y..3. Probability density function Though the CDF is a very fundamental concept, in practice we find it more convenient to deal with Probability Density Function (PDF). The PDF, f defined as the derivative of the CDF; that is or f ( x) df dx ( x) x is = (.7a) x (.7b) F x = f α dα x The distribution function may fail to have a continuous derivative at a point = a for one of the two reasons: i) the slope of the Fx ( x ) is discontinuous at x = a ii) Fx ( x ) has a step discontinuity at x = a The situation is illustrated in Fig..4..7

19 Fig..4: A CDF without a continuous derivative As can be seen from the figure, F x has a discontinuous slope at x = and a step discontinuity at x =. In the first case, we resolve the ambiguity by taking f to be a derivative on the right. (Note that F x is continuous to the right.) The second case is taken care of by introducing the impulse in the probability domain. That is, if there is a discontinuity in F at x = a of magnitude a P, we include an impulse P ( x a) example, for the CDF shown in Fig..3, the PDF will be, a δ in the PDF. For 3 5 f ( x) = δ x x x x δ 8 + δ + δ 8 (.8) P In Eq..8, f x has an impulse of weight 8 at x = as = =. This impulse function cannot be taken as the limiting case of 8 an even function (such as ga x ε ε ) because, lim f x dx = δ x dx 8 6 lim ε 0 ε 0 ε ε.8

20 = ε 0 8 ε However, lim F ( x) f x dx. This ensures,, x < 8 = 3 3, x < 8 Such an impulse is referred to as the left-sided delta function. As F is non-decreasing and F =, we have i) f ( x) 0 (.9a) (.9b) ii) f ( x) dx = Based on the behavior of CDF, a random variable can be classified as: i) continuous (ii) discrete and (iii) mixed. If the CDF, F x, is a continuous function of x for all x, then is a continuous random variable. If F x is a staircase, then corresponds to a discrete variable. We say that is a mixed random variable if F x is discontinuous but not a staircase. Later on in this lesson, we shall discuss some examples of these three types of variables. We can induce more than one transformation on a given sample space. If we induce k such transformations, then we have a set of k co-existing random variables..3.3 Joint distribution and density functions Consider the case of two random variables, say and. They can be characterized by the (two-dimensional) joint distribution function, given by { } F, x, y = P s: s x, s y (.0) As the domain of the random variable is known, it is convenient to denote the variable simply by..9

21 That is, F,, x y is the probability associated with the set of all those sample points such that under, their transformed values will be less than or equal to x and at the same time, under, the transformed values will be less than or equal to y. In other words, (, ) F x y is the probability associated with, the set of all sample points whose transformation does not fall outside the shaded region in the two dimensional (Euclidean) space shown in Fig..5. { } Fig..5: Space of ( s), ( s ) corresponding to F ( x, y ), Looking at the sample space S, let A be the set of all those sample points s S such that ( s) x. Similarly, if B is comprised of all those sample points s S such that ( s) y; then (, ) associated with the event AB. F x y is the probability Properties of the two dimensional distribution function are: i) F, x, y 0, < x <, < y < ii) F ( y) F ( x ), =, = 0,, iii) F,, = iv) F ( y) = F ( y),, v) F ( x ) = F ( x),,.0

22 vi) If x > x and y > y, then (, ) (, ) (, ) F x y F x y F x y,,, We define the two dimensional joint PDF as f, ( x, y) = F, ( x, y) (.a) x y y x F x, y = f α, β dα dβ or,, (.b) The notion of joint CDF and joint PDF can be extended to the case of k random variables, where k 3. Given the joint PDF of random variables and, it is possible to obtain the one dimensional PDFs, f ( x ) and f F ( x, ) = F ( x )., x That is, ( ), (, ) F x = f α β dβ dα x d f x f d d dx y. We know that, ( ) =, ( α, β) β α (.) Eq.. involves the derivative of an integral. Hence, ( x) df f x = = f x β dβ, dx x = x, f x f x y dy or,, = (.3a) f y f x y dx Similarly, = (.3b),, (In the study of several random variables, the statistics of any individual variable is called the marginal. Hence it is common to refer F x as the marginal.

23 distribution function of and f ( x ) as the marginal density function. F f y are similarly referred to.) y and.3.4 Conditional density Given f ( x y ), we know how to obtain f ( x ) or f,, y. We might also be interested in the PDF of given a specific value of y = y. This is called the conditional PDF of, given y y f ( x y ) f ( x, y ) ( y ) =, denoted f ( x y ) and defined as, = (.4) f where it is assumed that f y. Once we understand the meaning of 0 conditional PDF, we might as well drop the subscript on y and denote it by f ( x y ). An analogous relation can be defined for the pair of equations, f ( x y) and f ( y x) f ( x, y) ( y) f y x. That is, we have =, (.5a) f f ( x, y) ( x) =, (.5b) or f ( x, y) f ( x y) f ( y), f = (.5c) = f y x f x (.5d) The function f x y may be thought of as a function of the variable x with variable y arbitrary, but fixed. Accordingly, it satisfies all the requirements of an ordinary PDF; that is, f x y 0 (.6a) f x y dx = (.6b) and.

24 .3.5 Statistical independence In the case of random variables, the definition of statistical independence is somewhat simpler than in the case of events. We call k random variables,,..., k statistically independent iff, the k -dimensional joint PDF factors into the product k i = i ( x ) f (.7) i Hence, two random variables and are statistically independent, iff, f x y = f x f y (.8),, and three random variables,, Z are independent, iff fz,, x, y, z = f x f y fz z (.9) Statistical independence can also be expressed in terms of conditional PDF. Let and be independent. Then, = f x y f x f y,, Making use of Eq..5(c), we have = f x y f y f x f y or f ( x y) f ( x) Similarly, f ( y x) f ( y) = (.30a) = (.30b) Eq..30(a) and.30(b) are alternate expressions for the statistical independence between and. We shall now give a few examples to illustrate the concepts discussed in sec..3. Example.3 A random variable has 0, x < 0 F ( x) = K x, 0 x 0 00 K, x > 0 i) Find the constant K ii) Evaluate P( 5) and P( 5 < 7).3

25 iii) What is f ( x )? F = 00K = K =. 00 i) P x 5 = F 5 = 5 = ii) P 5 < 7 = F 7 F 5 = 0.4 0, x < 0 df ( x) f ( x) = = 0.0 x, 0 x 0 dx 0, x > 0 Note: Specification of f ( x ) or F x is complete only when the algebraic expression as well as the range of is given. Example.4 Consider the random variable defined by the PDF bx, f x = ae < x < where a and b are positive constants. i) Determine the relation between a and b so that f ii) Determine the corresponding F ( x ) iii) Find P[ ] <. x is a PDF. i) As can be seen f ( x) 0 for < x <. In order for f x to represent a legitimate PDF, we require bx bx ae dx = ae dx =. 0 That is, bx ae dx = ; hence b = a. 0 ii) the given PDF can be written as f ( x) bx ae, x < 0 = bx ae, x 0..4

26 For x < 0, we have x b b bx F ( x) = e α dα = e. Consider x > 0. Take a specific value of x = F = f x d x f = x d x But for the problem on hand, = f x dx f x d x bx Therefore, F ( x) = e, x > 0 We can now write the complete expression for the CDF as F ( x) bx e, x < 0 = bx e, x 0 b b iii) F F ( e = e ) Example.5 Let f ( x y),, 0 x y, 0 y, = 0, otherwise Find (a) i) f ( y ) and ii) f ( y ) (b) Are and independent?.5 (a) f ( y x) = f, f ( x, y) ( x).5

27 f x can be obtained from f, by integrating out the unwanted variable y over the appropriate range. The maximum value taken by y is ; in addition, for any given x, y x. Hence, Hence, x f ( x) = dy =, 0 x (i) f ( y ) (ii) f ( y.5) x, y = = 0, otherwise,.5 y = = 0, otherwise 4 b) the dependence between the random variables and is evident from the statement of the problem because given a value of = x, should be greater than or equal to x for the joint PDF to be non zero. Also we see that f y x depends on x whereas if and were to be independent, then f ( y x) f ( y) =..6

28 Exercise.3 For the two random variables and, the following density functions have been given. (Fig..6) Find a) f ( x y ),, Fig..6: PDFs for the exercise.3 b) Show that f ( y) y, 0 y 0 00 = y,0 < y Transformation of Variables The process of communication involves various operations such as modulation, detection, filtering etc. In performing these operations, we typically generate new random variables from the given ones. We will now develop the necessary theory for the statistical characterization of the output random variables, given the transformation and the input random variables..7

29 .4. Functions of one random variable Assume that is the given random variable of the continuous type with the PDF, f ( x ). Let be the new random variable, obtained from by the transformation = g. By this we mean the number ( s ) associated with any sample point s is = g ( s ) s Our interest is to obtain f theorem (Thm..). Theorem. y Let = g. To find f = g x. Denoting its real roots by n we will show that f ( y) y. This can be obtained with the help of the following y, for the specific y, solve the equation x, y = g x =,..., = g x n =,..., f x f xn = (.3) g' x g' x where g' ( x ) is the derivative of g( x ). n Proof: Consider the transformation shown in Fig..7. We see that the equation y = g x has three roots namely, x, x and x 3..8

30 Fig..7: transformation used in the proof of theorem. We know that f ( y) dy P[ y y dy] = < +. Therefore, for any given y, we need to find the set of values x such that y < g x y + d y and the probability that is in this set. As we see from the figure, this set consists of the following intervals: x < x x + d x, x + d x < x x, x < x x + d x where dx > 0, dx3 > 0, but dx < 0. From the above, it follows that [ < + ] = [ < + ] + [ + < ] + P[ x < x + d x ] P y y dy P x x d x P x dx x This probability is the shaded area in Fig..7. [ ] P x < x + dx = f x dx, dx = [ ] P x + dx < x x = f x dx, dx = [ ] P x < x + dx = f x dx, dx = dy g' ( x ) dy g' ( x ) dy g' ( x )

31 We conclude that ( x ) f x f x f x3 f ( y) dy = dy + dy + dy g' x g' g' x 3 (.3) and Eq..3 follows, by canceling dy from the Eq..3. Note that if g( x) = y = constant for every x in the interval ( 0, ) x x, then we have P [ = y] = P( x < x) = F ( x) F ( x) ; that is F discontinuous at y y F x F x =. Hence f ( y ) contains an impulse, y is δ y y of area We shall take up a few examples. Example.6 = g = + a, where a is a constant. Let us find f y. We have g' ( x ) = and x = y a. For a given y, there is a unique x satisfying the above transformation. Hence, = f ( x) g'( x) and as f y dy dy = ( ) f y f y a g' x =, we have Let f ( x) x, x = 0, elsewhere and a = Then, f ( y) y As y x = + =, and x ranges from the interval (, ) as (, 0)., we have the range of y.30

32 Hence f ( y) F ( x ) and y +, y 0 = 0, elsewhere F y are shown in Fig..8. Fig..8: F ( x ) and F y of example.6 As can be seen, the transformation of adding a constant to the given variable simply results in the translation of its PDF. Example.7 Let = b, where b is a constant. Let us find f ( y ). Solving for, we have x. As g' ( x) = b, we have =. Again for a given y, there is a unique b y f y = f b b. Let f ( x) x, 0 x = 0, otherwise and b =, then.3

33 f ( y) f ( x ) and y +, 4 y 0 = 4 0, otherwise f y are sketched in Fig..9. Fig..9: f ( x ) and f y of example.7 Exercise.4 Let = a + b, where a and b are constants. Show that y b f ( y) = f a a. If f ( x ) is as shown in Fig..8, compute and sketch b =. f y for a =, and Example.8, 0 a a = >. Let us find f y..3

34 g' x = ax. If y < 0, then the equation y = ax has no real solution. Hence f ( y ) = 0 for y < 0. If 0 y y, then it has two solutions, x = and a x y =, and Eq..3 yields a Let y y f + f, y 0 f a a ( y) y = a a 0, otherwise a =, and x f x = exp, < x < π (Note that exp( α ) is the same as e α ) Then f ( y) y y = exp + exp y π y exp, y 0 y = π 0, otherwise Sketching of f ( x ) and f y is left as an exercise. Example.9 Consider the half wave rectifier transformation given by 0, 0 =, > 0 a) Let us find the general expression for f ( y ) b) Let f ( x) 3, < x < = 0, otherwise We shall compute and sketch f ( y )..33

35 a) Note that g( ) is a constant (equal to zero) for in the range of (,0). Hence, there is an impulse in f ( y ) at y = 0 whose area is equal to F ( 0). As is nonnegative, f ( y ) = 0 for y < 0. As = for x > 0, we have f ( y) = f ( y) for y > 0. Hence f ( y) ( 0) f y w y + F δ y = 0, otherwise where w( y), y 0 = 0, otherwise b) Specifically, let f ( x) 3, < x < = 0, elsewhere Then, δ ( y), y = f y =, 0 < y 0, otherwise f ( x ) and f y are sketched in Fig..0. Fig..0: f ( x ) and f y for the example.9.34

36 Note that, a continuous RV is transformed into a mixed RV,. Example.0 Let, < 0 = +, 0 a) Let us find the general expression for f ( y ). b) We shall compute and sketch f ( x ) assuming that f that of Example.9. x is the same as a) In this case, assumes only two values, namely ±. Hence the PDF of has only two impulses. Let us write f = ( ) + ( + ) [ 0] [ 0] y as f y P δ y P δ y where P P and P = < b) Taking 3 f x of example.9, we have P = and P =. Fig has the sketches f ( x ) and f y. Fig..: f ( x ) and f y for the example.0 Note that this transformation has converted a continuous random variable into a discrete random variable..35

37 Exercise.5 Let a random variable with the PDF shown in Fig..(a) be the input to a device with the input-output characteristic shown in Fig..(b). Compute and sketch f ( y ). Fig..: (a) Input PDF for the transformation of exercise.5 (b) Input-output transformation Exercise.6 The random variable of exercise.5 is applied as input to the transformation shown in Fig..3. Compute and sketch f ( y )..36

38 We now assume that the random variable is of discrete type taking on the value x k with probability P k. In this case, the RV, = g is also discrete, assuming the value g( x ) k = with probability P k. k If y g( x) k however, y g( x) k = for only one x xk = for x = xk and x xm =, then [ ] [ ] P = y = P = x = P. If =, then [ ] k k k P = y = P + P. k k m Example. Let =. 6 a) If f ( x) = δ ( x i), find 6 i = 3 f y. b) If f ( x) = δ ( x i), find 6 i = f y. a) If takes the values (,,..., 6 ) with probability of, then takes the 6 values,,..., 6 with probability 6 6. That is, f ( y) = δ( x i ). 6 i = b) If, however, takes the values,, 0,,, 3 with probability 6, then takes the values 0,, 4, 9 with probabilities,,, respectively. That is, f ( y) = ( y) ( y 9) ( y ) ( y 4) 6 δ +δ + δ +δ 3.4. Functions of two random variables Given two random variables and (assumed to be continuous type), two new random variables, Z and W are defined using the transformation = (, ) and W = h(, ) Z g.37

39 f Given the above transformation and f ZW,,,, x y, we would like to obtain z w. For this, we require the Jacobian of the transformation, denoted z, w J x, y where z z z, w x y J = x, y w w x y z w z w = x y y x That is, the Jacobian is the determinant of the appropriate partial derivatives. We shall now state the theorem which relates f ( z w ) and ZW,, f x y.,, We shall assume that the transformation is one-to-one. That is, given (, ) z g x y (, ) w h x y =, (.33a) =, (.33b) then there is a unique set of numbers, (, ) x y satisfying Eq..33. Theorem.: To obtain f ZW,, z w, solve the system of equations g( x, y) = z, h( x, y) = w, for x and y. Let (, ) f ZW, ( z, w) = x y be the result of the solution. Then, (, ) f, x y J z, w x, y (.34) Proof of this theorem is given in appendix A.. For a more general version of this theorem, refer []..38

40 Example. and are two independent RVs with the PDFs, f ( x) =, x f ( y) =, y If Z = + and W =, let us find (a) f ( z w ) and (b) ZW,, f z. Z = + and a) From the given transformations, we obtain x ( z w) y = ( z w). We see that the mapping is one-to-one. Fig..4(a) depicts the (product) space A on which f,, x y is non-zero. Fig..4: (a) The space where f, is non-zero (b) The space where f ZW, is non-zero.39

41 We can obtain the space B (on which fzw, (, ) A space B space The line The line x = z w is non-zero) as follows: z + w = w = z + z + w = w = z + z w = w = z z w = w = z + x = The line The line y = y = The space B is shown in Fig..4(b). The Jacobian of the transformation is z, w J = = x, y Hence f ( z w) ZW, and J =. 4, z, w B, = = 8 0, otherwise fz z = fz, W z, w dw b) From Fig..4(b), we can see that, for a given z ( z 0 ), w can take values only in the z to z. Hence + z fz ( z) = dw = z, 0 z 8 4 z For z negative, we have z fz ( z) = dw = z, z z.40

42 Hence fz z = z, z 4 Example.3 Let the random variables R and Φ be given by, R = + and arc tan Φ = where we assume R 0 and π<φ<π. It is given that x + y f, ( x, y) = exp, < x, y <. π Let us find f ( r ) Φ ϕ. R,, write As the given transformation is from cartesian-to-polar coordinates, we can x = r cos φ and y = r sin φ, and the transformation is one -to -one; J r r cos ϕ sin ϕ x y = = sin ϕ cos ϕ = ϕ ϕ r r x y r Hence, f ( r ) R, Φ r r exp, 0 r, π < ϕ < π, ϕ = π 0, otherwise It is left as an exercise to find fr ( r ), f Φ ϕ and to show that R and Φ are independent variables. Theorem. can also be used when only one function Z g(, ) specified and what is required is f Z = is z. To apply the theorem, a conveniently chosen auxiliary or dummy variable W is introduced. Typically W = or W = ; using the theorem f ( z w ) is found from which f ( z ) can be obtained. ZW,, Z.4

43 Let Z = + and we require f Z z. Let us introduce a dummy variable W =. Then, = Z W, and = W. As J =, (, ) = (, ) f z w f z w w ZW,,,, (.35) and = ( ) f z f z w w dw Z If and are independent, then Eq..35 becomes = ( ) f z f z w f w dw (.36a) Z That is, f Z = f f (.36b) Example.4 Let and be two independent random variables, with f ( x), x = 0, otherwise f ( y) If Z, y = 3 0, otherwise = +, let us find [ ] P Z. From Eq..36(b), fz ( z ) is the convolution of f ( z ) and f out the convolution, we obtain fz ( z ) as shown in Fig..5. z. Carrying.4

44 Fig..5: PDF of Z = + of example.4 P[ Z ] is the shaded area =. Example.5 Let Z = ; let us find an expression for f z. Z Introducing the dummy variable W =, we have = = ZW W As J =, we have w fz ( z) = w f, ( zw, w) dw Let f ( x y), Then + xy, x, y, = 4 0, elsewhere + zw fz z = w dw 4.43

45 f, ( x, y ) is non-zero if ( x, y) A. where A is the product space y (Fig..6a). Let f ( z w ) be non-zero if ( z, w) B. Under x and ZW,, the given transformation, B will be as shown in Fig..6(b). Fig..6: (a) The space where f, is non-zero (b) The space where f ZW, is non-zero To obtain fz ( z ) from fzw, (, ) appropriate ranges. i) Let z < ; then + zw + zw fz ( z) = w dw = w dw z = + 4 ii) For z >, we have z zw + fz ( z) = w dw = z z 0 iii) For z <, we have z w, we have to integrate out w over the 3 z zw + fz ( z) = w dw = z z

46 Hence f ( z) Z z, 4 + z = +, z > 3 4 z z Exercise.7 Let Z = and W =. z f z = f, w dw w w a) Show that Z, b) and be independent with f ( x) =, x π + x and f ( y) y = ye, y 0 0, otherwise Show that z fz z = e, < z <. π In transformations involving two random variables, we may encounter a situation where one of the variables is continuous and the other is discrete; such cases are handled better, by making use of the distribution function approach, as illustrated below. Example.6 The input to a noisy channel is a binary random variable with = = = =. The output of the channel is given by Z = + [ 0] P[ ] P.45

47 where is the channel noise with Z f z. y f y = e, < y <. Find π Let us first compute the distribution function of Z from which the density function can be derived. ( ) = [ = 0] [ = 0 ] + [ = ] [ = ] P Z z P Z z P P Z z P As Z = +, we have [ = ] = P Z z 0 F z Similarly P[ Z z = ] = P ( z ) = F ( z ) Hence F ( z) = F ( z) + F ( z ). As f ( z) F ( z) Z Z d = Z, we have dz f Z ( z) z z = exp + exp π π The distribution function method, as illustrated by means of example.6, is a very basic method and can be used in all situations. (In this method, if = g, we compute F ( y ) and then obtain f ( y) df ( y) =. Similarly, for transformations involving more than one random variable. Of course computing the CDF may prove to be quite difficult in certain situations. The method of obtaining PDFs based on theorem. or. is called change of variable method.) We shall illustrate the distribution function method with another example. dy Example.7 Let Z = +. Obtain f ( z ), given Z f x y.,,.46

48 P[ Z z] = P[ + z] = P [ z x] This probability is the probability of (, ) lying in the shaded area shown in Fig..7. Fig..7: Shaded area is the P[ Z z] That is, z x FZ ( z) = d x f, ( x, y) dy z x fz ( z) = dx f, ( x, y) dy z z x = dx f, ( x, y) dy z f, ( x, z x) d x (.37a) = It is not too difficult to see the alternative form for f Z z, namely,.47

49 Z,, (.37b) = ( ) f z f z y y dy If and are independent, we have f ( z) f ( z) f ( z).37(b) is the same as Eq..35. =. We note that Eq. Z So far we have considered the transformations involving one or two variables. This can be generalized to the case of functions of n variables. Details can be found in [, ]..5 Statistical Averages The PDF of a random variable provides a complete statistical characterization of the variable. However, we might be in a situation where the PDF is not available but are able to estimate (with reasonable accuracy) certain (statistical) averages of the random variable. Some of these averages do provide a simple and fairly adequate (though incomplete) description of the random variable. We now define a few of these averages and explore their significance. The mean value (also called the expected value, mathematical expectation or simply expectation) of random variable is defined as m = E = = x f x d x (.38) [ ] where E denotes the expectation operator. Note that the expected value of a function of, g( ), is defined by m is a constant. Similarly, = = (.39) E g g g x f x d x Remarks: The terminology expected value or expectation has its origin in games of chance. This can be illustrated as follows: Three small similar discs, numbered, and respectively are placed in bowl and are mixed. A player is to be.48

50 blindfolded and is to draw a disc from the bowl. If he draws the disc numbered, he will receive nine dollars; if he draws either disc numbered, he will receive 3 dollars. It seems reasonable to assume that the player has a '/3 claim' on the 9 dollars and '/3 claim' on three dollars. His total claim is 9(/3) + 3(/3), or five dollars. If we take to be (discrete) random variable with the PDF f ( x) = δ( x ) + δ( x ) and g = 5 6, then 3 3 E g = ( 5 6 x) f ( x) d x = 5 That is, the mathematical expectation of g( ) is precisely the player's claim or expectation [3]. Note that g( x ) is such that g = 9 and g = 3. For the special case of n g probability distribution of the RV, ; that is, n n E = x f ( x) d x =, we obtain the n-th moment of the (.40) The most widely used moments are the first moment ( n =, which results in the mean value of Eq..38) and the second moment ( n =, resulting in the mean square value of ). E = = x f ( x) d x (.4) If g = ( m ) n, then E g gives n-th central moment; that is, n n ( ) = ( ) E m x m f x d x (.4) We can extend the definition of expectation to the case of functions of k( k ) random variables. Consider a function of two random variables, g(, ). Then,.49

51 (, ) (, ) (, ) E g = g x y f x y d x dy, (.43) An important property of the expectation operator is linearity; that is, if (, ) Z = g = α + β where α and β are constants, then Z =α +β. This result can be established as follows. From Eq..43, we have [ ] = ( α + β ), (, ) E Z x y f x y d x dy, (, ), (, ) = α x f x y d x dy + βy f x y d x dy Integrating out the variable y in the first term and the variable x in the second term, we have [ ] = α + β E Z x f x d x y f y dy =α +β.5. Variance Coming back to the central moments, we have the first central moment being always zero because, ( ) = ( ) E m x m f x d x = m m = Consider the second central moment = + E m E m m From the linearity property of expectation, 0 [ ] E m + m = E m E + m = E m + m = m =.50

52 The second central moment of a random variable is called the variance and its (positive) square root is called the standard deviation. The symbol σ is generally used to denote the variance. (If necessary, we use a subscript on σ ) The variance provides a measure of the variable's spread or randomness. Specifying the variance essentially constrains the effective width of the density function. This can be made more precise with the help of the Chebyshev Inequality which follows as a special case of the following theorem. Theorem.3: Let g( ) be a non-negative function of the random variable. If E g exists, then for every positive constant c, E g P g c (.44) c { } Proof: Let : A = x g x c and B denote the complement of A. E g = g x f x d x = g x f x d x + g x f x d x A Since each integral on the RHS above is non-negative, we can write If x E g g x f x d x A, then g( x) A c, hence E g c f x d x But [ ] A A f x d x = P x A = P g c That is, B E g c P g c, which is the desired result. Note: The kind of manipulations used in proving the theorem.3 is useful in establishing similar inequalities involving random variables..5

53 To see how innocuous (or weak, perhaps) the inequality.44 is, let g( ) represent the height of a randomly chosen human being with E g =.6 m. Then Eq..44 states that the probability of choosing a person over 6 m tall is at most! (In a population of billion, at most 00 million would be as tall as a 0 full grown Palmyra tree!) Chebyshev inequality can be stated in two equivalent forms: i) P m k σ, k > 0 (.45a) k ii) P m < k σ > (.45b) k where σ is the standard deviation of. To establish.45(a), let g = ( m ) and c = k σ in theorem.3. We then have, P m k σ In other words, k P m k σ k which is the desired result. Naturally, we would take the positive number k to be greater than one to have a meaningful result. Chebyshev inequality can be interpreted as: the probability of observing any RV outside ± k standard deviations off its mean value is no larger than k. With k = for example, the probability of m σ does not exceed /4 or 5%. By the same token, we expect to occur within the range ( ) m ± σ for more than 75% of the observations. That is, smaller the standard deviation, smaller is the width of the interval around m, where the required probability is concentrated. Chebyshev.5

54 inequality thus enables us to give a quantitative interpretation to the statement 'variance is indicative of the spread of the random variable'. Note that it is not necessary that variance exists for every PDF. For example, if α f x π α + x x =, < < and α > 0 (This is called Cauchy s PDF), then = 0 but is not finite..5. Covariance An important joint expectation is the quantity called covariance which is obtained by letting g( ) ( m ) ( m ), symbol λ to denote the covariance. That is, = in Eq..43. We use the [ ] E ( m ) ( m ) λ = cov, = (.46a) Using the linearity property of expectation, we have The [ ] [ ] λ = E m m (.46b) cov,, normalized with respect to the product σ σ is termed as the correlation coefficient and we denote it by ρ. That is, ρ = [ ] m m E σ σ (.47) The correlation coefficient is a measure of dependency between the variables. Suppose and are independent. Then, E xy f x y d x dy [ ] =, (, ) = x yf x f y dxdy = x f x d x y f y dy = m m.53

55 Thus, we have λ (and ρ ) being zero. Intuitively, this result is appealing. Assume and to be independent. When the joint experiment is performed many times, and given = x, then would occur sometimes positive with respect to m, and sometimes negative with respect to m. In the course of many trials of the experiments and with the outcome = x, the sum of the numbers x ( y m ) would be very small and the quantity, tends to zero as the number of trials keep increasing. sum number of trials, On the other hand, let and be dependent. Suppose for example, the outcome y is conditioned on the outcome x in such a manner that there is a. Then we greater possibility of ( y m ) being of the same sign as ( x m ) expect ρ to be positive. Similarly if the probability of ( ) and ( ) x m y m being of the opposite sign is quite large, then we expect ρ to be negative. Taking the extreme case of this dependency, let and be so conditioned that, given = x, then = ±α x, α being constant. Then ρ = ±. That is, for and be independent, we have ρ = 0 and for the totally dependent ( y x) = ± α case, ρ =. If the variables are neither independent nor totally dependent, then ρ will have a magnitude between 0 and. Two random variables and are said to be orthogonal if E[ ] = 0. If ( ) ρ or λ = 0, the random variables and are said to be uncorrelated. That is, if and are uncorrelated, then =. When the random variables are independent, they are uncorrelated. However, the fact that they are uncorrelated does not ensure that they are independent. As an example, let.54

56 f ( x) α α, < x < = α, 0, otherwise and =. Then, α = = x dx = x dx = α α α 0 As = 0, = 0 which means λ = = 0. But and are not independent because, if = x, then = x! Let be the linear combination of the two random variables and. That is = k + k where k and i i k are constants. Let E[ ] σ = σ, i =,. Let ρ be the correlation coefficient between and. We will now relate σ to the known quantities. [ ] σ = E E E [ ] = km + km E = E k + k + kk With a little manipulation, we can show that i = m, σ = kσ + kσ + ρkkσσ (.48a) If and are uncorrelated, then σ = kσ + kσ (.48b) Note: Let and be uncorrelated, and let Z = + and W =. i Then σ = σ = σ + σ. That is, the sum as well as the difference random Z W variables have the same variance which is larger than σ or σ! The above result can be generalized to the case of linear combination of n variables. That is, if.55

57 n = k, then i = i i n ki i ki kj i j i j i = i j σ = σ + ρ σ σ i < j (.49) where the meaning of various symbols on the RHS of Eq..49 is quite obvious. We shall now give a few examples based on the theory covered so far in this section. Example.8 Let a random variable have the CDF F 0, x < 0 x, 0 x 8 = x, x 4 6, 4 x ( x) We shall find a) and b) σ a) The given CDF implies the PDF f Therefore, ( x) 0, x < 0, 0 x 8 = x, x 4 8 0, 4 x 4 E = x dx + x dx [ ]

58 7 3 = + = 4 3 b) [ ] σ = E E 4 3 E = x d x + x d x = + = σ = 6 = 44 Example.9 Let f ( x) = cos π, where, = 0, Let us find E [ ] and < x < otherwise σ. From Eq..38 we have, [ ] E = cos π = = x dx π ( x) d x E = cos π = = 0.5 Hence 4 σ = = π 0.96 Example.0 Let and have the joint PDF f, ( x, y) x + y,0< x <,0 < y < = 0, elsewhere.57