Chapter 4 Continuous Random Variables In this chapter we study continuous random variables. The linkage between continuous and discrete random variables is the cumulative distribution (CDF) which we will take a closer look in this chapter. 4. PDF As we discussed in Chapter 3, random variables are objects characterized through their probability mass functions (for the discrete case). For continuous random variables, we have an analogous function called the probability density function (PDF). Definition. The probability density function (PDF) of a random variable X is a function which, when integrated over an interval [a, b], yields the probability of obtaining a X(ξ) b. We denote PDF of X as f X (x), and P[a X b] b a f X (x)dx. (4.) The key difference between a PMF and a PDF is the integration present in the PDF. However, it is possible to include the integration for PMF. Suppose we are interested in calculating the probability of a random variable X that lies in the interval [a, b], see Figure 4.. If X is continuous, this probability (according to the definition of PDF) is P[a X b] b a f X (x)dx. If X is discrete, then (according to the definition of PMF) the probability is P[a X b] (a) P[X x ] p X (x ) (b) b a p X (x )δ(x x ) dx, }{{} f X (x)
where (a) holds because there is no other non-zero probabilities in the interval [a, b] besides x, and (b) holds because the delta function has the property that δ(x x ) if x x, and δ(x x ) if x x. Therefore, we see that the PMF, which is a sequence of numbers, can be seen as a train of delta functions. The height of the delta function at x is p X (x ). The integration of this train of delta functions returns the probability P[a X b]. Figure 4.: PMF is a train of impulses, whereas PDF is (usually) a smooth function. Property. A PDF f X (x) should satisfy f X (x)dx. (4.) Example. Let f X (x) c( x ) for x, and otherwise. Then, the constant c can be found as follows. Since f X (x)dx c( x )dx 4c 3, and because f X(x)dx, we have c 3/4. Remark. What is P[X x ] if X is continuous? The long answer will be discussed in the next section. The short answer is: If the PDF f X (x) (which is a function of x) is continuous at x x, then P[X x ]. This can be seen from the definition of the PDF. As a x and b x, the integration interval becomes. Thus, unless f X (x) is a delta function at x, which has been excluded because we assumed f X (x) is continuous at x, the integration result must be zero. 4. CDF Definition. The cumulative distribution function (CDF) of a continuous random variable X is x F X (x) def P[X x] f X (x )dx. (4.3) c 7 Stanley Chan. All Rights Reserved.
Note that the definition of CDF for a continuous X is analogous to that for a discrete X, with the summation replaced by the integration. Properties of CDF As we have seen in Chapter 3, a CDF has several interesting properties. Here we provide more details. The first four properties come from the integration of the PDF. Property : F X () Reason: F X () f X(x )dx. Property : F X (+ ) Reason: F X (+ ) + f X(x )dx. Property 3: F X (x) is a non-decreasing function of x. Reason: Since F x (x) x f X(x )dx, and since f X (x ) for all x, if we increase x we integrate over a larger region. The integrated value therefore increases. See Figure 4. for a pictorial illustration. Property 4: F X (x). Reason: This follows from Properties -3. Figure 4.: A CDF is the integration of the PDF. Thus, the height of a stem in the CDF corresponds to the area under the curve of the PDF. The next property relates probability and CDF. Property 5: P[a X b] F X (b) F X (a). c 7 Stanley Chan. All Rights Reserved. 3
Reason: Note that F X (b) F X (a) b b a f X (x )dx f X (x )dx P[a X b]. a f X (x )dx Property 6 and Property 7 are important for discontinuities of F X (x). There are three terms involved in these two properties: (i) F X (b): The value of F X (x) at x b. (ii) lim h F X (b h): The limit of F X (x) from the left hand side of x b. (iii) lim h F X (b + h): The limit of F X (x) from the right hand side of x b. We say that F X (x) is Left-continuous at x b if F X (b) lim h F X (b h); Right-continuous at x b if F X (b) lim h F X (b + h); Continuous at x b if it is both right-continuous and left-continuous at x b. In this case, we have lim F X(b h) lim F X (b + h) F (b). h h Property 6: F X (x) is right-continuous. That is, lim F X(b + h) F X (b). (4.4) h Right-continuous essentially means that if F X (x) is piecewise, then it must have a solid left end and an empty right end. See Figure 4.3. Figure 4.3: Illustration of right continuous. Of course, F X (x) does not have to be piecewise. When F X (x) is continuous (i.e., both left and right continuous), the leftmost solid dot will overlap with the rightmost empty dot of the previous segment, leaving not gap between the two. c 7 Stanley Chan. All Rights Reserved. 4
Property 7: P[X b] is determined by P[X b] F X (b) lim h F X (b h). (4.5) Property 7 concerns about the probability at discontinuity. It states that when F X (x) is discontinuous at x b, then P[X b] is the difference between F X (b) and the limit from the left. In other words, the height of the gap determines the probability at discontinuity. If F X (x) is continuous at x b, then F X (b) lim h F X (b h) and so P[X b]. Figure 4.4: Illustration Property 7. Fundamental Theorem of Calculus Thus far we have only seen how to obtain F X (x) from f X (x). In order to go in the reverse direction, we need to recall the fundamental theorem of calculus. Fundamental theorem of calculus states that if a function f is continuous, then f(x) d dx x a f(t)dt for some constant a. Using this result for CDF and PDF, we have the following result: Theorem. The probability density function (PDF) is the derivative of the cumulative distribution function (CDF): f X (x) df X(x) dx d x f X (x )dx, (4.6) dx provided F X is differentiable at x. c 7 Stanley Chan. All Rights Reserved. 5
Remark: If F X is not differentiable at x, then, f X (x) P[X x] F X (x) lim h F X (x h), where the second equality follows from Property 7. Example. Consider a CDF F X (x) Our goal is to find the PDF f X (x). {, x <, 4 e x, x. Figure 4.5: Example. First, it is easy to show that F X () 3. This corresponds to a discontinuity at x 4, as shown in Figure 4.5. Because of the discontinuity, we need to consider three when determining f X (x): df X (x), x <, dx f X (x) P[X ], x, df X (x), x >. dx When x <, F X (x). So, df X(x). dx When x >, F X (x) 4 e x. So, df X(x) dx e x. When x, the probability P[X ] is determined according to Property 7 which is the height between the solid dot and the empty dot. This yields Therefore, the overall PDF is P[X ] F X () lim h F X ( h) 3 4 3 4., x <, 3 f X (x) 4, x, e x, x >. c 7 Stanley Chan. All Rights Reserved. 6
4.3 Expectation, Variance and Moments Like the discrete random variables, the mean and moments of a continuous random variable can be defined using integrations. Definition 3. The expectation of a continuous random variable X is E[X] x f X (x)dx. (4.7) Similarly, any function of X can be defined as: Definition 4. The expectation of a function g of a continuous random variables X is E[g(X)] g(x) f X (x)dx. (4.8) Note that in this definition, the PDF f X (x) is not changed by the function g. Definition 5. The kth moment of a continuous random variables X is E[X k ] Finally, we can define the variance of a continuous random variable: Definition 6. The variance of a continuous random variables X is where µ X def E[X]. x k f X (x)dx. (4.9) Var[X] E[(X µ X ) ], (4.) It is not difficult to show that the variance can also be expressed as or Var[X] (x µ X ) f X (x)dx, Var[X] E[X ] E[X]. c 7 Stanley Chan. All Rights Reserved. 7
4.4 Mean, Mode, Median In statistics, we are typically interested in three quantities: mean, mode and median. It is possible to obtain these three quantities from PDF and CDF. From PDF We first show how mean, mode and median can be obtained from the PDF. A pictorial illustration of mode and median is given in Figure 4.6. Median: The median is the point where the area under f X (x) on the two sides equal: Median x such that x f X (x )dx x f X (x )dx. (4.) Mode: The mode is the point where f X (x) attains the maximum. Mode argmax x f X (x) (4.) Mean: The mean is calculated from the definition of expectation: Mean E[X] xf X (x)dx. (4.3) Figure 4.6: Mode and median in a PDF and a CDF. From CDF Since CDF is the integration of the PDF, it is possible to derive the mean, mode, and median from the CDF. c 7 Stanley Chan. All Rights Reserved. 8
Median: The median is the point where F X (x) achieves : Median x such that F X (x). (4.4) Mode: The mode is the point where F X (x) attains the steepest slope, so that the derivative (i.e., PDF) attains the maximum. Mode argmax F X(x). (4.5) x Mean: The mean can be computed as follows. E[X] ( F X (x )) dx F X (x )dx. (4.6) Proof. For any random variable X, we can partition X X + X where X + and X are the positive and negative parts, respectively. Consider the positive part first. For any X >, it holds that ( F X (x )) dx [ P[X x ]] dx x f X (x)dxdx xf X (x)dx E[X]. x P[X > x ]dx f X (x)dx dx Now, consider the negative part. For any X <, it holds that F X (x )dx P[X x ]dx f X (x)dx dx x x f X (x)dxdx xf X (x)dx E[X], where in both cases we switched the order of the integration. c 7 Stanley Chan. All Rights Reserved. 9
4.5 Common Continuous Random Variables Uniform Random Variable Definition 7. Let X be a continuous uniform random variable. The PDF of X is f X (x) { b a a x b,, otherwise, (4.7) where [a, b] is the interval on which X is defined. We write X Uniform(a, b) to say that X is drawn from a uniform distribution on an interval [a, b]. Uniform random variables are very common in numerical simulations. The statistical properties of a uniform random variable is summarized as follows. Figure 4.7: Uniform distribution Proposition. If X Uniform(a, b), then Proof. E[X] E[X ] E[X] a + b (b a), and Var[X]. xf X (x)dx x f X (x)dx Var[X] E[X ] E[X] b a b a x b a dx a + b x b a dx a + ab + b 3 (b a). c 7 Stanley Chan. All Rights Reserved.
Exponential Random Variable Definition 8. Let X be an exponential random variable. The PDF of X is { λe λx, x, f X (x), otherwise, (4.8) where λ > is a parameter. We write X Exponential(λ) to say that X is drawn from an exponential distribution of parameter λ. Figure 4.8: Exponential distribution Proposition. If X Exponential(λ), then E[X] λ, and Var[X] λ. Proof. E[X] E[X ] xf X (x)dx xe λx + λxe λx dx xde λx x f X (x)dx x e λx + e λx dx λ λx e λx dx xe λx dx + λ E[X] λ Var[X] E[X ] E[X] λ, where we used integration by parts in calculating E[X] and E[X ]. x de λx c 7 Stanley Chan. All Rights Reserved.
Gaussian Random Variable Definition 9. Let X be an Gaussian random variable. The PDF of X is f X (x) (x µ) e σ (4.9) πσ where (µ, σ ) are parameters of the distribution. We write X N (µ, σ ) to say that X is drawn from a Gaussian distribution of parameter (µ, σ ). Gaussian random variables are also called normal random variables. As will be shown, the parameters (µ, σ ) are in fact the mean and the variance of the random variable. Gaussian random variables are extremely common in engineering and statistics. Among many reasons, one important reason is the Central Limit Theorem which we will discuss later. Figure 4.9: Gaussian distribution Proposition 3. If X N (µ, σ ), then E[X] µ, and Var[X] σ. Proof. E[X] (a) πσ πσ πσ ( (b) + µ πσ (c) µ, xe (x µ) σ dx (y + µ)e y ye y σ dy + σ dy πσ ) e y σ dy µe y σ dy c 7 Stanley Chan. All Rights Reserved.
where in (a) we substitute y x µ, in (b) we use the fact that the first integrand is odd so that the integration is, and in (c) we observe that integration over the entire sample space of the PDF yields. Var[X] πσ (a) σ π σ π ( ye y + σ ( π σ, where in (a) we substitute y (x µ)/σ. (x µ) e (x µ) σ dx y e y x µ dy, y σ ) + σ e y dy π ) e y dy Standard Gaussian In many practical situations, we need to evaluate the probability P[a X b] of a Gaussian random variable X. This involves integration of the Gaussian PDF. Unfortunately, there is no closed-form expression of P[a X b] in terms of (µ, σ ). One often has to use a table to find the numerical value of the probability. Standard Gaussian is created for this reason. Definition. A standard Gaussian (or standard Normal) random variable X has a PDF f X (x) e x. (4.) π That is, X N (, ) is a Gaussian with µ and σ. Definition. The Φ( ) function of the standard Gaussian is Φ(y) π y e x dx (4.) Essentially, Φ(y) is the CDF of the standard Gaussian. Once we have the Φ( ) function, it becomes easy to write probability in terms of Φ( ). c 7 Stanley Chan. All Rights Reserved. 3
Figure 4.: Definition of Φ(y). Example. Let X N (µ, σ ). Consequently, we have P[X b] b b µ σ πσ ( b µ Φ σ (x µ) e σ dx e y x µ dy, where y π σ ). P[a < X b] P[X b] P[X a] ( ) ( ) b µ a µ Φ Φ. σ σ 4.6 Function of Random Variables One common question we encounter in practice is the transformation of random variables. The question is simple: Given a random variable X with PDF f X (x) and CDF F X (x), and suppose that Y g(x) for some function g, then what are f Y (y) and F Y (y)? The general technique we use to tackle these problem involves the following steps: Step : Find the CDF F Y (y), which is P[Y y], using F X (x). Step : Find the PDF f Y (y) by taking derivative on F Y (y). To see how these steps are used, we consider the following examples. Example. Let X be a random variable with PDF f X (x) and CDF F X (x). Let Y X +3. Find f Y (y) and F Y (y). To tackle this problem, we first note that [ F Y (y) P[Y y] P[X + 3 y] P X y 3 ] ( ) y 3 F X. c 7 Stanley Chan. All Rights Reserved. 4
Therefore, f Y (y) d dy F Y (y) d ( ) y 3 dy F X ( ) ( ) y 3 d y 3 F X f X dy ( ) y 3. Example. Let X be a random variable with PDF f X (x) and CDF F X (x). Suppose that Y X, find f Y (y) and F Y (y). To solve this problem, we note that Therefore, the PDF is F Y (y) P[Y y] P[X y] P[ y X y] F X ( y) F X ( y). f Y (y) d dy F Y (y) d dy (F X( y) F X ( y)) F X( y) d dy y F X ( y) d dy ( y) y (f X( y) + f X ( y)). Example 3. Let X Uniform(, π). Suppose Y cos X. Find f Y (y) and F Y (y). First of all, we need to find the CDF of X. This can be done by noting that Thus, the CDF of Y is The PDF of Y is F X (x) x f X (x )dx x F Y (y) P[Y y] P[cos X y] P[cos y π cos y] π dx x π. F X (π cos y) F X (cos y) cos y. π f Y (y) d dy F Y (y) d dy π y, ( ) cos y π c 7 Stanley Chan. All Rights Reserved. 5
where we used the fact that d dy cos y y. In some cases we do not need to explicitly find the CDF in order to find the PDF. Below is an example. Example 4. Let X be a random variable with PDF f X (x) ae x e aex. Let Y e X, and find f Y (y). To solve this problem, we first note that F Y (y) P[Y y] P[e X y] P[X log y] log y ae x e aex dx. To find the PDF, we recall the fundamental theorem of calculus. This will give us log y f Y (y) d ae x e aex dx dy ( ) ( d d dy log y d log y y aelog y e aelog y ae ay. log y ) ae x e aex dx 4.7 Generating Random Numbers For common types of random variables, e.g., Gaussian or exponential, there are often built-in functions in MATLAB to generate the random numbers. However, for arbitrary PDFs, how can we generate the random numbers? Generating arbitrary random numbers can be done with the help of the following theorem. Theorem. Let X be a random variable with an invertible CDF F X (x), i.e., F X If Y F X (X), then Y Uniform(, ). exists. Proof. First, we know that if U Uniform(, ), then f U (u) for u and so F U (u) u f U (u)du u, c 7 Stanley Chan. All Rights Reserved. 6
for u. If Y F X (X), then the CDF of Y is F Y (y) P[Y y] P[F X (X) y] P[X F X (y)] F X (F X (y)) y. Therefore, we showed that the CDF of Y is the CDF of a uniform distribution. completes the proof. This The consequence of the theorem is the following procedure in generating random numbers: Step : Generate a random number U Uniform(, ). Step : Let Y F X (U). Then the distribution of Y is F X. Example. Suppose we want to draw random numbers from an exponential distribution. Then, we can follow the above two steps as Step : Generate a random number U Uniform(, ). Step : Find F X. Since X is exponential, we know that F X(x) e λx. This gives the inverse F X (x) log( x). Therefore, by letting Y F λ X (U), the distribution of Y will be exponential. c 7 Stanley Chan. All Rights Reserved. 7