6. Vector Random Variables

Transcription

1 6. Vector Random Variables In the previous chapter we presented methods for dealing with two random variables. In this chapter we etend these methods to the case of n random variables in the following was: B representing n random variables as a vector, we obtain a compact notation for the joint pmf, cdf, and pdf as well as marginal and conditional distributions. We present a general method for finding the pdf of transformations of vector random variables. Summar information of the distribution of a vector random variable is provided b an epected value vector and a covariance matri. We use linear transformations and characteristic functions to find alternative representations of random vectors and their probabilities. We develop optimum estimators for estimating the value of a random variable based on observations of other random variables. We show how jointl Gaussian random vectors have a compact and eas-to-workwith pdf and characteristic function. Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

2 6.1 Vector Random Variables The notion of a random variable is easil generalized to the case where several quantities are of interest. A vector random variable is a function that assigns a vector of real numbers to each outcome ζ in S, the sample space of the random eperiment. We use uppercase boldface notation for vector random variables. B convention is a column vector (n rows b 1 column), so the vector random variable with components,, 1, n corresponds to 1 n We will sometimes write,,, 1 n to save space and omit the transpose unless dealing with matrices. Possible values of the vector random variable are denoted b 1,,, n where i corresponds to the value of i. 1 T n Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

3 6.1.1 Events and Probabilities 6.1. Joint Distribution Functions Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

4 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

5 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

6 6.1.3 Independence Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

7 6. Functions of Several Random Variables Functions of vector random variables arise naturall in random eperiments. For eample,, 1, n ma correspond to observations from n repetitions of an eperiment that generates a given random variable. We are almost alwas interested in the sample mean and the sample variance of the observations. In another eample 1,,, n ma correspond to samples of a speech waveform and we ma be interested in etracting features that are defined as functions of for use in a speech recognition sstem One Function of Several Random Variables Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

8 6.. Transformations of Random Vectors Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

9 6..3 *pdf of General Transformations Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

10 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

11 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

12 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

13 6.3 Epected Values of Vector Random Variables Mean Vector and Covariance Matri Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

14 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

15 6.3. Linear Transformations of Random Vectors Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

16 6.3.3 *Joint Characteristic Function *Diagonalization of Covariance Matri Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

17 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

18 6.4 Jointl Gaussian Random Vectors Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

19 6.4.1 *Linear Transformation of Gaussian Random Variables 6.4. *Joint Characteristic Function of a Gaussian Random Variable Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

20 6.5 Estimation of Random Variables In this book we will encounter two basic tpes of estimation problems. In the first tpe, we are interested in estimating the parameters of one or more random variables, e.g., probabilities, means, variances, or covariances. In Chapter 1, we stated that relative frequencies can be used to estimate the probabilities of events, and that sample averages can be used to estimate the mean and other moments of a random variable. In Chapters 7 and 8 we will consider this tpe of estimation further. In this section, we are concerned with the second tpe of estimation problem, where we are interested in estimating the value of an inaccessible random variable in terms of the observation of an accessible random variable. For eample, could be the input to a communication channel and could be the observed output. In a prediction application, could be a future value of some quantit and its present value. Estimators: Maimum a posteriori (MAP) estimator Maimum likelihood (ML) estimator Minimum MSE Estimator MAP and ML Estimators We have considered estimation problems informall earlier in the book. For eample, in estimating the output of a discrete communications channel we are interested in finding the most probable input given the observation =, that is, the value of input that maimizes P : ma P In general we refer to the above estimator for in terms of as the maimum a posteriori (MAP) estimator. The a posteriori probabilit is given b: P P P P and so the MAP estimator requires that we know the a priori probabilities P. Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

21 In some situations we know P but we do not know the a priori probabilities, so we select the estimator value as the value that maimizes the likelihood of the observed value =: map We refer to this estimator of in terms of as the maimum likelihood (ML) estimator. We can define MAP and ML estimators when and are continuous random variables b replacing events of the form b d. If and are continuous, the MAP estimator for given the observation is given b: ma and the ML estimator for given the observation is given b: ma f f Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

22 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37 Eample 6.5 Comparison of ML and MAP Estimators Let and be the random pair in Eample Find the MAP and ML estimators for in terms of. From Eample 5.16 and 5.3, the conditional pdf of given is given b: else e e f 0, 0,,, e f and e e f 1 the conditional pdfs are: for e e e e e e f f f 0, 1 1,, for e e e e f f f,,, The MAP estimator f ma for e f, As the function is a smooth curve in, using the derivative to determine the ma and min does not help and the bounds must be at either or. As the function decreases as increases beond, the maimum must occur at. Therefore the MAP estimator is. ˆ MAP The ML estimator f ma for e e f 0, 1 The derivative is again not useful; however, as increases beond, the denominator becomes larger so the conditional pdf decreases. Therefore the ML estimator is. ˆ ML In this eample the ML and MAP estimators agree.

23 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37 Eample 6.6 Jointl Gaussian Random Variables Find the MAP and ML estimator of in terms of when and are jointl Gaussian random variables. Jointl Gaussian RV from section 5.9:, 1 ep 1 1, m m m m f The conditional pdf of given is given b: 1 ep 1 1 m m f which is maimized b the value of for which the eponent is zero. Therefore 0 ˆ MAP m m MAP m m ˆ The conditional pdf of given is given b: 1 ep 1 1 m m f which is again maimized b the value of for which the eponent is zero. Therefore 0 ˆ ML m m ML m m ˆ Therefore we conclude that the two estimators are not equal. In other words, knowledge of the a priori probabilities of will affect the estimator.

24 6.5. Minimum MSE Linear Estimator ˆ g. In general, the Another estimate for is given b a function of the observation ~ estimation error, ˆ g, is nonzero, and there is a cost associated with the ~ error, cost c c g. We are usuall interested in finding the function g() that minimizes the epected value of the cost, Ec ~ Ec g. For eample, if and are the discrete input and output of a communication channel, and c is zero when g and one otherwise, then the epected value of the cost corresponds to the probabilit of error, that is, that g.when and are continuous random variables, we frequentl use the mean square error (MSE) as the cost: ~ e E E g In the remainder of this section we focus on this particular cost function. We first consider the case where g() is constrained to be a linear function of, and then consider the case where g() can be an function, whether linear or nonlinear. First, consider the problem of estimating a random variable b a constant a so that the mean square error is minimized: min E a E a E a a The best a is found b taking the derivative with respect to a, setting the result to zero, and solving for a. The result is E a 0 * a E which makes sense since the epected value of is the center of mass of the pdf. Note that the conjugate of a is shown in case a is comple. * The mean square error for this estimator is equal to VAR E a Now consider estimating b a linear function g a b min E a b a, b. Equation (6.53a) can be viewed as the approimation of a b the constant b. And then finding the best a. This is the minimization posed in Eq. (6.51) and the best b is * b E a E a E Substitution into Eq. (6.53a) implies that the best a is found b min E E a E a Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

25 We once again differentiate with respect to a, set the result to zero, and solve for a: E E a E E E 0 E E a E E COV, a VAR 0 0 The best coefficient a is found to be * COV, a VAR,, Therefore, the minimum mean square error (mmse) linear estimator for in terms of is ˆ * * a b ˆ a * E * a E ˆ, E E The term E is simpl a zero-mean, unit-variance version of. Thus E is a rescaled version of that has the variance of the random variable that is being estimated, namel.the term E[] simpl ensures that the estimator has the correct mean. The ke term in the above estimator is the correlation coefficient:, that specifies the sign and etent of the estimate of relative to E. If and are uncorrelated then the best estimate for is its mean, E[]. On the other hand, if, 1 then the best estimate is equal to ˆ E E We draw our attention to the second equalit in Eq. (6.54): * E E a E E 0 This equation is called the orthogonalit condition because it states that the error of the best linear estimator, the quantit inside the braces, is orthogonal to the observation E The orthogonalit condition is a fundamental result in mean square estimation. Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

26 Computing the cost function for this estimator where the second equalit follows from the orthogonalit condition. Note that when 1 the mean square error is zero. This implies that * * P a b 0 so that is essentiall a linear function of. * * P a b 1, Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

27 6.5.3 Minimum MSE Estimator (generall nonlinear) In general the estimator for that minimizes the mean square error is a nonlinear function of. The estimator g() that best approimates in the sense of minimizing mean square error must satisf min E g g The problem can be solved b using conditional epectation: E g E E g E g E g f The integrand above is positive for all ; therefore, the integral is minimized b minimizing E g for each. But g() is a constant as far as the conditional epectation is concerned, so the problem is equivalent to Eq. (6.51) and the constant that minimizes E g is g * E The function is called the regression curve which simpl traces the conditional epected value of given the observation =. The mean square error of the best estimator is: g E E d f * e E R * e VAR R f d d Linear estimators in general are suboptimal and have larger mean square errors. Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

28 6.5.4 Estimation using a Vector of Observations The MAP, ML, and mean square estimators can be etended to where a vector of observations is available. Here we focus on mean square estimation. We wish to estimate b a function g() of a random vector of observations so that the mean square error is minimized: min E g g To simplif the discussion we will assume that and the have zero means. The same derivation that led to Eq. (6.58) leads to the optimum minimum mean square estimator: The minimum mean square error is then: g * E g E E f * e E R * e VAR R f d d Note that the solution is based on an auto-correlation matri and a cross-correlation vector. Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

29 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

30 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

31 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

32 6.6 *Generating Correlated Vector Random Variables Man applications involve vectors or sequences of correlated random variables. Computer simulation models of such applications therefore require methods for generating such random variables. In this section we present methods for generating vectors of random variables with specified covariance matrices. We also discuss the generation of jointl Gaussian vector random variables Generating Random Vectors with Specified Covariance Matri Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

33 6.6. Generating Vectors of Jointl Gaussian Random Variables %% % Eample 6.34 % The necessar steps for generating the Gaussian random variables % with the covariance matri from Eample 6.3. clear %close all U1=rand(10000, 1); % Create a 1000-element vector U1. U=rand(10000, 1); % Create a 1000-element vector U. R=-*log(U1); % Find TH=*pi*U; % Find 1=sqrt(R).*sin(TH); % Generate 1. =sqrt(r).*cos(th); % Generate. m1 = mean(1) m = mean() v1=sqrt(cov(1)) v=sqrt(cov()) 1= 1+sqrt(3)*; % Generate 1. =-1+sqrt(3)*; % Generate. figure plot(1,,'+') figure plot(1,,'+') % Plot scattergram % Plot scattergram m1 = m = v1 = v = Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

34 %% % Eample 6.34 % The necessar steps for generating the Gaussian random variables % with the covariance matri from Eample 6.3. clear close all 1=randn(10000, 1); % Create a 1000-element vector U1. =randn(10000, 1); % Create a 1000-element vector U. m1 = mean(1) m = mean() v1=sqrt(cov(1)) v=sqrt(cov()) 1= 1+sqrt(3)*; % Generate 1. =-1+sqrt(3)*; % Generate. figure plot(1,,'+') figure plot(1,,'+') % Plot scattergram % Plot scattergram m1 = m = v1 = v = Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

35 Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

36 Summar The joint statistical behavior of a vector of random variables is specified b the joint cumulative distribution function, the joint probabilit mass function, or the joint probabilit densit function. The probabilit of an event involving the joint behavior of these random variables can be computed from these functions. The statistical behavior of subsets of random variables from a vector is specified b the marginal cdf, marginal pdf, or marginal pmf that can be obtained from the joint cdf, joint pdf, or joint pmf of. A set of random variables is independent if the probabilit of a product-form event is equal to the product of the probabilities of the component events. Equivalent conditions for the independence of a set of random variables are that the joint cdf, joint pdf, or joint pmf factors into the product of the corresponding marginal functions. The statistical behavior of a subset of random variables from a vector, given the eact values of the other random variables in the vector, is specified b the conditional cdf, conditional pmf, or conditional pdf. Man problems naturall lend themselves to a solution that involves conditioning on the values of some of the random variables. In these problems, the epected value of random variables can be obtained through the use of conditional epectation. The mean vector and the covariance matri provide summar information about a vector random variable. The joint characteristic function contains all of the information provided b the joint pdf. Transformations of vector random variables generate other vector random variables. Standard methods are available for finding the joint distributions of the new random vectors. The orthogonalit condition provides a set of linear equations for finding the minimum mean square linear estimate. The best mean square estimator is given b the conditional epected value. The joint pdf of a vector of jointl Gaussian random variables is determined b the vector of the means and b the covariance matri. All marginal pdf s and conditional pdf s of subsets of have Gaussian pdf s. An linear function or linear transformation of jointl Gaussian random variables will result in a set of jointl Gaussian random variables. A vector of random variables with an arbitrar covariance matri can be generated b taking a linear transformation of a vector of unit-variance, uncorrelated random variables. A vector of Gaussian random variables with an arbitrar covariance matri can be generated b taking a linear transformation of a vector of independent, unit-variance jointl Gaussian random variables. Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37

37 CHECKLIST OF IMPORTANT TERMS Conditional cdf Conditional epectation Conditional pdf Conditional pmf Correlation matri Covariance matri Independent random variables Jacobian of a transformation Joint cdf Joint characteristic function Joint pdf Joint pmf Jointl continuous random variables Jointl Gaussian random variables Karhunen-Loeve epansion MAP estimator Marginal cdf Marginal pdf Marginal pmf Maimum likelihood estimator Mean square error Mean vector MMSE linear estimator Orthogonalit condition Product-form event Regression curve Vector random variables Notes and figures are based on or taken from materials in the tetbook: Alberto Leon-Garcia, Probabilit, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 008, ISBN: of 37