2: Distributions of Several Variables, Error Propagation

Size: px

Start display at page:

Download "2: Distributions of Several Variables, Error Propagation"

Maximilian James
5 years ago
Views:

1 : Distributions of Several Variables, Error Propagation Distribution of several variables. variables The joint probabilit distribution function of two variables and can be genericall written f(, with the normalization condition: f(, d d The marginal distribution of is the projection of this function on the variable : g( f(, d The conditional distribution of given 0 is a cut at a given 0 : f(, 0 f( 0 f(, 0 d Two variables are independent if their joint probabilit distribution function can be epressed as: p(, f( g( In this case, the marginal and the conditional distributions are equal. The epectation of an function g of and is : E[g(, ] g(, f(, d d In particular, the mean and the variance of, sa,, can be written: µ E( f(, d d σ E( f(, d d Eercise: Bumping cars: at the funfair, bumping cars all move at constant speed V and at random angle ϕ. What is the distribution of V? Same problem with a cloud of flies of constant speed, in 3-D? See sample solutions at: --> BumpingCars.m, Flies3d.m. Covariance and correlation The covariance of and is defined as: cov(, E[( ( ] E( E( E( + E( E( E( cov(, Clearl, cov(, σ and cov(, σ. B definition, the covariance matri is defined as: ( ( V V V σ cov(, V V cov(, σ

2 The correlation factor Its basic propert is that is the dimensionless quantit ρ cov(, σ σ ρ The proof is instructive: let s use the fact that the variance of an quantit, including a linear combination a +, is positive: V (a + E([a + (a + ] E([a + ] (a + a + + a a ( ( a 0 a σ + σ + a cov(, This second order polnomial in a has at most one root, i.e. its discriminant is 0: cov(, σ σ 0 Thus cov(, σ σ, ρ. The polnomial V (a+ has a root when a+ is constant (zero variance, i.e. when there eists a linear combination between and, in which case ρ ±. The covariance (and therefore the correlation factor of two independent variables is zero: cov(, ( ( f( g( d d ( ( f( d ( ( g( d ( ( 0 As a consequence, the variance of the sum of two (or more independent variables is the sum of the variances of each of the variables: V ( + σ + σ + cov(, σ + σ The reciprocal is NOT true: if two variables have zero correlation, there are NOT necessaril independent. Eercise: find a counter-eample n variables: All the above can be easil generalized to the case of n > variables. The joint probabilit distribution function is f(,,, n f( and the covariance between two variables i and j is: V ij ( i µ i ( j µ j f( n d d d n ( i µ i ( j µ j f(d n The covariance matri can be genericall written: σ cov(, n V.. cov( n, σn Its basic properties are:

3 The diagonal elements are the variances of the individual variables It is smmetric and positive semi-definite (no eigenvalue is negative, see proof in. It is diagonal if (but not onl if! all n variables are independent Its determinant is zero if and onl if a least one linear relation eists between variables: Eercise: variables: Beware of the trap! i Σ i j α j j + β How are covariance cov(, and the correlation factor ρ affected b a change of a + b c + d Global correlation coefficient Consider the correlation ρ( k, Y between the variable k and ever possible linear combination Y of all the other variables i,i k. A useful quantit is the global correlation coefficient ρ k, defined as the largest value of ρ( k, Y. This quantit is a measure of the total amount of correlation between the variable k and all the others. If ρ k 0, then k is uncorrelated with all other variables, while if ρ k, there eists a linear combination of i,i k equal to k. It can be shown that ρ k V kk (V kk Eercise: Let,, z be 3 independent variables of mean 0 and variance. What is the global correlation coefficient of each of the variables,, + + az with respect to the others? See the solution in --> globalcorr.m.3 The joint characteristic function The definition of the multi-variable joint characteristic function is straightforwardl etended from the single variable case: Φ(t, t t n < e it +it + +it n n > Let s consider for simplification the case of variables and let s assume furthermore that the are independent: f(, f ( f (. In this case, Φ(t, t < e it +t > f ( f ( e it +it d d Φ (t Φ (t For independent variables the joint characteristic function is factorizable. In fact the inverse statement also holds: if the joint characteristic function can be factorized, then the variables are independent. 3

4 .4 Change of variables The same formalism applies as in section Let assume that the (multi-variable has the probabilit distribution function f(. Under the change of variables h(, the infinitesimal weight remains constant: f(d g(d Thus, g( f( f( J where J is the jacobian of the transformation. Again this assumes that onl one is solution of the equation h(. We can derive from this a simple but important propert of independent variables: if and are two independent variables, an functions of and onl u( and v( are themselves independent. Proof: g(, f(, u 0 0 v f(, u v f ( u f( v g ( g ( Eercise: Back to the distribution of r r Assuming again that r and r are two random variables uniforml distributed between 0 and, find the probabilit distribution function of r r using the variable change: (r, r (r r, r Eercise: Generating a normall distributed variable. Your computer can generate two sequences of independent random variables and, uniforml distributed between 0 and. Show that the variables ln sin(π ln cos(π are independent and normall distributed. Write a small simulation program to visualize the scatter plot vs and to displa the histogram of. See the sample program: --> CG.m Propagation of Errors. One function The problem we want to address is the following: we have performed the measurement of n quantities i, i,..., n. Assume we know the covariance matri V, which in the general case is NOT diagonal because the measured quantities are not necessaril independent. What will be the variance of an function (... n (? An epansion around µ, the average, gives: ( (µ + ( i µ i µ i i 4

5 Thus, V [(] < [( < ( > ] > < (( (µ > < ( ( i µ i µ > i i j i j < ( i µ i ( j µ j > µ j µ V ij ( j which is known as the law of propagation of errors. In the special case where the variables i are independent, we have V ij δ ij σ i and the law of propagations of errors reduces to V ( i σ i ( Eample: Variance of the arithmetic mean Suppose that the quantities i represent measurements of the same quantit X. The i s are still independent variables since the error ou make at measurement i has nothing to do with that at the measurement i +. If each measurement has an uncertaint σ i, then the variance of is i n i V ( n If all measurements have the same uncertaint σ, which is a common situation, then i σ i v( n σ n σ The uncertaint on the arithmetic mean decreases as / n.. Several functions i Let s now consider the general case of m functions k, k,..., m of the i s. As previousl, for an k, k ( k (µ + ( i µ i k µ and the covariance between k and l is: i V kl (k < [ k ( < k ( >][ l ( < l ( >] > k l < ( i µ i ( j µ j > j i j i j k l j V ij ( 5

6 This is the law of propagations or errors, which can we written in a nice compact wa: where V G V T G V is the m m covariance matri of the k s V is the n n covariance matri of the i s G is the gradient m n matri: G ij i j T G is the transpose of the G matri (n m Diagonalizing the covariance matri: A particular and useful transformation is to take for G the matri which diagonalizes V, i.e. to take for the i s the n eigenvectors of V. In this case, because V is smmetric, G is an unitar matri (G T G. Therefore V G V T G G V G is diagonal. Conclusion: the eigenvectors of the covariance matri are uncorrelated. The eigenvalues of the covariance matri are positive or null, which shows that it is positive semi-definite. Moreover, tr(v tr(g V G tr(v, which means that the sum of the variances remains unchanged when transforming i into i. In the special case where all i s are independent, V ij ( δ ij σi law of propagation of errors reduces to: is a diagonal matri and the V kl ( i σ i ( k ( l Eample: a -dimensional radar One can idealize a radar as a device which measures the polar coordinates r and ϕ of a remote object with uncorrelated errors σ r and σ ϕ. Suppose we are interested in the cartesian coordinates r cos ϕ and r sin ϕ, what is the covariance matri V? Just appling the above formula, with one gets the following epression for V : V V Looking at this matri is quite instructive: ( σ r 0 0 σ ϕ ( cos ϕ r sin ϕ G sin ϕ r cos ϕ ( σ r cos ϕ + r σϕ sin ϕ (σr σϕ r sin ϕ cos ϕ (σr σϕ r sin ϕ cos ϕ σr sin ϕ + r σϕ cos ϕ 6

7 In the case of a radar with infinite precision in ϕ (σ ϕ 0 the covariance matri can be rewritten ( V σr cos ϕ sin ϕ cos ϕ sin ϕ cos ϕ sin ϕ and the correlation coefficient is: ρ sin ϕ cos ϕ ± sin ϕ cos ϕ which means that and are perfectl correlated: the lie on the line ϕ ϕ measured In the case of a radar with infinite precision in r (σ r 0 the covariance matri can be rewritten ( r σϕ V sin ϕ σϕ ( r sin ϕ cos ϕ σϕ r sin ϕ cos ϕ r σϕ σ cos ϕ ϕ and the correlation coefficient is: ρ ± which means that and are perfectl correlated: the lie on the circle r r measured For ϕ 0, π/, π, 3π/ the correlation between and vanishes because of smmetr: the aes of the error ellipse (this term will be justified later on are horizontal and vertical For σ r rσ ϕ, the correlation between and vanishes because of smmetr: the error ellipse becomes a circle. The smbolic program used to generate the output in L A TEXformat is shown here: --> RADAR.m 3 The multi-normal distribution 3. Independent variables Let s start with the ver simple case of two independent variables. The Bi-normal distribution can be written as the product of two elementar normal distributions, since the variables are independent: f(, e ( µ π σ σ e ( µ σ π σ π V e T ( µ V ( µ where V is the covariance matri of the i s: ( σ V 0 0 σ In the case of n independent variables, the epression is slightl modified: f(,..., n ( π n e T ( µ V ( µ V 7

8 3. General case Let s show that the above formula is still valid in the case of correlated i s. To simplif, let s assume that the i s all have a zero average value (if not a simple change of variable will ensure it. As seen above (in paragraph. one can diagonalize the V matri with an unitar matri U (i.e. T U U, and the eigenvectors U will be uncorrelated. Let s assume the will be independent as well. Then, f( ( π n V e T V Performing the variable change, and noting that T T T U T U and V UV U (since U diagonalizes V, one gets: f( T U ( π n V e T U U V T U U ( π n e T V V QED 3.3 Properties of the multi-normal distribution The multi-normal distribution ehibits the following properties:. It is a function of means, variances and covariances onl.. The variables are independent if and onl if the covariance matri is diagonal (we saw that in general a diagonal covariance matri does not impl the independence of the variables. 3. (hpersurfaces of equiprobabilit are (hperellipses. 4. A projection onto one or several variables is obtained b removing from the covariance matri V the row(s and column(s of the other variables 5. A cut (setting one or several variables to a fied value is obtained b removing from the inverse of the covariance matri V the corresponding row(s and column(s. 6. A set of variables made up as linear combinations of the i s is also multi-normall distributed. In particular, the distribution of the single variable a i i T A is N( T A, T A V A 7. Consider N independent observations and form the quantities: i N N il l s ij N N ( il il ( jl jl l i and s ij are independent if and onl if the i s are distributed according to a multi-normal distribution 8

9 Eample: the -dimensional case In the case of the bi-normal distribution, the covariance matri and its inverse can be written: ( σ V ρσ σ ρσ σ σ ( V σ ρσ σ σσ ( ρ ρσ σ This gives the length epression for the joint probabilit (again assuming µ 0 and µ 0: σ f(, πσ σ ρ e ( ρ ( σ + σ ρ σσ Projection: projecting on, for eample, will give a normal distribution of mean 0 and standard deviation σ Cut: cutting at a given 0 will give a narrower and shifted distribution for : σ ρ σ µ µ + ρ σ σ ( 0 µ Eercise: Generating correlated gaussian variables As seen in section..4, it is eas to generate two sequences of normall distributed variables z and z. Show that the quantities + ρ ρ z + + ρ ρ z are normall distributed, and correlated with the correlation factor ρ. Write a MATLAB program to visualize the scatter plot vs for different values of ρ :,.8,.6,.4,.,.. See --> CORREL.m, CORREL.m, Multinornal.m Generating correlated variables from other distributions: the described above algorithm works onl for gaussian distributions because the posses the unique propert that if, are gaussian, then α + β will remain gaussian. A popular though CPU demanding algorithm for generating correlated variables is the so called Cario/Nelson algorithm:. Find the function which transforms z, a variable normall distributed to another variable, distributed according to the desired distribution. Generate a set of z, z pairs according to the above variable 3. Transform z into z. The will have a slightl different correlation coefficient which can be accounted for z z 9

0.24 adults 2. (c) Prove that, regardless of the possible values of and, the covariance between X and Y is equal to zero. Show all work.

0.24 adults 2. (c) Prove that, regardless of the possible values of and, the covariance between X and Y is equal to zero. Show all work. 1 A socioeconomic stud analzes two discrete random variables in a certain population of households = number of adult residents and = number of child residents It is found that their joint probabilit mass