PROBABILITY 2018 HANDOUT 2: GAUSSIAN DISTRIBUTION

Transcription

1 PROAILITY 218 HANDOUT 2: GAUSSIAN DISTRIUTION GIOVANNI PISTONE Contents 1. Standard Gaussian Distribution 1 2. Positive Definite Matrices 5 3. General Gaussian Distribution 6 4. Independence of Jointly Gaussian Random Variables 7 References 8 This handout covers multivariate Gaussian distributions and the relevant matrix theory. Two classical references are [3] and [1] (many reprints available). A modern advanced reference for positive definite matrices is [2]. 1. Standard Gaussian Distribution 1 (Change of variable formula in R d ). Let A, Ă R d be open and φ be a diffeomerphism from A onto. Let Jφ: A Ñ Mat pd ˆ dq be the Jacobian mapping of φ and Jφ 1 : Ñ Mat pd ˆ dq the Jacobian mapping of φ 1, so that Jφ 1 pjφ φ 1q 1. For each nonnegative f : Ñ R n, fpyq dy f φpxq det pjφpxqq dx A Exercise 1. A s, 2πrˆs, `8r, R 2 R 2 z tpx, yq P R 2 x ě, y u, φpθ, ρq pρ cos θ, ρ sin θq. j ρ sin θ cos θ Jφpθ, ρq, det pjφpθ, ρqq ρ ρ cos θ sin θ ij e px2`y 2 q{2 dxdy ij e pρ2 cos 2 θ`ρ 2 sin 2 θq{2 ρ dθdρ R 2 s,2πrˆs,`8r ij e ρ2 {2 ρ dθdρ 2π s,2πrˆs,`8r 2. (Image of an absolutely continous measure) Let ps, F, µq be measure space, p: S Ñ R ą a probability density, px, Gq a measurable space, φ: S Ñ X a measurable function. If φ has a measurable inverse, then the image measure is characterised by f dφ # pp µq pf φqp dµ pf φqpp φ 1 φq dµ fp φ 1 dφ # µ Date: Version March 22,

2 hence φ # pp µq pp φ 1 q µ. Eq. (1) applied to f φ and the diffeomorphism φ 1 gives f dpφ # lq f φpxq dx f φ φ 1 pyq ˇˇdet `Jφ 1 pyq ˇˇ dy A fpyq ˇˇdet `Jφ 1 pyq ˇˇ dy fpyq ˇˇdet `Jφ φ 1 pyq ˇˇ 1 dy This shows that the image of the Lebesgue measure l under a diffeomorphism is φ # l ˇˇdet `Jφ 1 ˇˇ l ˇˇdet `Jφ φ 1 ˇˇ 1 l Exercise 2. A s, 1rˆs, 1r, R 2, φpu, vq p? 2 log u cosp2πvq,? 2 log u sinp2πvqq,» 1 Jφpu, vq 2 p 2 log uq 1{2 2 fi u cosp2πvq 2π? 2 log u sinp2πvq 1 2 p 2 log uq 1{2 2 ffi u sinp2πvq 2π? 2 fl, log u cosp2πvq det pjφpu, vqq 2π u, det `Jφ φ 1 2π px, yq e. px2`y2 q{2 The image of the uniform probability measure on s, 1r 2 under φ is p2πq 1 e px2`y 2q{2 dxdy. 3 (Marginalization). The previous argument does not apply when Φ is not 1-to-1. We will show in the chapter on conditioning that in such a case Φ # pp µq ˆp Φ # pµq where ˆp is the conditional expectation of p with respect to Φ. However, there are two common and simple cases namely, the finite state space case and the marginalisation. Assume µ µ 1 b µ 2 on S S 1 ˆ S 2 and consider the marginal projection Φ: px 1, x 2 q ÞÑ x 1. Then Φ 1 pa 1 q A 1 ˆ S 2 and µpφ 1 pa 1 qq µpa 1 ˆ S 2 q µ 1 pa 1 q hence, Φ # pµq µ 1. Let p be a density on S with respect to µ. For each positive f : S 1 we have ij f dφ # pp µq f Φ dpp µq fpx 1 qppx 1, x 2 q µpdx 1, dx 2 q ˆ fpx 1 q ppx 1, x 2 q µ 2 pdx 2 q µ 1 pdx 1 q so that Φ # pp µq p 1 px 1 q µ 1, p 1 px 1 q ppx 1, x 2 q µ 2 pdx 2 q For example, if ppx 1, x 2 q p2πq 1 e px2 1`x2 2 q{2, then ppx 1, x 2 q dx 2 p2πq 1{2 e x2 1 {2 p2πq 1{2 e x2 2 {2 dx 2 cp2πq 1{2 e x2 1 {2 with c ş p2πq 1{2 e x2 2 {2 dx 2 1 as the further integration with respect to dx 1 shows. Notice that the argument applies to all ppx 1, x 2 q cfpx 1 qfpx 2 q. 4. The real random variable Z is standard Gaussian, Z N 1 p, 1q, if its distribution ν has density ˆ R Q z ÞÑ γpzq p2πq 1 2 exp 1 with respect to the Lebesgue measure. It is in fact a density, see above the computation of its two-fold product. 2 2 z2

3 Exercise 3. All moments µpnq ş z n γpzq dz exists. As zγpzq γ 1 pzq, integration by parts produces a recurrent relation for the moments. [Hint: Write ş ş z n γpzq dz z n 1 zγpzq dz ş z n 1 p γ 1 pzqq dz and perform an integration by parts] Exercise 4. If f : R Ñ R absolutely continuous i.e., fpzq fpq ` şz ş f 1 puq du, with f 1 puq γpuq du ă `8 then ş zfpzq γpzq dz ă `8. In fact, ˆ z zfpzq γpzq dz ˇˇˇˇz fp ` f 1 puq duˇˇˇˇ γpzq dz ď z f pq z γpzq dz ` ˇˇˇˇz f 1 puq du ˇ γpzq dz. The first term in the RHS equals a 2{π fpq, while in the second term we have for z ě, z ˇ f 1 puq du ˇ ď p ď u ď zq f 1 puq du. We have ˇˇˇˇz z ˆ f 1 puq du ˇ γpzq dz ď z 8 f 1 puq zγpzq dz du f 1 puq u p ď u ď zq f 1 puq du 8 u γpzq dz p γ 1 pzqq dz du f 1 puq γpuq du ă 8. A similar argument applies to the case z ď. This implies zfpzqγpzq dz fpzqp γ 1 pzqq dz f 1 pzq γpzqdz. Exercise 5. The Stein operator is δfpzq zfpzq f 1 pzq. We have fpzqg 1 pzqγpzq dz δf pzqgpzqγpzqdz We define the Hermite polynomials to be H n pzq δ n 1. For example, H 1 pzq z, H 2 pzq z 2 1, H 3 pzq z 3 3z. Hermite polynomials are orthogonal with respect to γ, H n pzqh m pzqγpzq dz if n ą m. 5. Let Z N 1 p, 1q, Y b ` az, a, b P R. Then E pxq b, E px 2 q a 2 ` b 2, Var pxq a 2. If a, then φpzq b ` az is a diffeomorphism with inverse φ 1 pxq a 1 px bq, hence the density of X is ˆ 1 γpa 1 px bqq a 1 p2πa 2 q 1{2 exp px bq2 2a2 If a then the distribution of X b is the Dirac measure at b. We say that X is Gaussian with mean b and variance a 2, X N 1 pb, a 2 q. Viceversa, if X N 1 pµ, σ 2 q and σ 2 1, then Z σ 1 px µq N 1 p, 1q. 6. The characteristic function of a probability measure µ is pµptq e itx µpdxq cosptxq µpdxq ` i sinptxq µpdxq, i? 1 If two probability measure have the same characteristic function, then they are equal. 3

4 Exercise 6. For the standard Gaussian probability measure we have pγptq cosptzq γpzqdz e t2 2. In fact, by derivation under the integral d dt pγptq z sinptzq γpzqdz sinptzqγ 1 pzq dz tγptq and pγpq 1. The characteristic function of X N 1 pµ, σ 2 q is E `e itx E `e itpµ`σzq e itµ E e ipσt qz e tµ` 1 2 σ2 t 2 Exercise 7. The characteristic function pµ of the probability measure µ on R is nonnegative definite. Take t 1,..., t n in R with n 1, 2,.... The matrix j n T rpµpt i t j qs n i,j 1 e ipt i t j qx µpdxq is Hermitian, that is the transposed matrix is equal to the conjugate matrix equivalently, T is equal to its adjoint T. An Hermitian matrix T is non-negative definite if for all complex vector ζ P C n it holds ζ T ζ ě. In our case j nÿ ζ e ipt i t j qx µpdxq ζ ζ i ζ j e ipt i t j qx µpdxq i.j 1 nÿ i.j 1 i,j 1 ζ i e itix ζ j e it jx nÿ µpdxq ζ i e it ix Exercise 8. let X N 1 pb, σ 2 q and f : R Ñ R continuous and bounded. lim σñ E pfpxqq fpbq. i 1 2 µpdxq ě. Show that Exercise 9. Let X be a real random variable with density p with respect to the Lebesgue measure, and let Z N 1 p, 1q. Assume X and Z are independent i.e., the joint random variable px, Zq has density pbγ with respect to the Lebesgue measure of 2. Compute the density of X ` Z. [Hint: make a change of variable px, zq ÞÑ px ` z, zq then marginalize.] 7. The product of absolutely continuous probability measures is pp 1 µ 1 q b pp 2 µ 2 q pp 1 b p 2 q µ 1 b µ 2 The R d -valued random variable Z pz 1,..., Z d q is multivariate standard Gaussian, Z N n p d, I d q if its components are IID N 1 p, 1q. We write ν d ν bd to denote the d-fold product measure. The distribution ν d γ bd of Z N n p, Iq has the product density nź ˆ R n Q z ÞÑ γpzq φpz j q p2πq n 2 exp 1 2 }z}2 j 1 Exercise 1. The moment generating function t ÞÑ E pexp pt Zqq P R ą is nź ˆ1 ˆ1 R n Q t ÞÑ M Z ptq exp 2 t2 i exp 2 }t}2 j 1 M Z is everywhere strictly convex and analytic. 4

5 Exercise 11. The characteristic function ζ ÞÑ pγ n pζq E `exp `? 1ζ Z is 2ź ˆ ˆ R n Q ζ ÞÑ pγ n pζq exp 1 2 ζ2 i exp 1 2 }ζ}2 pγ n is non-negative definite. j 1 2. Positive Definite Matrices 8. We collect here a few useful properties of matrices. denotes transposition. (1) Denote by Mat pm ˆ nq the vector space of mˆn real matrices. We have Mat pm ˆ 1q Ø R m. Let Mat pn ˆ nq be the vector space of n ˆ n real matrices, GLpnq the group of invertible matrices, Sympnq the vector space of real symmetric matrices. (2) Given A P Mat pn ˆ nq, a real eigen-value of A is a real number λ such that A λi is singular i.e., det pa λiq. If λ is an eigen-value of A, u an eigen-vector of A associated to λ if Au λu. (3) y identifying each matrix A P Mat pm ˆ nq with its vectorized form vecpaq P R mn, the vector space Mat pm ˆ nq is an Euclidean space for the scalar product xa, y vecpaq vecpq Tr pa q. The general linear group GLpnq is an open subset of Mat pn ˆ nq. (4) A square matrix whose columns form an orthonormal system, S rs 1 s n s, s i s j pi jq, has determinant 1. The property is characterised by S S 1. The set of such matrices is the orthogonal group Opnq. (5) Each symmetric matrix A P Sympnq has n real eigen-values λ i, i 1,..., n and correspondingly an orthonormal basis of eigen-vectors u i, i 1,..., n. (6) Let A P Mat pm ˆ nq and let r ą be its rank i.e., the dimension of the space generated by its columns, equivalently by its rows. There exist matrices S P Mat pm ˆ rq, T P Mat pn ˆ rq, and a positive diagonal r ˆ r matrix Λ, such that S S T T I r, and A SΛ 1{2 T. The matrix SS is the orthogonal projection onto image A. In fact image SS image A, SS A A, and SS is a projection. Similarly, T T is the ortogonal projection unto image A. (7) A symmetric matrix A P Sympnq is positive definite, A P Sym`pnq, respectively strictly positive definite, A P Sym``pnq, if x P R n implies x 1 Ax ě, respectively ą. Sym`pnq is a closed pointed cone of Sympnq, whose interior is Sym``pnq. A positive definite matrix is strictly positive definite if it is invertible. (8) A symmetric matrix A is positive definite, respectively strictly positive definite, if, and only if, all eigen-values are non-negative, respectively positive. (9) A symmetric matrix is positive definite if, and only if, A 1 for some P M n. Moreover, A P GL n if, and only if, P GL n. (1) A symmetric matrix A is positive definite if, and only if A 2 and is positive definite. We write A 1 2 and call the positive square root of A. Exercise 12. If you are not familiar with the previous items, try the following exercise. Consider the matrices j cos θ sin θ Rpθq, θ P R. sin θ cos θ Check that Rpθq Rpθq I, det Rpθq 1, and Rpθ 1 qrpθ 2 q Rpθ 1 ` θ 2 q. Compute the matrix j λ1 Σpθq Rpθq Rpθq, λ λ 1, λ 2 ě. 2 5

6 Chech that det Σpθq λ 1 λ 2, Σpθq Σpθq, the eigenvalues of Σpθq are λ 1, λ 2, and ΣpθqRpθq Rpθq diag pλ 1, λ 2 q. Compute «ff λ 1{2 1 Apθq Rpθq λ 1{2 Rpθq, λ 1, λ 2 ě. 2 Check that ApθqApθq ApθqApθq Σpθq. Exercise 13. Let A P Opnq and Z N n p, Iq. Check that AZ N n p, Iq. let P Mat pn ˆ rq, r ă n, and assume that the columns are orthonormal. Check that Z N r p, Iq. [Hint: complete to an orthogonal matrix by adding columns, r Cs P Opnq and use the marginalization.] 1 Exercise 14. Let Z N 1 p, 1q, A P Mat p2 ˆ 1q. Check that AZ has no density 1j with respect to the Lebesgue measure. Exercise 15. Let Z N 2 p, Iq, A 1 1 P Mat p1 ˆ 2q. Compute the density od AZ. Proposition General Gaussian Distribution (1) Definition Let Z N n p, Iq, A P Mat pm ˆ nq, b P R m, Σ AA. Then Y b ` AZ has a distribution that depends on Σ and b only. The distribution of Y is called Gaussian with mean b and variance Σ, N m pb, Σq. (2) Statility If Y N m pb, Σq, P Mat pr ˆ mq, c P R r, then c`y N r pc ` b, Σ q. (3) Existence Given any non-negative definite Σ P Sym`pnq and any vector b P R n, the Gaussian distribution N n pb, Σq exists. (4) Density If Σ P Sym``pnq e.g., Σ P Sym`pnq and moreover det pσq, then the Gaussian distribution N m pb, Σq, has a density with respect to the Lebesgue measure on R n given by given by ˆ p Y pyq p2πq m 2 det pσq 1 2 exp 1 2 py bqt Σ 1 py bq. (5) No density If the rank of Σ is r ă m, then the distribution of N m pb, Σq is supported by the image of Σ. In particular it has no density w.r.t. the Lebesgue measure on R n. (6) Characteristic function Y N m pb, Σq if, and only if, the characteristic function is ˆ R m Q t ÞÑ exp 1 2 t Σt ` ib t Proof. (1) Assume b 1, b 2 P R m, A i P Mat pm ˆ n i q, Y i b i ` A i Z i, Z i N ni p, Iq, i 1, 2. If b 1 b 2 then the expected values of Y 1 and Y 2 are different, hence the distribution is different. Assume b 1 b 2 b, and consider the distribution of Y i b A i Z i, i 1, 2. We can write A i S i Λ 1{2 i Ti, which in turn implies implies Σ S i ΛS i, but Σ SΛS, hence S 1 S 2 S and Λ 1 Λ 2 Λ (a part the order). We are reduced to the case Y i b SΛTi Z i, T i P Mat pn i ˆ rq with both with orthonormal columns. The conclusion follows from T 1 Z 1 T 2 Z 2. 6

7 (2) Y N m pb, Σq means Y b ` AZ with Z N n p, Iq and AA Σ. It follows c ` Y c ` pb ` AZq pc ` bq ` paqz, wth paqpaq AA Σ. (3) Take Y b ` Σ 1{2 Z, Z N n p, Iq. (4) Use the change of variable formula in Y b ` AZ with A Σ 1{2 to get p Y pyq ˇˇdet `A 1 ˇˇ p Z pa 1 py bqq. The express each term with Σ. (5) use the decomposition Σ SΛS and note that some elements on the diagonal of Λ are zero. (6) The if part is a computation, the only if part requires the injection property of characteristic function. Exercise 16 (Linear interpolation of the rownian motion). Let Z n, n 1, 2... be IID N 1 p, 1q. Given ă σ! 1, define recursively the times t and t n`1 t n ` σ 2. Let T tt n n, 1,...u. Define recursively pq, pt n`1 q pt n q ` σz n. As pt n q ř n i 1 σz i σ ř n i 1 Z i, then Var ppt n qq σ 2 Var p ř n i 1 Z iq nσ 2 t n. For each t P R ą zt, define ptq by linear interpolation i.e., ptq t n`1 t pt n q ` t t n pt n`1 q, t P rt n, t n`1 s. t n`1 t n t n`1 t n Compute the variance of ptq and the density of ptq. 4. Independence of Jointly Gaussian Random Variables Proposition 2. Consider a partitioned Gaussian vector j ˆ j Y1 b1 Σ11 Y N Y n1`n 2, 2 b 2 j Σ 12 Σ 22. Let r i Rank pσ ii q, Σ ii S i Λ i S i with S i P Mat pn i ˆ r i q, S i S I ri, and Λ i P diag`` pr i q, i 1, 2. (1) The blocks Y 1, Y 2 are independent, Y 1 K Y 2, if, and only if, Σ 12, hence Σ 21 Σ 12. More precisely, if, and only if, there exist two independent standard Gaussian Z i N ri p, Iq and matrices A i P Mat pn i ˆ r i q, i 1, 2, such that j j j Y1 A1 Z1. Y 2 A 1 Z 2 (2) (The following property is sometimes called Schur complement lemma.) Write Σ`22 S Λ S 2. Then, j j j I Σ12 Σ`22 Σ11 Σ 12 I I Σ 21 Σ 22 Σ`22Σ 21 I j j Σ11 Σ 12 Σ`22Σ 21 I Σ 21 Σ 22 Σ`22Σ 21 I j Σ11 Σ 12 Σ`22Σ 21, Σ 22 7 Σ 21

8 hence the last matrix is non-negative definite. The Shur complement of the partitioned covariance matrix Σ is Σ 1 2 Σ 11 Σ 12 Σ`22Σ 21 P Sym`pn 1 q. (3) Assume det pσq. Then both det `Σ 1 2 and det pσq22. If we define the partitioned concentration to be j K Σ 1 K11 K 12, K 22 then K 11 Σ K 21 and K 1 11 K 12 Σ 12 Σ Exercise 17. Let Σ P Sym`pnq and let r Rank pσq. We know that Σ SΛS with S P Mat pn ˆ rq, S S I r, λ P diag`` prq. Let us define Σ` SΛ 1 S. Then it follows by simple computation that Σ`Σ ΣΣ` SS. Also, ΣΣ`Σ Σ and Σ`ΣΣ` Σ`. If Y N n p, Σq, then Y SS Y. In fact, Y SS Y pi SS qy is a Guassian random variable with variance pi SS qsλs pi SS q because pi SS qs S SS S S S. Proof. (1) If the blocks are independent, they are uncorrelated. Conversely, if Σ ii S i Λ i S i, i 1, 2, define A i S i Λ 1{2 i to get j j A1 A1 Σ. A 2 A 2 (2) Computations using Ex. 17. (3) From the computation above we see that the Schur complement is positive definite and that ˆ j Σ11 Σ det 12 det `Σ Σ 21 Σ 1 2 det pσ22 q. 22 It follows that det pσq implies both det `Σ 1 2 and det pσ22 q. The condition j j j K11 K 12 Σ11 Σ 12 I K 21 K 22 Σ 21 Σ 22 I is equivalent to I K 11 Σ 11 ` K 12 Σ 21 K 11 Σ 12 ` K 12 Σ 22. Right-multiply the second equation by Σ 1 22 and substitute in the first one, to get K 11 Σ 1 2 I, hence K 1 11 Σ 1 2. The other equality follows by left-multiplying the second equation by K Exercise 18 (Whitening). Let Y N n pb, Σq. Assume Σ has rank r and decomposition Σ SΛS, S S I r, λ P diag`` prq. Then Z Λ 1{2 S py bq ia a white noise, Z N r po, Iq. Moreover, b ` SΛ 1{2 Z Y. In fact, Y pb ` SΛ 1{2 Zq py bq SΛ 1{2 Λ 1{2 S py bq pi SS qpy bq. 8

9 References [1] T. W. Anderson, An introduction to multivariate statistical analysis, third ed., Wiley Series in Probability and Statistics, Wiley-Interscience [John Wiley & Sons], Hoboken, NJ, 23. MR [2] Rajendra hatia, Positive definite matrices, Princeton Series in Applied Mathematics, Princeton University Press, Princeton, NJ, 27. MR (27k:155) [3] Paul R. Halmos, Finite-dimensional vector spaces, The University Series in Undergraduate Mathematics, D. Van Nostrand Co., Inc., Princeton-Toronto-New York-London, 1958, 2nd ed. MR Collegio Carlo Alberto address: 9