Properties and approximations of some matrix variate probability density functions

Size: px

Start display at page:

Download "Properties and approximations of some matrix variate probability density functions"

Melina Patterson
5 years ago
Views:

1 Technical report from Automatic Control at Linköpings universitet Properties and approximations of some matrix variate probability density functions Karl Granström, Umut Orguner Division of Automatic Control 16th December 011 Report no.: LiTH-ISY-R-304 Address: Department of Electrical Engineering Linköpings universitet SE Linköping, Sweden WWW: AUTOMATIC CONTROL REGLERTEKNIK LINKÖPINGS UNIVERSITET Technical reports from the Automatic Control group in Linköping are available from

2 Abstract This report contains properties and approximations of some matrix valued probability density functions. Expected values of functions of generalised Beta type II distributed random variables are derived. In two Theorems, approximations of matrix variate distributions are derived. A third theorem contain a marginalisation result. Keywords: Extended target, random matrix, Kullback-Leibler divergence, inverse Wishart, Wishart, generalized Beta.

3 Properties and approximations of some matrix variate probability density functions Karl Granström, Umut Orguner Abstract This report contains properties and approximations of some matrix valued probability density functions. Expected values of functions of generalised Beta type II distributed random variables are derived. In two Theorems, approximations of matrix variate distributions are derived. A third theorem contain a marginalisation result. 1 Some matrix variate distributions 1.1 Wishart distribution Let S d ++ be the set of symmetric positive denite d d matrices. The random matrix X S d ++ is Wishart distributed with degrees of freedom n d and d d scale matrix N S d ++ if it has probability density function pdf 1, Denition 3..1] px W d X ; n, N 1 X n d 1 etr 1 N 1 X, nd Γ d n N n where, for a > d 1, the multivariate gamma function, and its logarithm, can be expressed in terms of the ordinary gamma function as 1, Theorem 1.4.1] d Γ d a π dd 1 Γ a i 1/, log Γ d a dd 1 log π + log Γ a i 1/. 3a 3b Let A ij denote the i, j:th element of a matrix A. The expected value and covariance of X are 1, Theorem ] EX ij ] nn ij, 4 CovX ij, X kl nn ik N jl + N il N jk. 5 1

4 1. Inverse Wishart distribution The random matrix X S d ++ is inverse Wishart distributed with degrees of freedom v > d and inverse scale matrix V S d ++ if it has pdf 1, Denition 3.4.1] px IW d X ; v, V 6 v d 1 V v d 1 etr 1 X 1 V. 7 Γ d v d 1 X v The expected value and covariance of X are 1, Theorem 3.4.3] V ij EX ij ], v d > 0, 8 v d CovX ij, X kl v d 1 V ij V kl + V ik V jl + V il V jk, v d 4 > 0. v d 1v d v d Generalized matrix variate Beta type II distribution Let S d + be the set of symmetric positive semi-denite d d matrices. The random matrix X S d ++ is generalized matrix variate Beta type II distributed with matrix parameters Ψ S d + and Ω > Ψ, and scalar parameters a and b, if it has pdf 1, Denition 5..4] px GB II d X; a, b, Ω, Ψ 10 X Ψ a d+1 Ω + X a+b β d a, b Ω + Ψ b, X > Ψ 11 where, for a > d 1 and b > d 1, the multivariate beta function is expressed in terms of the multivariate gamma function as 1, Theorem 1.4.] β d a, b Γ daγ d b Γ d a + b. 1 Let 0 d be a d d all zero matrix. If Ψ 0 d, the rst and second order moments of X are 1, Theorem 5.3.0] a EX ij ] b d 1 Ω ij 13 a EX ij X kl ] b db d 1b d 3 {ab d + } Ω ijω kl +a + b d 1Ω jl Ω ik + Ω il Ω kj ], b d 3 > 0 14

5 Expected values of the GB II d -distribution This appendix derives some expected values for the matrix variate generalized beta type-ii distribution..1 Expected value of the inverse Let U be matrix variate beta type-ii distributed with pdf 1, Denition 5..] pu B II d U; a, b 15 d+1 U a I d + U a+b β d a, b 16 where a > d 1, b > d 1, and I d is a d d identity matrix. Then U 1 has pdf 1, Theorem 5.3.6] pu 1 Bd II U 1 ; b, a. 17 Let X Ω 1/ UΩ 1/ where Ω S d ++. The pdf of X is 1, Theorem 5..] px GB II d X; a, b, Ω, 0 d 18 and subsequently the pdf of X 1 Ω 1/ U 1 Ω 1/ is px 1 GB II d X 1 ; b, a, Ω 1, 0 d 19 The expected value of X 1 is 1, Theorem 5.3.0] E X 1] b a d 1 Ω Expected value of the log-determinant Let y be a univariate random variable. The moment generating function of y is dened as µ y s E y e sy ], 1 and the expected value of y is given in terms of µ y s as Ey] dµ ys ds. s0 3

6 Let y log X, where px Bd II X; a, b. The moment generating function of y is µ y s E X s ] X s p X dx 3a X s β 1 d a, b X a 1 d+1 I d + X a+b dx 3b β 1 d a, bβ da + s, b s β 1 d a + s, b s X a+s 1 d+1 I d + X a+s+b s dx 3c β da + s, b s Bd II X; a + S, b s dx 3d β d a, b β da + s, b s β d a, b Γ da + sγ d b s Γ d a + s + b s Γ d a + b Γ d aγ d b 3e Γ da + sγ d b s, 3f Γ d aγ d b The expected value of y is E y] E log X ] d Γ d a + sγ d b s 1 ds Γ d aγ d b s0 dγd a+s dγdb s ds Γ d a + s + ds Γ d b s s0 d log Γd a + s + d log Γ db s ds ds + d log Γa + s i 1/ ds s0 ψ 0 a i 1/ ψ 0 b i 1/, d log Γb s i 1/ ds 4a 4b 4c 4d s0 4e 4f where ψ 0 is the digamma function, also called the polygamma function of order zero. If py GB II d Y ; a, b, Ω, 0 d, then Z Ω 1/ Y Ω 1/ has pdf Bd II Z; a, b 1, Theorem 5..]. It then follows that ] E log Y ] E log Ω 1/ Ω 1/ Y Ω 1/ Ω 1/ 5a ] E log Ω 1/ + log Ω 1/ Y Ω 1/ + log Ω 1/ 5b ] E log Ω + log Ω 1/ Y Ω 1/ 5c log Ω + E log Z ] 5d ] log Ω + ψ 0 a i 1/ ψ 0 b i 1/. 5e 4

7 3 Approximating a GB II d -distribution with an IW d - distribution This section presents a theorem that approximates a GB II d -distribution with an IW d -distribution. 3.1 Theorem 1 Theorem 1. Let px GB II d X; a, b, Ω, 0 d, and let qx IW d X ; v, V be the minimizer of the Kullback-Leibler kl divergence between p X and q X among all IW d -distributions, i.e. Then V is given as q X arg min KL p X q X. 6 q IW d V and v is the solution to the equation a + 1 i ψ 0 +ψ 0 v d i v d 1a d 1 Ω, 7 b b + 1 i ψ 0 ] d log v d 1a d 1 0, 8 4b where ψ 0 is the digamma function a.k.a. the polygamma function of order Proof of Theorem 1 The density q X is given as q X arg min qx arg max qx arg max qx KL p X q X log Γ d v d 1 arg max qx p X log q X dx v d 1d p X v d 1d log Γ d v d 1 arg max qx fv, V log + v d 1 log V ] dx v log X + Tr 1 X 1 V log + v d 1 v E p log X ] + Tr log V 1 E p X 1 ] V 9a 9b 9c 9d 9e 5

8 Dierentiating the objective function fv, V with respect to V gives dfv, V dv Setting to zero and solving for V gives v d 1 V 1 1 E p X 1 ]. 30 V v d 1 E p X 1 ] 1 v d 1a d 1 Ω 31 b where the expected value is calculated based on a result derived in Section. Dierentiating the objective function with respect to v gives dfv, V d dv log + 1 log V d log Γ v d 1 d 1 dv E p log X ] 3a d log + 1 log V 1 v d i ψ 0 1 E p log X ]. 3b Setting the result equal to zero gives v d i 0 log V d log ψ 0 E p log X ] v d i log V d log ψ 0 log Ω ψ 0 a 1 i 1 ψ 0 b 1 ] i 1 33a 33b where the expected value of log X is derived in Section. Inserting V from 31 gives v d 1a d 1 0 log Ω + d log d log b ] a + 1 i b + 1 i log Ω ψ 0 ψ 0 v d 1a d 1 d log 4b a + 1 i ψ 0 which is the equation for v in the theorem. ] b + 1 i v d i ψ 0 + ψ 0 v d i ψ

9 3.3 Corollary to Theorem 1 Corollary 1. A closed form solution for v can be obtained using only 7 together with matching the rst order moments. The expected values of the densities p and q are 1, Theorems 5.3.0, 3.4.3] a E p X] b d 1 Ω, V E q X] v d v d 1 v d Equating the expected values and solving for v gives a d 1 v d Remarks to Theorem 1 b d 1 a d 1 b a b d 1 b a 36a a d 1 Ω. 36b b. 37 The equations for V 7 and v 8 in Theorem 1 correspond to matching the expected value of X 1 and log X, E q X 1 ] E p X 1 ], E q log X ] E p log X ]. 38a 38b Notice that in Theorem 1, substituting a value for v into 7 gives the analytical solution for V. The parameter v can be found by applying a numerical root- nding algorithm to 8, see e.g., Section 5.1]. Examples include Newton- Raphson or modied Newton algorithms, see e.g., Section 5.4], for more alternatives see e.g., Chapter 5]. In the following corollary, we supply an alternative to root-nding to obtain a value for v. Matching the expected values, as in Corollary 1, can be seen as an approximation of matching the expected values of the log determinant 38b. Indeed, with numerical simulations one can show that the v given by 37 is approximately equal to the solution of 8, the dierence is typically on the order of one tenth of a degree of freedom. References 3, 1, 4] contain discussions about using moment matching to approximate a GB II d -distribution with a IW d -distribution. Theorem 1 denes an approximation by minimising the kl divergence, which results in matching the expected values 38. The kl criterion is well-known in the literature for its moment-matching characteristics, see e.g. 5, 6]. 7

10 4 Approximating the density of V x with a W d - distribution This section shows how the distribution of a matrix valued function of the kinematical target state x can be approximated with a Wishart distribution. 4.1 Theorem Theorem. Let x be Gaussian distributed with mean m and covariance P, and let V x Vx S nx ++ be a matrix valued function of x. Let pv x be the density of V x induced by the random variable x, and let qv x W d V x ; s, S be the minimizer of the kl-divergence between pv x and qv x among all W- distributions, i.e. Then S is given as and s is the solution to the equation s d log qv x arg min KL p V x q V x. 39 q W S 1 s C II 40 s i + 1 ψ 0 + C I log C II 0 41 where C I E log V x ] and C II E V x ]. 8

11 4. Proof of Theorem The density q V x is q V x arg min qv x arg max qv x arg max qv x KL p V x q V x p V x log q V x dv x s s d 1 log S + arg max p x qv x s arg max qv x q V x arg max qv x p V x sd s log log Γ d log S + s d 1 + s d 1 sd log log Γ d + s d 1 log V x + Tr 1 ] S 1 V x dv x sd log log Γ d E x sd log log Γ d s log V x + Tr 1 ] S 1 V x dx s s log S log V x + Tr 1 ] S 1 V x s s E x log V x ] + Tr log S 1 S 1 E x V x ] Let C I E x log V x ] and C II E x V x ]. This results in sd s log log Γ d s arg max qv x 4a 4b 4c 4d 4e. 4f s d 1 log S + C I + Tr f s, S. 43 Dierentiating the objective function f s, S with respect to S, setting the result equal to zero and multiplying both sides by gives ss 1 + S 1 C II S 1 0 S 1 s C II. 44 Note that the expected value for V x under the Wishart distribution q is ss C II. Thus the expected value under q is correct regardless of the parameter s. Dierentiating the objective function f s, S in 43 with respect to s gives df s, S d ds log 1 s i + 1 ψ 0 1 log S + 1 C I 45 d log s 1 ψ 0 s i log C II + 1 C I 46 where we substituted S with 44 to obtain 46. Equating the result to zero and multiplying both sides by gives 41 in Theorem. 1 S 1 C II 9

12 4.3 First corollary to Theorem Corollary. C I and C II can be calculated using a Taylor series expansion of V x around x m. A third order expansion yields d log V x C I log V m + dx j1 j dx i P ij, xm d V x C II V m + dx j dx i P ij. xm j1 47a 47b In 47 the i:th element of the vector x and the i, j:th element of the matrix P are x i and P ij, respectively. Moreover, the matrix V m is the function V x evaluated at the mean m of the random variable x. The expected values taken via second order Taylor expansions of log V x and V x around x m are n x d log V x E x log V x ] E x log V m + dx i x i m i xm n x d ] log V x + dx j1 j dx i x i m i x j m j xm n x d log V x log V m + dx j dx i P ij xm C I, j1 48a 48b and dv x E x V x ] E x V m + dx i x i m i xm d ] V x + dx j1 j dx i x i m i x j m j xm d V x V m + dx j dx i P ij xm C II. j1 49a 49b Note that this is equal to a third order Taylor series expansion, because the addition of the third order Taylor series terms would not change the results above because all odd central moments of the Gaussian density are zero. Hence, the error of the above approximation is of the order O E p x m 4, i.e. the error of a third order Taylor series expansion. 10

13 4.4 Remarks to Theorem The equations for S 40 and s 41 in Theorem correspond to matching the expected values of V x and log V x, E q V x ] E p V x ], E q log V x ] E p log V x ]. 50a 50b Similarly to 8, numerical root-nding can be used to calculate a solution to 41. Note that using a moment matching approach similar to Corollary 1 to nd a value for s is not advisable, since this would lead to further approximations because the true distribution pv x is unknown, and would possibly require a more complicated numerical solution. 5 Marginalising IWX V WV over V This section presents a result that is similar to the following property 1, Problem 5.33]: if p S Σ W d S ; n, Σ and pσ IW d Σ ; m, Ψ then the marginal density of S is ps GB II d X; n, m d 1, Ψ, 0 d. 51 Theorem 3. Let px V IW d X ; v, V/γ and let pv W d V ; s, S. The marginal for X is px GB II s d X;, v d 1, S γ, 0 d. 5 11

14 5.1 Proof of Theorem 3 We have px given as px px V pv dv IW d X ; v, V γ { v d 1d v d 1 Γ d { sd s Γd S s { v d 1 Γ d V v+s d etr W d V ; s, S dv 53a 53b } 1 v d 1/ X v V γ } 1 s d 1 V etr 0.5S 1 V dv 53c s Γ d X v s S X γ { v d 1 s Γ d Γ d X v s S sd v + s d 1 Γd W d V ; v + s d 1, X 1 γ { v d 1 s Γ d Γ d X v s S v+s d 1d v + s d 1 Γ d { v d 1 s Γ d Γ d Γ d v + s d 1 s+v d 1 Γ d Γ v d 1 s d Γd 1 β s d, v d 1 X s d 1 X + S γ β d s, v d 1 etr 0.5X 1 V γ } 1 v+s d 1d γ v d 1d + S 1 V dv } 1 v+s d 1d γ v d 1d v+s d 1 X S 1 γ X v S s 53d + S 1 dv 53e } 1 v+s d 1d γ v d 1d γx 1 + S 1 v+s d 1 } 1 γ v d 1d S v+s d 1 X 1 γ + X S 1 γ v d 1d γ v d 1d s+v d 1 S γ v d 1 X 1 S γ + X S 1 v+s d 1 S γ + X v+s d 1 X s d 1 S v d 1 X v S s 53f 53g 53h 53i 53j which, by 1, Theorem 5..], is the probability density function for GB II s d X;, v d 1, S γ, 0 d. 54 1

15 References 1] A. K. Gupta and D. K. Nagar, Matrix variate distributions, ser. Chapman & Hall/CRC monographs and surveys in pure and applied mathematics. Chapman & Hall, 000. ] J. Stoer and R. Bulirsch, Introduction to Numerical Analysis, nd ed. New York: Springer-Verlag, ] J. W. Koch, Bayesian approach to extended object and cluster tracking using random matrices, IEEE Transactions on Aerospace and Electronic Systems, vol. 44, no. 3, pp , Jul ] J. Lan and X. Rong Li, Tracking of extended object or target group using random matrix part I: New model and approach, in Proceedings of the International Conference on Information Fusion, Singapore, Jul. 01, pp ] C. M. Bishop, Pattern recognition and machine learning. New York, USA: Springer, ] T. Minka, A family of algorithms for approximate Bayesian inference, Ph.D. dissertation, Massachusetts Institute of Technology, Jan

17 Avdelning, Institution Division, Department Datum Date Division of Automatic Control Department of Electrical Engineering Språk Language Svenska/Swedish Engelska/English Rapporttyp Report category Licentiatavhandling Examensarbete C-uppsats D-uppsats Övrig rapport ISBN ISRN Serietitel och serienummer Title of series, numbering ISSN URL för elektronisk version LiTH-ISY-R-304 Titel Title Properties and approximations of some matrix variate probability density functions Författare Author Karl Granström, Umut Orguner Sammanfattning Abstract This report contains properties and approximations of some matrix valued probability density functions. Expected values of functions of generalised Beta type II distributed random variables are derived. In two Theorems, approximations of matrix variate distributions are derived. A third theorem contain a marginalisation result. Nyckelord Keywords Extended target, random matrix, Kullback-Leibler divergence, inverse Wishart, Wishart, generalized Beta.

On the Reduction of Gaussian inverse Wishart Mixtures

On the Reduction of Gaussian inverse Wishart Mixtures Karl Granström Division of Automatic Control Department of Electrical Engineering Linköping University, SE-58 8, Linköping, Sweden Email: karl@isy.liu.se