Applications of Fisher Information

Transcription

1 Applications of Fisher Information MarvinH.J. Gruber School of Mathematical Sciences Rochester Institute of Technology 85 Lomb Memorial Drive Rochester,NY

2 Overview Fisher Information Information Geometry Finding Distances between distributions of ridge regression estimators. Einstein s Equations and General Relativity

3 Fisher Information Matrix Let!=(! 1,! 2,,! n ). Let p(x,! 1,! 2,,! n ) be a probability density function. Then the elements of the Fisher information matrix are $ # 2 log p(x,! g ij (!) = "E 1,! 2,,! n )'! & ) % #! i #! j (

4 Information Geometry Information Geometry is Riemannian Geometry for Manifolds that are families of statistical distribution functions. The elements of the Riemannian metric are the entries in the Fisher information Matrix.

5 Rao(1945) Distance The distance between two distributions is the distance along the geodesic that connects them. On a surface or a manifold a geodesic is the path of shortest length between two reasonably close points.

6 Riemannian Metrics and Geodesics Let p(x,!) be a class of probability densities,e.g.,normal, binomial. Let " be the set of all values of the parameter!. Assume! #R n. Given a quadratic differential metric ds 2 = n n $ $ g ij (!)d! i d! j i=1 j=1 Let! =!(t),t 1 % t % t 2 be a curve joining P=!(t 1 ),Q =!(t 2 ). Then t & n n ( *( d(p,q) = - 2 ' $ $ g ij (!) + t 1 )( i=1 j=1,( 1/ 2 dt The curve between P and Q with minimum distance is the geodesic curve

7 Geodesics-Solutions to the Euler Lagrange Equations The Euler Lagrange Equations are obtained using the Calculus of Variations. n! g ij " i+! # ijk " i j=1.. n i=1 n! j=1.. " j = 0,k = 1,,n with the boundary conditions "(t 1 ) = $,"(t 2 ) = %. The quantity # ijk = 1 ( &g jk + &g ki ' &g + ij * - 2 )* &" i &" j &" k,- is called the Christoffel symbol of the first kind.

8 Multivariate Normal Distribution The multivariate normal distribution with mean vector µ and dispersion! takes the form 1 f(x,µ,!)= Exp % # 1 (2") p/2! 1/ 2 2 (x # µ ) $ (!#1 p ' (x # µ) *,x +R & ) Consider a family of multivariate normal distributions.for distributions with different means and the same dispersion d 2 = (µ 1 # µ 2 ) $! #1 (µ 1 # µ 2 ) For distributions with the same mean and different dispersions! 1 and! 2 p - d 2 = 1 ln 2, i 2 i=1 where the, i are the relative eigenvalues,i.e., solutions to the equation! 1 #,! 2 = 0.

9 My Own Research The application of information geometry to statistical inference. The comparison of the distance between prior and posterior distributions in the context of Bayesian Statistics. Some work that I have done can be found in Gruber,M.H.J. (2008) Some Applications of the Rao Distance to Shrinkage Estimators.Communications in Statistics Theory and Methods.37;

10 Prior and Posterior Distributions Let! be a parameter in R m.let X represent a random sample of size n from a population.let "(!) be a prior distribution, f(x!) is the sampling distribution of X. Using Bayes Theorem the posterior distribution g(! x)= $ # "(!)f(x!) "(!)f(x!d! where # is the parameter space. This is the posterior distribution.

11 Example for Linear Model Consider a linear model Y=X!+" where E("!)=0,D("!)=# 2 I. Assume that the errors are normally distributed and that the! parameters have a multivariate normal prior distribution where E(!)=$,D(!)=F. The posterior distribution can take the form of a multivariate normal distribution with mean!!=( X% X+# 2 F) &1 [ X% Y + F &1 $# 2 ] and dispersion ( X% X+# 2 F) &1 # 2.

12 Sample Result 1 For two linear Bayes estimators with different prior means but the same variance the squared distance between the prior distributions is 2 d prior = (! 1 "! 2 )# F "1 (! 1 "! 2 ) The squared distance between the posterior distributions is 2 d posterior d posterior % d prior = (! 1 "! 2 )# F "1 ( X# X + $ 2 F "1 ) "1 F "1 (! 1 "! 2 ) Let c be a constant where c>1.if (c-1)$ 2 ( X# X) "1 2 % F then d posterior % 1 d 2 c prior

13 Same prior mean, different prior dispersions Sample Result 2 When F=(! 2 / k)i the posterior mean is the ridge regression estimator of Hoerl and Kennard (1970)! "=( X# X+kI) -1 X# Y Consider two different values of k, k 1 and k 2. The distance between the posterior distributions d $ M if and only if for constants c 1 = e 2M > 1 and c 2 = e %2 M < 1 (1 c 1 )& i + k 2 $ c 2 k 1 $ (1% c 2 )& i + k 2

14 Information Geometry and General Relativity The ideas in this section of the talk are Taken from Rodriguez, C.C. (1998) Are We Cruising on a Hypothesis Space.arXiv Physics/

15 Riemann,Ricci and Einstein Tensor Riemann Curvature Tensor l R ijk where =!!x " l j ik " m ij = 1 $ g km " 2 ijk p k=1 #!!x " l k ij p $ + (" js s=1 l s " ik l # " ks " s ij ) and g ij are the inverses of the elements of the Fisher Information Matrix. Ricci Curvature Tensor R ij = p $ l=1 l R ilj The trace of the Ricci curvature is called the scalar curvature. Einstein Curvature Tensor G ab = R ab # 1 2 gab R

16 Einstein Equation! a measure of local $ " # spacetime curvature% & =! a measure of $ " # matter energy density% & In a vacuum the Ricci tensor is zero. R ij = 0 In general the Einstein tensor is proportional to the matter energy density. G='T

17 Parallels between Relativity and Statistical Inference Relativity:Mass energy is the source of gravity and the strength of the gravity field is measured by the curvature of space time. Statistical Inference:Information is the source of the curvature of the hypothesis space(statistical manifolds).

18 Parallels between Relativity and Statistical Inference(continued) Relativity:The dynamics of how mass energy curves space-time are controlled by the field equation G=κT. Statistical inference:the field equation controls the dynamics of how prior information produces models.

19 Parallels between Relativity and Statistical Inference(continued) Relativity: The field equation for a vacuum is the Euler Lagrange equation that extemizes S = " Rd! d! = det g d 4 x where the integral is taken over the interior of a four dimensional region W, R is the scalar curvature and g is the metric. Statistical Inference: (1)The form of the hypothesis spaces based on no prior information must satisfy G=0. (2)Given a hypothesis space with prior information g(q) the Einstein Tensor G quantifies the amount of prior information locally contained in the model at each point q.

20 Problem Einstein's equation works in spaces of any dimension. What can Einstein's equation tell us about the prior and posterior distribution for situations involving multivariate data?

21 Summary Two applications of Fisher Information have been given. Finding distances between distributions of ridge type estimators. How Einstein Equations from general relativity can help us with problems of statistical inference.