GLAM An Introduction to Array Methods in Statistics

Transcription

1 GLAM An Introduction to Array Methods in Statistics Iain Currie Heriot Watt University GLAM A Generalized Linear Array Model is a low-storage, high-speed, method for multidimensional smoothing, when data forms an array, Royal Statistical Society Conference University of Edinburgh model has a row and column structure which allows it to be written as a Kronecker product. September 09 Swedish male mortality data (HMD) Raw mortality surface 10 Deaths : D Exposures : E D,E :

2 Generalized linear models Log mortality for Swedish males age 70 Structure Data: vectors y of deaths and e of exposures Model: a model matrix B of B-splines a parameter vector θ a link function Error distribution: Poisson Algorithm Scoring algorithm µ = E(y), log µ = log e + Bθ Observed mortality B spline regression P spline regression B spline coefficients P spline coefficients B Wδ Bˆθ = B Wδ z where z = B θ + W δ 1 (y µ) is the working vector and Wδ is a diagonal matrix of weights. Bspline Penalties Eilers & Marx (1996) imposed penalties on differences between adjacent coefficients (θ 1 2θ 2 + θ 3 ) (θ c 2θ c 1 + θ c ) 2 = θ D 2D 2 θ 2-dimensional smoothing where D 2 is a second order difference matrix. Algorithm Penalized scoring algorithm (B Wδ B + P)ˆθ = B Wδ z, Let B a, n a c a, be a 1-d B-spline model matrix defined along age. Let B y, n y c y, be a 1-d B-spline model matrix defined along year. The 2-d model matrix is given by the Kronecker product B = B y B a, n a n y c a c y. P = λd 2D 2 is a roughness penalty. This is the method of P -splines.

3 2d B spline basis B spline Penalties in 2-d Each regression coefficient is associated with the summit of one of the hills. Smoothness is ensured by penalizing the coefficients in rows and columns P = λ a I cy D ad a + λ y D yd y I ca Amazing formula Generalized linear array models or GLAM [B y B a ]θ, n a n y 1 B a ΘB y, n a n y Structure GLAM log E[D] = log E + B a ΘB y Computational procedure with B = B y B a Bθ B a ΘB y B W δ B G(B a ) WG(B y ) conceptually attractive low footprint very fast generalizes to d-dimensions Definition: Row tensor of X, n c, G(X) = [X 1 c] [1 c X], n c 2.

4 Raw mortality surface Examples of GLAMs Mortality shocks: Swedish data and the Spanish flu Joint modelling of mortality surfaces: Insurance data by lives v amounts Density estimation: Old Faithful data Smooth + Shocks Modelling shocks Additive model: smooth surface + smooth period shocks [ [B y B a ]θ + I ny B ] a θ, B = [B y B a : I ny B ] a, Additive GLAM: B a ΘB y + B a Θ

5 Smooth Shocks Joint modelling of insurance data Insurance data by lives and amounts. Additive model: smooth 2d-surface + smooth age-dependent gaps Lives: [B y B a ]θ Amounts: [B y B a ]θ + [ ] 1 ny B a θ. Additive GLAM Lives: B a ΘB y Amounts: B a ΘB y + B a Θ1 ny. Log(mortality) Amounts = 70 Lives Log(mortality) Lives Amounts = Observed, smoothed and forecast log mortality by lives and amounts.

6 Normalized Density 2-d Density Estimation Form a fine 2-d grid of counts Apply 2-d P -spline smoothing with Poisson errors & log link Model matrix B 2 (x 2 ) B 1 (x 1 ) third order penalties Example: Old Faithful Geyser Data 272 data points 217 grid 238 counts of 1, 17 of 2, and (98%!) counts of 0. Duration (minutes): bin width = 1 sec Waiting time (minutes): bin width = 1 min Normalized Density Histogram of waiting times Density Waiting time Duration Density d marginal density 1 d density Waiting time (minutes): bin width 1 min

7 Histogram of duration times Density d marginal density 1 d density References P -splines: Eilers & Marx (1996) Statistical Science, 11, GLAM: Currie, Durban & Eilers (06) Journal of the Royal Statistical Society, Series B, 68, Eilers, Currie & Durban (06) Computational Statistics & Data Analysis, 50, Mortality shocks: Kirkby & Currie (09) Statistical Modelling, to appear. Mortality data: Human Mortality Database GLAM web page iain/research/glam.html Duration time (seconds): bin width 1 sec