for Sparsely Observed Functional Data University of Glasgow

Transcription

1 Spatial Correlation Estimation for Sparsely Observed Functional Data Workshop on High dimensional and dependent functional data, Bristol, Sep 2012 Surajit Ray University of Glasgow

2 Contributors t Chong Liu Mark Friedl Giles Hooker Doctoral Student(BU) Dept. of Earth Sciences and Geography(BU) Dept. of Statistics (Cornell University) 1

3 Outline Context t and Background Moment Based Method Likelihood klh Based Method Method Curve Reconstruction Results Future Work 2

4 ``Gap Filling for missing remote sensing observation 3

5 M lti year remote Multi t sensing i d data t 4

6 ``Gap Filling for missing remote sensing observation Longitudinal remote sensing data are collected for studying various climate ecosystem phenomenon Often a lot of observations are missing due to various reasons o o o o o Cloud Cover Aerosol Content Change of instrument Fire Snow Geoscientists would love to fill those gaps for applying standard statistical techniques. 5

7 Properties of remotely sensed ecosystem data Strong spatial correlation. Neighbouring Nihb i pixels might have correlation around Might have different correlation along different directions Strong seasonal components. Abrupt spatial changes gesdue to change geof landcover. Some landcover has more missing than other landcovers. Missingness is not at random. 6

8 Background and Context t Model for independent curves K Y() t () t () t i ik k i k 1 ik where eigen scores are independent across i. However, in real world, many applications have correlated Y i (t) across i. example: spatial temporal temporal data, online auction data, time course gene expression data. Most existing work treat them as iid i.i.d. Asymptotic property holds for mild correlation. 7

9 Previous Research Yao,Muller and Wang 2004 outlined the moment based methods to estimate covariance surface and eigenstructure of the random process assuming i.i.d. curves. o Also suggested reconstructing curve trajectories ( gap filling ) using expected principal component scores. Question: How to estimate eigenstructure and reconstructing trajectories assuming correlated curves? 8

10 Concurrent Research o Reduced Rank Mixed Effects Models for Spatially Correlated Hierarchical Functional Data JASA 2010, Zhou et al. Mentioned by Maurice Berk during his multilevel talk. o Principal components analysis for sparsely observed correlated functional data using a kernel smoothing approach, Electron. J. Statist. Volume 5 (2011). Paul and Peng o Some that I have missed 9

11 Our Model lf for correlate data 2. Our Model: where more general model: K Y() t () t () t i ik k i k 1 0, cov(, ) ip jq i j p, k q p q k 0, p q cov( ip, jq ) matern( d( i, j), k, k) k, p q k Goal: estimating or more generally and k k 10

12 Estimation Eti of Parameters Moment Based Method local linear smoothing of the covariance surface 0 and lag d cross covariance surface d where d = 1,2,...D, take ratios of eigenvalues as covariance estimates and fit matern or other parametric models. Likelihood based method Marginal likelihood is hard to optimize. Turn to joint likelihood of Y and and treat as random effects. Use EM(expectation maximization) to solve the optimal parameters ρ,α k andβ k 11

13 Moment tb based method Covariance Model 2 cov( Yi, Yj) cov( i, j) ' I 2 = diag(cov( i1, j1),cov( i2, j2),...,cov( ik, jk )) ' I Smooth covariance to get estimate of 0 cov( Yi, Yi) raw covariance G ( T, T ) ( Y ( T ))( Y ( T )) i is it is is it it Smooth G ( T, T ) ij is jt using local linear smoother and get 12

14 Moment tb based method Smooth lag d covariance to get estimate of where d(i, j) d raw covariance (, ) cov( Y, Y ) d i j i j G ( T, T ) ( Y ( T ))( Y ( T )) ij is jt is is jt jt Smooth G (, ) using local linear smoother and get ij Tis T jt d (i, j) 13

15 Moment tb based method 14

16 Moment tb based method How to use the lagged covariance surfaces Calculate Eigenvalue ratio eigenvalues of : 0 0,, k k 1,2... K eigenvalues of : dk,, k 1,2... K d For given k, we have cor d ( ik, jk ) d,k for d = 1,2,...D 0,k These dk, can be used to fit model parameters 0, k 15

17 Simulation Results for moment based method Simulation scheme 3 eigenfunctions, 100 curves, 10 time points. AR(1) type spatial correlation(parameter is ρ ) =0.2, 0.4, 0.6, 0.8. noise standard deviation σ = 0.05, 0.2, 0.5, 1. Results for estimation of ρ eigenfunction 1 to 3 with moment based method 16

18 Simulation Results: Ratio of first eigen values 17

19 Simulation Results: Ratio of 2nd eigen values 18

20 Simulation Results: Ratio of 3rd eigen values 19

21 Likelihood Mthd Method Spatial correlation introduced d through hrandom effects Random effects are zero mean and satisfy the following Then 0, cov(, ) ip jq i j p, k q p q k where 20

22 Likelihood Method 21

23 Likelihood Steps Model: Marginal Likelihood Direct maximizing of this likelihood over ρ,σ 2,Λ and Θ is a difficult non convex optimization problem. Solution: EM procedure poce ueby treating random effects ec sas missing data aaand focus on the joint log likelihood Joint log likelihood 22

24 EM Step E Step: M Step Note that Λ and ρ are separated from Θ and σ 2 optimization over these parameters one at a time and do it iteratively. 23

25 Simulation Results for moment based method Simulation scheme 3 eigenfunctions, 100 curves, 10 time points. AR(1) type spatial correlation(parameter is ρ) =0.2, 0.4, 0.6, 0.8. noise standard deviation σ = 0.05, 0.2, 0.5, 1. Results for estimation of ρ eigenfunction 1 to 3 with moment based method 24

26 EM estimates of correlation parameter 25

27 Reconstruction Results Given the estimate of spatial correlation structure, we can compute the expected principal component scores In the iidcurve i.i.d case, the information set only include curve i, ik ˆ ik E( ik Y 1,Y 2,...,Y N ) ˆ ik E( ik Y i ) For AR(1) correlated curves, curves were reconstructed using using and ˆ ik ˆ ik Performance measured by sum of squared errors over all curves. Negative log ratio suggest better performance 26

28 Histogram of log ratio of squared error 27

29 Future Work Consistency For moment based estimates Strict geometric constraints for solving the EM steps. Hypothesize btt better convergence results from this approach. Following Peng and Paul s, 2009 paper on A geometric approach to maximum likelihood estimation of the functional principal components from sparse longitudinal data Software: SPACE (Spatial Principal Analysis by Conditional Estimation) First version with limited spatial correlation choices. Follow up versions with more general spatial correlation. 28

30 Future Work Application to remote sensing data Apply SPCAE to gap fll fill remotely sensed data. Estimate optimal weights of how to impute missing observations from Neighbouring pixels Neighbouring year ( using PPC approach of Liu et al 2012) Biosphere 29

31 Ak Acknowledgement ld Collaborators: Giles Hooker, Cornell University Mark kfidlb Friedl, Boston University it Chong Liu, Boston University ( Doctoral Students) Grants: NSF Award No: # : Functional Data Modeling of Climate Ecosystem Dynamics 09/01/09 08/31/13 NSF Award No: # : GLACIER Global Gl l Change Initiative Education & Research. 03/15/ /14/2014 Related paper: Liu, C., Ray, S., Hooker, G., Friedl, M.F. (2012) Functional Factor Analysis For Periodic Remote Sensing Data. Annals of Applied Statistics, 6:2,