Statistics Research in Remote Sensing Data Analysis for Climate Science at the Jet Propulsion Laboratory

Size: px

Start display at page:

Download "Statistics Research in Remote Sensing Data Analysis for Climate Science at the Jet Propulsion Laboratory"

Cornelius Harmon
5 years ago
Views:

1 Statistics Research in Remote Sensing Data Analysis for Climate Science at the Jet Propulsion Laboratory Amy Braverman Jet Propulsion Laboratory, California Institute of Technology Mail Stop Oak Grove Drive Pasadena, CA October 18,

2 Overview Motivation Understanding the content of massive remote sensing data sets Data fusion for massive remote sensing data sets Service-oriented architectures and the analysis of massive, distributed data sets Using observations to evaluate climate models Summary 2

3 NASA s Earth Observing Fleet (EarthObservingFleetmpg) 3

4 4 Massive data set analysis

5 Massive dataset analysis NASA s Earth Observing System satellites return massive quantities of high-resolution, multivariate, spatio-temporal data Empirical probability distributions derived from these data are the signatures of physical processes generating them How to best summarize these data when you don t know how users will use them? Traditional approach: grid" the data and represent each grid cell by mean and standard deviation Can we do better? 5

6 Massive dataset analysis x 2 x 2 x 2 x 1 x 1 x 1 n x 1 x 2 1 x 11 x 12 2 x 21 x 22 N x N1 x N2 k ˆx 1 ˆx 2 N k δ k 1 q 11 q 12 N 1 δ 1 K q K1 q K2 N K δ K k ˆx 1 ˆx 2 N k δ k 1 q 1 q 2 N σ1 2 + σ2 2 6

Data fusion for remote sensing The true field is spatially continuous,

error and missing data True field Instrument 1 view Instrument 2 view

Statistical model: truth" = Y (s) (at point location s); observations =

7 Data fusion for remote sensing The true field is spatially continuous, but remote sensing instruments see discretized images with measurement error and missing data True field Instrument 1 view Instrument 2 view Can we infer the true field from the images? Statistical model: truth" = Y (s) (at point location s); observations = Zj at pixel Bj (s); and measurement error (Bj (s)) 1 Zj (Bj (s)) = Y (u)du + (Bj (s)) Bj (s) u Bj (s) 7

8 Data fusion for remote sensing 8

9 Statistics and SOA s NASA Distributed Active Archive Centers 9

10 Statistics and SOA s Local data storage (/temp/) User Program Files (subsets) Browser Search Get Subset DAAC server Files (subsets) DAAC archive Results User program must encode all functionality beyond gross-level access Requires knowledge of specific instrument characteristics including retrieval methods, format, measurement errors and biases, etc Difficulties multiply with more than one data source 10

Statistics and SOA s Local data storage (/temp/) Data server Data archive User Program Data structures Data structures (Results of computation) Results Push as much computation as possible to

11 Statistics and SOA s Local data storage (/temp/) Data server Data archive User Program Data structures Data structures (Results of computation) Results Push as much computation as possible to locations where data reside How to choreograph" data analysis to take advantage of this? How does the network topology defined by the system architecture constrain data analysis? How do data analysis objectives constrain the architecture? 11

12 Climate model evaluation Every five years the Intergovernmental Panel on Climate Change (IPCC) reviews the scientific literature on climate science and produces a report summarizing the state of knowledge A foundation of the IPCC report is the analysis of a set of climate model predictions based on different models running under different scenarios" (eg, double CO2) Should all the models count equally, or should some models be given more credence than others? Can we use present and past observations to decide which models are most reliable? 12

13 Climate model evaluation If the atmosphere behaves as the model specifies, then we would expect the observations to look like the model output to within the inherent variability of the model output Observations: Y 0 = Y 01,,Y 0N0 Sampling distribution of the median, location [35N,235E] Output of model j: Y j = Y j1,,y jnj Statistic: g( ): g(y 0 )=g 0, g(y j )=g j Estimate the sampling distribution of g j by resampling Likelihood of observing g 0 given model j sampling distribution is a figure of merit 13

14 Climate model evaluation Let A = j be the event that model j best represents the physical system Let g 0 = g(y 0 ) be a statistic computed from the time series of observations Let f (x A = j) be the sampling distribution (density) of that statistic given A = j f (g 0 A = j) is the likelihood of g 0 given A = j P(g 0 A = j) = g 0 +/2 f (x A = j)dx, g 0 /2 small P(A = j g 0 ) P(g 0 A = j)p(a = j) 14

15 Summary There are many opportunities for Statistics research and practice at JPL We need people to solve fundamental problems UCLA Statistics graduate students who have done research with us: Hai Nguyen (PhD, 2009): post-doc , now JPL staff Yuliya Marchette: data analysis and machine learning for hurricane reserach, Irina Kukuyeva: multivariate analysis for Earth and planetary science data, Linda Gharibans: analysis of distributed data and SOA s, Mark Nakamura: statistical downscaling of global climate model output Copyright 2011, California Institute of Technology Government sponsorship acknowledged 15

Mining Climate Data. Michael Steinbach Vipin Kumar University of Minnesota /AHPCRC

Mining Climate Data. Michael Steinbach Vipin Kumar University of Minnesota /AHPCRC Mining Climate Data Michael Steinbach Vipin Kumar University of Minnesota /AHPCRC Collaborators: G. Karypis, S. Shekhar (University of Minnesota/AHPCRC) V. Chadola, S. Iyer, G. Simon, P. Zhang (UM/AHPCRC)