Correcting biased observation model error in data assimilation Tyrus Berry Dept. of Mathematical Sciences, GMU PSU-UMD DA Workshop June 27, 217 Joint work with John Harlim, PSU
BIAS IN OBSERVATION MODELS Consider the standard filtering problem, x i = f (x i 1 ) + ω i 1 y i = h(x i ) + η i We assume the true observation function h(x) is unknown An approximate model is available h(x) so that y i = h(x i ) + η i = h(x i ) + b i + η i Where b i h(x i ) h(x i ) is called the bias
EXAMPLE 1: LORENZ-96 Consider the standard 4-dimensional Lorenz-96, ẋ j = x j 1 (x j+1 x j 2 ) x j + 8 We observe 2 of the 4 variables We draw ξ i U(, 1) and let the observations be, { xk ξ h(x k ) = i >.8 β k x k 8 else β k N (.5, 1/5). h is applied to 7 randomly chosen variables Remaining 13 are directly observed
EXAMPLE 1: LORENZ-96 The result is a bimodal distribution, cloudy/clear Obs Model Error = True Obs - h(true State)
CORRECTING THE BIAS Our goal is to find p(b i y i ) We can then adjust the filter by defining a new innovation Where ˆb i = E p(bi y i )[b i ] ˆɛ i = ɛ i + ˆb i = y i h(x f i ) + ˆb i We also inflate the obs covariance by R i = R o + ˆR bi Where ˆR bi = E p(bi y i )[(b i ˆb i )(b i ˆb i ) ]
CORRECTING THE BIAS If we can estimate p(b i y i ) we can fix the obs We will use Bayes to find p(b i y i ) = p(b i )p(y i b i ) We will use a simple prior p(b i ) = N (ɛ i, P y i ) ɛi = y i h(xi f ) is the innovation is the innovation covariance estimate P y i The real challenge is to estimate p(y i b i ) We will learn p(y i b i ) from training data using the kernel estimation of conditional distributions
CORRECTING THE BIAS Below plots have y i 4 Left is clear, right is cloudy Notice bimodal likelihood Probability.6.5.4.3.2 Prior, p(bl) Likelihood, p(yi bl) Posterior, p(bl yi) Bias Prior, ˆbi Bias Posterior, ˆµbi True Model Error, bi Probability.3.25.2.15.1 Prior, p(bl) Likelihood, p(yi bl) Posterior, p(bl yi) Bias Prior, ˆbi Bias Posterior, ˆµbi True Model Error, bi.1.5-14 -12-1 -8-6 -4-2 Obs Model Error -14-12 -1-8 -6-4 -2 Obs Model Error
LEARNING THE CONDITIONAL DISTRIBUTION Given training data (y i, b i ) our goal is to learn p(y i b i ) For a kernel K (α, β) = e α β 2 δ 2 we define Hilbert spaces { N } { N } H y = a i K (y i, ) : a R N, H b = a i K (b i, ) : a R N i=1 i=1 For example the kernel density estimate (KDE) ˆq is in H y ˆq(y) = 1 m N N K (y i, y) Eigenvectors φ l of K ij = K (y i, y j ) form an orthonormal basis for H y. Similarly ϕ k are a basis for H b. i=1
LEARNING THE CONDITIONAL DISTRIBUTION We assume that p(y b) can be approximated in H y H b Let C yb ij = φ i, ϕ j and C bb ij = ϕ i, ϕ j then define ( ) 1 C y b = C yb C bb + λi We can then define a consistent estimator of p(y b) by ˆp(y b) = N i,j=1 C y b i,j φ i (y)ϕ j (b)ˆq(y) We define eigenfunctions with Nystöm extension ϕ j (b) = λ 1 j N ϕ j (b i )K (b i, b) i=1
OVERVIEW Learning Phase: Given training data set (x i, y i ) Compute the biases b i = y i h(x i ) Learn the conditional distribution p(y b) Filtering: Forecast x f i Use prior p(b) = N (ɛ i, P y i ) innovation ɛ i = y i h(x f i ) Combine with conditional to find p(b y i ) = p(b)p(y i b) Estimate conditional mean ˆb i and covariance ˆR bi Adjust innovation ˆɛ i = ɛ i + ˆb i and R i = R o + ˆR bi Apply Kalman update, continue to the next filter step
OVERVIEW Primary Filter Prior Posterior p(x i ) p(x i y i ) Secondary Filter Error Prior Error Posterior p(b) p(b y i ) +Training Data Observation Likelihood y i p(y i b)
LORENZ-96 RESULTS Obs True State 1 2 2 4 15 1 5 EnKF 2 4-5 2 4 1 2 3 4 5 Time -1
LORENZ-96 RESULTS Works well with small measurement noise Observations need to be precise, but not accurate 5 4.5 4 EnKF, Unobstructed Obs EnKF, Obstructed Obs, Obstructed Obs 5 4.5 4 EnKF, Unobstructed Obs EnKF, Obstructed Obs, Obstructed Obs 3.5 3.5 3 3 RMSE 2.5 2 RMSE 2.5 2 1.5 1.5 1 1.5.2.3.4.5.6.7.8.9 1 Measurement Noise Standard Devation, R o.5.1.15.2.25.3.35.4.45.5 Observation Time
EXAMPLE 2: MULTI-CLOUD SATELLITE-LIKE OBS Consider a 7-dim l model for a column of atmosphere Baroclinic anomaly potential temperatures, θ 1 and θ 2 Boundary layer anomaly potential temperature, θ eb Vertically averaged water vapor content, q Cloud fractions: congestus f c, deep f d, and stratiform f s Extrapolate anomaly potential temperature at height z T (z) = θ 1 sin( zπ Z T ) + 2θ 2 sin( 2zπ Z T ), z [, 16] Khouider, B., J. Biello, and A. J. Majda, 21: A stochastic multicloud model for tropical convection.
EXAMPLE 2: MULTI-CLOUD SATELLITE-LIKE OBS Extrapolate anomaly potential temperature at height z T (z) = θ 1 sin( zπ Z T ) + 2θ 2 sin( 2zπ Z T ), z [, 16] Brightness temperature-like quantity at wavenumber-ν [ h ν (x, f ) = (1 f d f s ) (1 f c ) ( θ eb T ν () + + f c T (z c )T ν (z c ) + zd z c + (f d + f s )T (z d )T ν (z d ) + zc T (z) T ] ν z (z) dz Setting f = is the clear sky model z d T (z) T ν (z) dz, z T (z) T ν (z) dz) z (1)
EXAMPLE 2: MULTI-CLOUD SATELLITE-LIKE OBS Weighting functions define RTM at different wavenumbers 16 14 12 1 height (z) 8 6 4 2.5.1.15 weighting function ( Tν z )
EXAMPLE 2: MULTI-CLOUD SATELLITE-LIKE OBS Biases at the 16 observed wavenumbers
EXAMPLE 2: MULTI-CLOUD SATELLITE-LIKE OBS Multimodal likelihood functions Probability 12 1 8 6 4 2 Prior, p(bl) Likelihood, p(yi =.184 bl) Posterior, p(bl yi =.184) True Model Error, bi -.2 -.15 -.1 -.5 Obs Model Error Probability 1 9 8 7 6 5 4 3 2 1 Prior, p(bl) Likelihood, p(yi =.195 bl) Posterior, p(bl yi =.195) True Model Error, bi -.2 -.15 -.1 -.5 Obs Model Error
EXAMPLE 2: MULTI-CLOUD SATELLITE-LIKE OBS.3.25.2 Truth.7.6.5 Truth θ 1.15.1.5 -.5 -.1 2.5 2 1.5 1 7 71 72 73 74 75 76 77 78 Time Truth θ 2.4.3.2.1 -.1 1.45 1.4 1.35 1.3 1.25 7 71 72 73 74 75 76 77 78 Time Truth θ eb.5 -.5-1 q 1.2 1.15 1.1 1.5 1-1.5 7 71 72 73 74 75 76 77 78 Time.95 7 71 72 73 74 75 76 77 78 Time
EXAMPLE 2: MULTI-CLOUD SATELLITE-LIKE OBS.45.4.35.3.25 Truth fc.2.15.1.5 fd.35.3.25.2 7 71 72 73 74 75 76 77 78 Time.2 Truth.18.16 fs.14.12.1 Truth.15.8.1.5.6.4.2 7 71 72 73 74 75 76 77 78 Time 7 71 72 73 74 75 76 77 78 Time
EXAMPLE 2: MULTI-CLOUD SATELLITE-LIKE OBS θ 1, MSE (percent of variance) θ eb, MSE (percent of variance) 1 9 8 7 6 5 4 3 2 1 1-1 1 Measurement Noise (percent of obs variance) 1 9 8 7 6 5 4 3 2 1 1-1 1 Measurement Noise (percent of obs variance) θ 2, MSE (percent of variance) q, MSE (percent of variance) 1 9 8 7 6 5 4 3 2 1 1-1 1 Measurement Noise (percent of obs variance) 1 9 8 7 6 5 4 3 2 1 1-1 1 Measurement Noise (percent of obs variance)
EXAMPLE 2: MULTI-CLOUD SATELLITE-LIKE OBS fd, MSE (percent of variance) 1 9 8 7 6 5 4 3 2 1 fc, MSE (percent of variance) 1 9 8 7 6 5 4 3 2 1 1-1 1 Measurement Noise (percent of obs variance) 1-1 1 Measurement Noise (percent of obs variance) 1 9 8 fs, MSE (percent of variance) 7 6 5 4 3 2 1 1-1 1 Measurement Noise (percent of obs variance)