Tue 4/19/2016. MP experiment assignment (due Thursday) Final presentations: 28 April, 1-4 pm (final exam period)

Transcription

1 Data Assimilation Tue 4/9/06 Notes and optional practice assignment Reminders/announcements: MP eperiment assignment (due Thursday) Final presentations: 8 April, -4 pm (final eam period) Schedule optional meetings with me if feedack or assistance is needed Be sure to emphasize the analysis aspect! Handout/assignment provides additional guidance for content and evaluation Etra credit option: YouTue presentation of your project! CamStudio (free) Read short scenario for Thursday (will discuss in class)

2 Data Assimilation Ojectives: - Outline asic approach uilding from simple eample - Introduce central concepts and terminology - Clarify differences etween main methods: - 3DVAR, 4DVAR, Kalman filter, ensemle Kalman filter - Increase appreciation for our reliance upon DA in modeling and oservational work

3 Why study DA? Everyone using gridded analyses for any study or for model IC/LBC should know from where it came Quality of (re)analyses determined y quality of DA system used to create it (Reanalysis,, CFSR, MERRA, ERA, NARR, GFS, NAM, etc. ) Those who have DA skills will have no prolem finding a jo (see net slides)! Offerings in MEAS are too limited in this area

4 Recent DA jo opportunity

5 Jo perspective from colleague Neil Jacos (Panasonic Weather Services) GL: My sense is that there is a shortage of epertise in this area An etreme shortage. The prolems are ) even people with met ackgrounds are lacking some of the more intensive math knowledge, and ) you just can t find people who know Fortran. We are always looking Anyway, there are literally 6-7 people I know in the US who know this stuff well Kleist, Derer, Whitaker, Hamill, etc. They are all largely self taught. There are those who are great at the theory like Eugeina, ut you need someone with the complete package, and that includes oth math theory, as well as a software engineer s level of Fortran knowledge. Starting salaries for someone with the compete package just out of grad school (Ph.D.) would easily go 50k in NC and 300k+ on the west coast. Granted, there are proaly only 5 or so jo opportunities out there, ut 60% are unfilled.

6 Jo perspective from colleague Neil Jacos (Panasonic Weather Services) GL: Also, what advice would you offer to a group of MS and PhD atmospheric scientists who will e entering the jo market with regard to modeling, programming, and DA skills? Learn as much Fortran as possile from a software engineering side versus science side. Science ased Fortran does solve prolems, ut running operational code needs someone who can write very logical *efficient* code, and not just code that gets the right answer. Having eperience with HPC is a ig deal. Parallel processing will always e used over single threaded Fortran jos. Don t e afraid to look outside of just meteorology/da positions. Those type of math and programming skills are needed everywhere from landing proes on Mars to power trading in spot markets. My point is if you know the math and can write efficient code, you ll e assured a very well-paying jo. There is a general lack of qualified people in many different markets, so having a good foundation can set you up perfectly to go in many directions.

7 Jo perspective from colleague Peter Neilley (IBM/The Weather Company) Also, what advice would you offer I share your intuition that there is not a proper alance in qualified DA talent vs. NWP talent in our field to meet the needs of our science. Its hard to quantify that imalance, ut something that indirectly speaks to it is the percentage of computer time that majors center spend on DA vs the forward models. I elieve ECMWF is roughly on DA vs Model while I think NOAA is closer to 30-70, as is the UKMO. This numers speak to the importance of DA, which indirectly speaks to need for scientific talent to support the system. That said, there are two general classes of NWP jos, ut generally just type of DA jo. NWP jos are generally either model development or model use (e.g. for research/diagnostic/phenomena study purposes). The latter doesn't need to know or in some cases well understand the model. But on the DA side, there generally is just the development kind of jo. There isn't (as much of) an equivalent of the diagnostic NWP position for DA.

8 Jo perspective from colleague Peter Neilley (IBM/The Weather Company) Also, what advice would you offer Maye if I'm in Raleigh one day I could give the students a rown ag on my perspective of success in the enterprise, and in particular the private sector. But my fundamental recommendation to students these days is to develop a hyrid of talents. The quote I like to use is "find your second passion", the first passion eing meteorology of course. A second passion is what will make a student stand out in a sudiscipline and if they are strong there it will make them very employale. Eamples of second passions are software development, societal applications (e.g. financial markets, risk management, communication, renewales, etc.), teaching, etc. Perhaps NWP/DA development could e consider a second passion too. The point is eing just a meteorologist is generally not sufficient anymore.

9 ECMWF 500-m anomaly correlation Updated graph: Courtesy Dr. Adrian Simmons, ECMWF (July 00)

10 Data Assimilation.) Introduction.) Older Empirical Techniques a.) Successive correction methods (SCM).) Nudging 3.) Least Squares methods Optimum Interpolation (OI) Variational (cost function) approaches 3DVar 4DVar Kalman Filtering 4.) Dynamical alancing of initial conditions 5.) Oservational data QC

11 Data Assimilation (DA) Manual gridding of data used in initial NWP efforts Richardson (9) and Charney et al. (950) undertook time-consuming manual interpolation procedures Immediately recognized as not acceptale

12 Data Assimilation (DA) Talagrand (997) defined DA as Using all the availale information to determine as accurately as possile the state of the atmospheric (or oceanic) flow. Availale information includes much more than oservations: dynamical relations etween variales, error statistics, climatology, etc. Consider also the value of information at different times and places from analysis domain Info from one field is related to other fields (e.g., wind, geopotential height)

13 Data Assimilation There are not enough oservations at a given time to initialize a primitive equation model Data density is non-uniform Many oserved variales are not dependent variales in PE model (e.g., satellite radiance, radar reflectivity) Clear from start that first guess needed in model initialization (aka ackground or prior ) Initially, climatology, or comination of climo and shortterm forecast used. Now, short-term forecast used

14 Data Assimilation Even in Richardson s first NWP forecast, raw oservations alone could not e used to initialize a successful numerical forecast (as Lynch demonstrated) Must modify data in dynamically consistent manner to provide a valid initial condition Traditionally, Data Assimilation involves two facets: (i) Ojective analysis (OA) transfer of irregular oservations to a grid, with quality control (use st guess or ackground field) (ii) Data initialization ojective analysis contains noise that would result in large, spurious gravity waves, must filter, alance Modern DA essentially comines these steps

15 DA flow diagram (after Kalnay Fig. 5..a) Oservations (+/- 3 h) Background or first guess Gloal analysis (statistical interpolation and alancing) Operational forecasts Initial Conditions Gloal forecast model 6-h forecast Model is mechanism that propagates info in time & space E.g., model can transmit info from data rich to data poor areas

16 4DDA Over oceans, analysis consists largely of Asynoptic data (ship/uoy reports from all hours, satellite data, aircraft data) Earlier model short-term forecasts These sources are difficult to uild into ojective analyses Major operational centers have comined OA and initialization into a continuous cycle of data assimilation (e.g., 4DVAR at ECMWF)

17 3.) Least Squares Methods Results from Least Squares method carries over to more comple methods, so introduce first Start with simplest eample possile: Two independent temperature oservations, T & T ; assume instruments uniased (errors are random, not systematic) (whiteoard, starting with equation # thru #)

18 3.) Least Squares - variational For optimum interpolation method, minimized least-square error with respect to weights For variational methods: All information, weighted y their statistical error characteristics, used to derive cost function Cost function is minimized to yield analysis that is most likely estimate of the true atmospheric state at a given time

19 3. Least Squares: variational Cost function (from Kalnay 003): temperature os, T & T from different, independent data sources with normally distriuted errors & standard deviations and We can define the cost function J as J ( T ) T T T T The cost function is clearly related to the square of difference etween the actual temperature T and the oservations How can we minimize J(T)? ()

20 Variational approach Gaussian statistics for normal distriutions: Minimizing J(T) yields an epression for maimum likelihood of T in terms of oservations and their error statistics If we take T T T T T J ( T ) T 0 Let T a e the maimum likelihood value, and solve: Ta T T

21 Least squares: Variational Does this make sense, physically? a T T T Yes, an oservation with a small error variance (reliale) is weighted more heavily than one with a large error variance In this way, construct the est possile analysis using all availale data and knowledge of data error characteristics

22 3.) Least Squares Methods Operational systems use a ackground or first guess field, in addition to oservations Analogous development using oservation and ackground (first guess) value yields: T a T W Tos T (3)

23 3.) Least Squares Methods T a T W Tos T (3) The analysis is derived y adding innovation to st guess, weighted y optimal weight W a os The optimal weight is the ackground error variance divided y the total error variance os (4) (5) The analysis precision is the sum of the ackground and oservational precision

24 3.) Least Squares Methods W a (6) The analysis error variance is = to the ackground error variance weighted y ( optimal weight) These equations were developed here for an etremely simple eample, ut they have eactly the same form as in OI, 3DVar, Kalman Filtering

25 Analysis cycling We can cycle eq. (3) (6) if the ackground field is a model forecast Suppose we have analysis at time t = t i (e.g., 00 UTC), and we want susequent analysis for t = t i+ (e.g. 03 UTC) Two phases in cycle:.) Forecast phase to update T and.) Analysis phase to update T a and (using T os, ) a os

26 Etension to DA In forecast phase of analysis cycle, we compute a ackground field: (7) T ( t ) M T ( t ) i a i In (7), M is a Model, which does not have to e a dynamical model ut is, in practice Must also estimate for net analysis time

27 Analysis cycling methods We must also estimate for the future time: (a) Optimum Interpolation approach: ( t ) a ( t ) i a i (8) In (8), a is > ut <, a simple assumption aout error growth assumed in model M Then, compute new weight W using (4) () Kalman Filtering approach: Still use (7), ut instead of a simple assumption as (8), compute ackground error variance from model

28 Kalman Filter method Tt( ti ) M[ Tt( ti)] m (9) Future true Temperature Model forecast from perfect analysis Model error E.g., Use linearized version of model, tangent linear model, or TLM, to isolate error growth (several methods) ( t ) M i a, i m (0)

29 Analysis cycling.) Get oservations for new time, model forecast for ackground.) Determine at new time from (8 - OI) or (0 - KF) 3.) Compute analysis, determine analysis error variance for use in net cycle ( ) a

30 Comments In general, we do not and can not directly oserve the variales we are analyzing - Variales, times, or locations can differ etween os, analysis - E.g., radars measure reflectivity or radial velocity, satellites measure radiance; these are not dependent variales in model - These quantities are related to what we are analyzing, however Must use an oservation operator or oservation forward operator H ( T ) to project ackground information onto oservation space, in order to compute statistics H includes vertical and horizontal interpolations, and transformations ased on physical laws (e.g., radiation laws to convert ackground T or q to radiances)

31 Comments In 980s and early 990s, DA systems used retrieved analysis variales e.g., Satellite sounder data were processed to produce profiles of T, q, that looked like rawinsonde data This proved to e inferior relative to utilizing the raw measurements directly in oservational forward model Why? Retrieval technique doesn t have all the other data or ackground availale; the other data help to make the retrieval more accurate and maimize usefulness of data It is etremely difficult to know the error covariance of the retrieved profiles; e.g., the radiance error covariance is much etter known ecause it is due to instrument error (more likely to e uniased)

32 Etension to 4-D: Notation and Terminology Generalize to complete NWP model prolem: (i) Analysis for a field of model variales (ii) A ackground field availale at grid points (iii) A set of p oservations y at irregular points Can e D or 3D analysis Comine all model variales into large arrays of length n, where # of model variales n i j k H is oservation operator that transforms ackground field to oservation space; H T, transpose of H, converts ack to model space o a r i

33 Etension to 4-D: Notation and Terminology Also: (i) error variance ecomes error covariance matri (ii) optimal weight ecomes optimal gain matri y o Note that is used to denote the oservations: - The oservations are in different spatial (and temporal) locations from the model grid points - Oserved quantities are often not same physical parameters that are analyzed (as discussed efore)

34 Multivariate OI Prolem First, we tackled the OI prolem, which is similar in form to 3DVar and Kalman Filtering, ut with more approimations and other practical disadvantages For OI, (3) ecomes: In (), t a W ) a y os H ( a t ()

35 Multivariate OI Prolem Equivalently, we could write a W y os H ( ) We can also write () as: d W d a y where () os H ( ) The innovation or oservational increments vector B is the ackground error covariance matri R is the oservation error covariance matri

36 Multivariate OI Prolem a W ) y H ( Wd os (3) Analysis otained y adding ackground to the innovation, weighted y optimal weight matri. The st guess of the oservations is otained y applying H to ackground W BH T ( R HBH T ) (4) The optimal weight is the ackground error covariance in oservation space (BH T ) divided y total error variance (4) is otained y minimizing the least squares equation with respect to the weights to find optimum (not shown)

37 Multivariate OI Prolem P a I WH B (5) The analysis error variance is = to the ackground error covariance weighted y the identity matri I minus the optimal weight matri If you understood the earlier epressions for the simple prolem, then you can understand these, ecause the meanings are eactly analogous

38 OI and simple eample We can see that the following are analogous: W y H( ) a os (3) T a T W T os T (3) And the optimal weight equations are as well: W os BH T W HBH T R B = ackground error covariance, R = oservation error covariance

39 Multivariate OI Prolem P a I WH B (5) a W (6) The analysis error variance is = to the ackground error covariance weighted y the identity matri I minus the optimal weight matri

40 Comments on OI Method Oservation errors are due to several sources: (i) Instrument error (random) (ii) Error of representativeness (iii) Errors in transformation etween os, model space (H) R R R R inst rep H (6) In practice, high-density oservation clusters merged into superoservations that comine information from individual oservations efore assimilation Most critical aspect of prolem is determining B, ecause R is generally diagonal (or can e made so) if oservations are independent As efore, form of Kalman Filtering equations is the same as for OI, ecept for determination of ackground error covariance (forecasted from model)

41 3DVar 3DVar case: Parallel to earlier eample, we can arrive at (3) y minimizing a cost function with respect to analysis variales: (7) (8) ) ( ) ( ) ( H y R H y B J o T o T ) ( o T T a H y R H H R H B 0 ) ( a J

42 How does 3DVar Differ from OI? 3DVar and OI are similar in form, ut in practice 3DVar has several major advantages OI makes several approimations that are asent in variational methods: Method of solution is local (grid point y grid point) and sequential over variales The ackground error covariance is also locally approimated In 3DVar, the cost function can e minimized gloally (simultaneously everywhere) with all data used simultaneously In 3DVar, can easily add new constraints to cost function (e.g., a alance condition, or direct QC procedures)

43 3DVar From our notes: Zapotocny et al. (000), eq. (3): Difference is use of alance constraint in operational cost function, as discussed in article on page 609 ) ( ) ( ) ( H y R H y B J o T o T (7)

44 4DVar 4DVar is used at ECMWF, availale in MM5 (Zou et al. 997) and now WRF (9) Similar in form to 3DVar, ut etter incorporation of oservations from times that differ from t a Find analysis that minimizes difference etween model solution, oservations over some time interval Model assumed perfect in this process o i i i T o i N i i t t T t t t y H R y H B J ) ( ) ) ( ( ) (

45 4DVar 3DVar and 4DVar: (9) 4DVar cost function includes summation over time of each oservational increment computed with respect to the model integrated to the time of the oservation Cost function minimized wrt initial state of model, ut analysis at end of time interval is given y model integration so analysis must e a solution of model equations o i i i T o i N i i t t T t t t y H R y H B J ) ( ) ) ( ( ) ( ) ( ) ( ) ( H y R H y B J o T o T (7) Background integrated to same time as oservations using model

46 4DVar Used at ECMWF, availale in MM5, WRF (Zou et al. 997) Some centers have skipped straight from earlier techniques to 4DVar (e.g., ECMWF) Why has NCEP invested so heavily in 3DVar development? Insights from MM5/WRF Team: Stronger reliance on model itself in 4DVar, can e a disadvantage 4DVar is computationally very epensive, and many systems cannot utilize all oservations in time to get operational IC to model Not clear whether enefit from 4DVar greater than could e derived from etter use of high-density os in 3DVar (Kalnay 003)