HOW TO APPLY THE OPTIMAL ESTIMATION METHOD TO YOUR LIDAR MEASUREMENTS FOR IMPROVED RETRIEVALS OF TEMPERATURE AND COMPOSITION R. J. Sica 1,2,*, A. Haefee 2,1, A. Jaai 1, S. Gamage 1 and G. Farhani 1 1 Department of Physics and Astronomy, The University of Western Ontario, Canada, sica@uwo.ca 2 Federa Office of Meteoroogy and Cimatoogy, MeteoSwiss, Payerne, Switzerand ABSTRACT The optima estimation method (OEM) has a ong history of use in passive remote sensing, but has ony recenty been appied to active instruments ike idar. The OEM s advantage over traditiona techniques incudes obtaining a fu systematic and random uncertainty budget pus the abiity to work with the raw measurements without first appying instrument corrections. In our meeting presentation we wi show you how to use the OEM for temperature and composition retrievas for Rayeigh-scatter, Ramanscatter and DIAL idars. 1 INTRODUCTION 120 110 100 90 80 Retrieved Traditiona 70 60 50 40 30 150 200 250 300 350 Temperature (K) Figure 1: Two channe HSEQ temperature retrieva from Purpe Crow Lidar measurements on 24 May 2012 (red curve) compared to the traditiona method. Rayeigh-scatter idars are one of the best sources for temperature measurements in the midde atmosphere. Cooing in the midde atmosphere associated with warming in the ower atmosphere is an important measurement in assessing atmospheric change, as, athough it is sti compex, interpreting the midde atmospheric temperatures is simper than interpreting changes in surface temperature. However, the magnitude of the changes is sma, on the order of 1 per decade. Thus, it is critica to have avaiabe an anaysis technique that can perform a fu uncertainty budget on a profieby-profie basis. The OEM is a great choice for this appication, and we appied it successfuy to the Western Purpe Crow Lidar [1]. Figure 1 shows a 2-channe temperature retrieva from Purpe Crow Lidar measurements on 24 May 2012 (red curve). The figure aso shows temperatures cacuated using the traditiona method of Hauchecorne and Chanin for the ow-gain (green) and high-gain (bue) data channes [2]. The temperature profie used for the retrieva is the U.S. Standard Atmosphere (cyan curve). The horizonta dotted ine is the height above which the temperature profie begins to make a significant contribution to the retrieva. Figure 2 shows the uncertainty budget for the 2- channe temperature retrieva. The retrieva determines the statistica uncertainty (bue ine), in addition to the systematic uncertainties: tieon pressure (orange stars), ozone density (yeow +), ozone cross section (purpe *), density The Authors, pubished by EDP Sciences. This is an open access artice distributed under the terms of the Creative Commons Attribution License 4.0 (http://creativecommons.org/icenses/by/4.0/).
120 110 100 90 80 70 60 50 40 30 Stat p 0 [O 3 ] O 3 < Ray Exct ; Ray Exct < Ray < Ray (z) Gravity MMM Tota 10-10 -4 10-10 0 Uncertainty (K) Figure 2: Uncertainty budget for the temperature retrieva in Figure 1. Detais are given in the text. profie for Rayeigh extinction (green squares), Rayeigh extinction cross section (bue diamonds), variation of Rayeigh-scatter cross section with height (red trianges), gravity mode (bue trianges), variation of mean moecuar mass with height (orange trianges) and tota uncertainty (back ine). The horizonta dashed ine is the height above which the temperature profie makes a significant contribution to the retrieva. pied the OEM to the retrieva of water vapor using measurements from the MeteoSwiss RALMO idar [3]. The RALMO water vapor measurements, in terms of data quaity and caibration, are among the best avaiabe in the word. In addition, anciary instruments such as radiosondes and microwave radiometers are avaiabe in Payerne for vaidation of the RALMO measurements. Figure 3 shows the retrieved water vapor mixing ratio (red curve) using the OEM on 5 September 2009. The bue curve is the mixing ratio using the traditiona anaysis method. The green curve is the radiosonde measurement. The sonde is aunched at the start of the 30 min RALMO average. The dot-dashed ine is the mixing ratio profie used by the OEM. The horizonta dashed ine shows the height beow which the retrieva is due primariy to the measurement and not the. In this paper we wi discuss two technica aspects of appying OEM to idar measurements, cacuation of anaytic forms for the Jacobians and practica considerations for working with non-inear counting systems. The Optima Estimation Method 14 Retrieva & Mode Parameters, ) x, b measurement = Forward Mode + detector noise y = F(x, b)+ 12 10 8 6 4 Traditiona 2 OEM Sonde 0 10-10 -1 10 0 10 1 Water Vapor (g/kg) estimate & covariances a, S a, S y ) xa Sa, Sy Uncertainties: etrieved parameters, mode parameters, mode smoothing minimize: cost =[y F(ˆx, b)] T S 1 y [y F(ˆx, b)]+[ˆx x a] T S 1 a [ˆx x a] Retrieved Parameters & Uncertainties Rodgers, C. D. (2011), Inverse Methods for Atmospheric Sounding: Theory and Practice, Word Scientific. Figure 4: The OEM procedure. The retrieva is computed from an state a of the retrieva vector x, mode parameters, b and covariances S using measurements, y. Figure 3: Four channe temperature retrieva of water vapour from the MeteoSwiss RALMO idar on 5 September 2009 (red curve) compared to the traditiona method. After success with temperature retrieva, we ap- 2 FORWARD MODELS Figure 4 shows the basics of the Optima Estimation Method [3]. Our retrievas are firstprincipe retrievas; that is, our forward mode 2
(FM) incudes a the instrumenta and atmospheric parameters necessary to reproduce the raw measurements. Our forward modes are based on the idar equation. For Rayeigh-scatter temperature retrievas we use the foowing form of the idar equation for the true counts N t (z)=ct 2 ψ(z) p(z) z 2 kt(z) + B t (1) where in this compact form a the instrumenta parameters and constants are in the function ψ, except for the idar constant C which is expicity shown as it is often a retrieved quantity. The atmospheric transmission is T. The true background, B t, can be constant or the anaytica form of the background appropriate to a given system. The pressure, p(z) can be specified or computed from the temperature using the assumption of hydrostatic equiibrium [1]. Water vapour mixing ratio retrievas require a more compex set of equations, as many water vapour idars wi have 4 channes (2 digita and 2 anaog) for nitrogen (N) and water vapour (H). For the inear case, appicabe to anaog systems or ow-gain photomutipiers, the true counts are given by: N H = OT LT H C H n air z 2 e q + B H N N = OT LT N C N n N2 + B N q = nq. z 2 (2) Here the overap function, O, varies with atitude, n is number density and the transmission refects the ineastic scattering process. A og retrieva is used as water vapour mixing ratio, q, cannot be negative. 3 PRACTICAL JACOBIANS The derivative of the FM, F, with respect to the retrieva vector, x is caed the kerne, K. The kerne is a m k matrix whose eements are: K ij = F i(x) x j. (3) Since K is a matrix of derivatives it can aso be caed the Jacobian (which is the term we wi use, athough in atmospheric science K is sometimes caed the weighting function). The size of the Jacobian, that is, the number of retrieva parameters k, in reation to the number of independent measurements m, reates to the reguarization of the probem. When m < k the probem is i-posed (under-determined); when m > k the probem is over-constrained (overdetermined). Our idar inversions have been restricted to the over-constrained situation. To cacuate the Jacobians, consider a retrieva of temperature from Rayeigh-scatter measurements for a system with 2 detection channes, which coud be anaog, digita or a combination of both and at different height resoutions. The data vector has m detector sampes, y, defined at 1 heights for the first channe and 2 heights for the second channe, where 1 + 2 = m. We want to retrieve x(k) quantities consisting of, for exampe, temperature at some number of retrieva heights k, dead time(s) and background(s) depending on the number of channes. The m coumns of the Jacobian contain the sensitivity of a measurement to the retrieva vector. Parameters are retrieved on the retrieva grid, using the measurement vector specified on the data grid. To visuaize this consider a measurement vector comprised of a singe channe of photocounts from which you want to retrieve temperature. The data grid is then a series of photocount measurements as a function of time (height) as shown in Figure 5. For our specific case we retrieve a temperature at each height on the retrieva grid. Since a temperature has to be specified at a eves of the 3
yi xj Ni+3 yi+1 yi+9 Tj+1 Tj+2 xj+1... xj+5 Figure 5: Interpoation of a retrieved parameter (here temperature, T ) to photocounts, N, on the data grid. data grid to evauate the idar equation, the FM performs an interpoation of temperature from the retrieva grid to the data grid. We choose to use inear interpoation. Linear interpoation is a reasonabe choice for these retrievas as the both the data and retrieva grid spacing is much ess than an atmospheric scae height. Using the Chain Rue we write the temperature Jacobian as: K Tij = F i(t ) T i T i T j = F ( ) (4) i(t ) z j+1 z i T i z j+1 z j where the second term on the right hand side of Eq. (4) is due to the interpoation of the temperature on the retrieva grid to the data grid. 3.1 Anaytica Derivatives It is often not possibe to cacuate the anaytica derivative of a mode or retrieva parameter in the FM, in which case a numerica derivative can be cacuated. Typicay this derivative can be cacuated using a simpe finite difference scheme. However, if the anaytica derivative can be determined, its exact form is a better choice than the numerica derivative. Some of the anaytica derivatives are quite simpe, such as for the idar constant or a constant background. Others are not possibe, such as the temperature dependence of the Rayeigh-scatter FM under the assumption of hydrostatic equiibrium, where T additionay appears in the integra used to determine pressure. One Jacobian which can be determined anayticay is the derivative of the FM with respect to density. Consider the simper case where optica depth (or transmission) is not being retrieved, so transmission is specified on the data grid and an interpoation is not required. From Eq. (1) we see, using the Chain Rue, that the Jacobian of the FM with respect to density is: N t, = N t, T (5) n i T n i where and i range from 1 to m to form a m m matrix. For discrete measurements the optica depth, τ, is given by τ = σn i z (6) for equay spaced measurements and a constant cross section with atitude, σ, as in the case of water vapour and temperature retrievas. Note that τ can be a specific optica depth, e.g. aeroso optica depth, as required. The transmission is then T = exp = ( σn i z exp( σn i z), ) (7) where from the product above we see that the transmission Jacobian is: T = σt z for i n i = 0 for i >. (8) We can now cacuate the Jacobian with respect to density using Eqs. (5) and (8): N t, n i = 2(N t, B t, )σ z for i = 0 for i >. (9) 4
This form of the Jacobian is fast computationay and avoids any numerica issues with the exponentia quantities invoved. 3.2 Effect of Detector Saturation The FMs given in Eqs. (1) and (2) refer to the true count rate. Systems which measure in the daytime or have arge dynamic range may have signas which are significanty noninear due to imitations in the detection system. For instance, for daytime water vapour retrievas the background count rate is extremey arge and the observed background counts are not equa to the true (corrected) counts. Furthermore, in the specification of the parameters for the retrievas, the background term is estimated using the observed counts, not the true counts, so this difference must be accounted for in a quantities in the FM. Consider a non-parayzabe system where the observed counts are reated to the true counts by N o = N t 1 + γn t (10) where γ is the counting system dead time. The derivative of the observed counts with respect to the true counts is then N o N t =(1 γn o ) 2. (11) To appy our previous resut for the Jacobian of the FM with respect to density for the noninear case, we must find the derivative of the observed photocounts with respect to n, using the observed photocounts N o and the observed background, B o. Using the Chain Rue and Eq. (9) we can show that 4 CONCLUSIONS Whie each different type of scattering process requires a different FM, many of the retrieva and mode parameters are the same. We demonstrated two tricks common to a retrievas, one an efficient anaytica form for the transmission density Jacobian and the other incusion of detector noninearity in the Jacobians. Currenty, we are in the processes of deveoping genera retrieva toos for use by the community in appying the OEM, which incorporate these resuts as we as some other finer points of the retrieva mechanics we have earned in deveoping these techniques. ACKNOWLEDGEMENTS We thank C. Dennison for advice on the cacuation of the transmission Jacobian. This project has been supported by the Natura Sciences and Engineering Research Counci (NSERC Discovery Grant & CREATE), Canadian Space Agency (CSA) and MeteoSwiss. References [1] Sica, R. J., and A. Haefee, 2015: Retrieva of temperature from a mutipe-channe Rayeigh-scatter idar using an optima estimation method, App. Opt. 54 (8), 1872-1889. [2] Hauchecorne, A., and M. Chanin, 1980: Density and temperature profies obtained by idar between 35 and 70 km, Geophys. Res. Lett., 7(8), 565-568. [3] Sica, R. J., and A. Haefee, 2016: Retrieva of water vapor mixing ratio from a mutipe channe Raman-scatter idar using an optima estimation method, App. Opt. 55 (4), 763-777. N o, = N o, N t, =(1 γn o, ) 2 n i N t, n i [ ( ) ] No, 2 B o, (σ z). 1 γn o, (12) [4] Rodgers, C. D., 2011: Inverse Methods for Atmospheric Sounding: Theory and Practice, Word Scientific, Singapore. 5