Variational assimilation of meteorological images for motion estimation

Transcription

1 Variational assimilation of meteorological images for motion estimation T. Corpetti work with P. Heas, N. Papadakis and E. Mémin IRISA - COSTEL Brest, Journée Statistique Spatio-temporelle Feb., 14 th 2008

2 Goal Estimation of time-consistent horizontal motion fields at different layers of the atmosphere Observations : (sparse) sequences of meteorological images Use of physical prior knowledge

3 Goal Estimation of time-consistent horizontal motion fields at different layers of the atmosphere Observations : (sparse) sequences of meteorological images Use of physical prior knowledge

4 Methodological framework Main difficulty : State space of very large dimension An image of pixels yields unknown variables = Tools from stochastic approaches can not be applied. Variational assimilation : enables to deal with large systems thanks to the adjoint formulation

5 Methodological framework Main difficulty : State space of very large dimension An image of pixels yields unknown variables = Tools from stochastic approaches can not be applied. Variational assimilation : enables to deal with large systems thanks to the adjoint formulation

6 Outline Introduction - Overview Dense motion estimation in geophysics Problems with frame-to-frame estimators Introduction of dynamical constraints Variational assimilation Introduction System definition Adjoint formulation System to solve Application : motion estimation for atmospherical flows 1 Assimilation of motion for «bidimensional» atmospherical flows 2 Assimilation of motion and pressure maps for different layers of the atmosphere in an perfect model 3 Assimilation of motion for different layers of the atmosphere in an imperfect model

7 Outline Introduction - Overview

8 Introduction : motion estimation Motion estimation in physics basic assumption A given point keeps its brightness along its displacement de(x, t) dt = E(x, t) t = Optical Flow Constraint Equation (OFCE) + E(x, t) v(x, t) 0 This problem is undetermined (homogeneous areas, aperture problem) = Lucas-Kanade, correlation, optical-flow are some well-known methods to solve the OFCE by adding some constraints

9 Introduction : motion estimation Motion estimation in physics basic assumption A given point keeps its brightness along its displacement de(x, t) dt = E(x, t) t = Optical Flow Constraint Equation (OFCE) + E(x, t) v(x, t) 0 This problem is undetermined (homogeneous areas, aperture problem) = Lucas-Kanade, correlation, optical-flow are some well-known methods to solve the OFCE by adding some constraints

10 Introduction : motion estimation Lucas-Kanade and correlation approaches The unknown optic flow vector is constant within some neighborhood of size n = Convolution of the OFCE with a Gaussian kernel K n (to eliminate homogeneous areas) with Lucas-Kanade = Searching for a window that best fits the OFCE around a given point (mainly use in meteo for instance) Advantages : Global transportation well estimated - Fast Drawbacks : Incapacity of estimating local measurements, smooth motion fields

11 Introduction : motion estimation Lucas-Kanade and correlation approaches The unknown optic flow vector is constant within some neighborhood of size n = Convolution of the OFCE with a Gaussian kernel K n (to eliminate homogeneous areas) with Lucas-Kanade = Searching for a window that best fits the OFCE around a given point (mainly use in meteo for instance) Advantages : Global transportation well estimated - Fast Drawbacks : Incapacity of estimating local measurements, smooth motion fields

12 Introduction : motion estimation Lucas-Kanade and correlation approaches The unknown optic flow vector is constant within some neighborhood of size n = Convolution of the OFCE with a Gaussian kernel K n (to eliminate homogeneous areas) with Lucas-Kanade = Searching for a window that best fits the OFCE around a given point (mainly use in meteo for instance) Advantages : Global transportation well estimated - Fast Drawbacks : Incapacity of estimating local measurements, smooth motion fields

13 Introduction : motion estimation Lucas-Kanade and correlation approaches The unknown optic flow vector is constant within some neighborhood of size n = Convolution of the OFCE with a Gaussian kernel K n (to eliminate homogeneous areas) with Lucas-Kanade = Searching for a window that best fits the OFCE around a given point (mainly use in meteo for instance) Advantages : Global transportation well estimated - Fast Drawbacks : Incapacity of estimating local measurements, smooth motion fields

14 Introduction : motion estimation Optical-flow : local approaches Addition of a smoothing term Minimization of a global energy function ([Horn & Schunck 81]) : ( 2 E(x, t) + E(x, t) v(x, t)) + α ( u 2 + v 2) dx (1) t Ω Advantages : 1 Coherent and precise estimations = such approaches can be very competitive! Drawbacks : 1 Can be long 2 Not adapted for geophysical flows

15 Introduction : motion estimation Optical-flow : local approaches Addition of a smoothing term Minimization of a global energy function ([Horn & Schunck 81]) : ( 2 E(x, t) + E(x, t) v(x, t)) + α ( u 2 + v 2) dx (1) t Ω Advantages : 1 Coherent and precise estimations = such approaches can be very competitive! Drawbacks : 1 Can be long 2 Not adapted for geophysical flows

16 Introduction : motion estimation in geophysics It can be shown that both assumptions are likely to be invalid in a context of fluid displacements. 1 The first order smoothing term is equivalent to minimize ( ) α div 2 v + curl 2 v dx (2) Ω 2 For compressible fluids and depending on the visualization, the OFCE may be not valid = There exists some fluid-dedicated motion estimators [Corpetti02,Corpetti06,Cuzol07,Heas07,Runhau05,Runhau06] Ω ( 2 E(x, t) + E(x, t) v(x, t) + Ediv v(x, t)) + α ( div v 2 + curl v 2) dx t (3)

17 Problems of frame-to-frame estimators Illustration : vorticity spectrum when many data are corrupted DNS Optical flow Synthetic PIV sequence Instantaneous motion field Vorticity spectrum

18 Resume Adapted spatial smoothers (like div-curl) are able to improve the motion estimation but it is not a physically-sound regularization When a sequence is available, it is better to exploit the whole sequence and a dynamical prior knowledge to directly estimate the motion from image sequences = variational assimilation is a convenient framework to manage this problem

19 Resume Adapted spatial smoothers (like div-curl) are able to improve the motion estimation but it is not a physically-sound regularization When a sequence is available, it is better to exploit the whole sequence and a dynamical prior knowledge to directly estimate the motion from image sequences = variational assimilation is a convenient framework to manage this problem

20 Outline Introduction - Overview Dense motion estimation in geophysics Problems with frame-to-frame estimators Introduction of dynamical constraints Variational assimilation Introduction System definition Adjoint formulation System to solve Application : motion estimation for atmospherical flows 1 Assimilation of motion for «bidimensional» atmospherical flows 2 Assimilation of motion and pressure maps for different layers of the atmosphere in an perfect model 3 Assimilation of motion for different layers of the atmosphere in an imperfect model

21 Tracking : 3 main classes 1 Prediction Find the future state knowing past and current observations 2 Filtering Find the current state knowing past and current observations Bayesian filtering, Kalman filtering, particle filtering

22 Tracking : 3 main classes 1 Smoothing Find the all states knowing past, current and future observations spatio-temporal regularizers Our context (posterior analysis of a specific event)

23 Variational assimilation Goal : recover a state space X (x, t) driven with a +/- known dynamic Formulation of the problem : Dynamical law & Initial condition + observations X (x, t) + M(X (x, t)) = ɛm(x) t (4) X (x, t 0) = X 0(x) + ɛ n(x), (5) Y(x, t) = H(X (x, t)) + ɛ o (x, t) (6) M : dynamic model (linear or not) ; Y : observations obtained through the (non) linear operator H ; X 0 initial condition (t 0 ) and (ɛ m, ɛ n, ɛ o ) : noises. Remarks One can also define a perfect dynamical model with unknown control variables u : X t (x, t) + M(X (x, t), u) = 0. Observations can be obtained from multiple sensors (only images in this application)

24 Variational assimilation Goal : recover a state space X (x, t) driven with a +/- known dynamic Formulation of the problem : Dynamical law & Initial condition + observations X (x, t) + M(X (x, t)) = ɛm(x) t (4) X (x, t 0) = X 0(x) + ɛ n(x), (5) Y(x, t) = H(X (x, t)) + ɛ o (x, t) (6) M : dynamic model (linear or not) ; Y : observations obtained through the (non) linear operator H ; X 0 initial condition (t 0 ) and (ɛ m, ɛ n, ɛ o ) : noises. Remarks One can also define a perfect dynamical model with unknown control variables u : X t (x, t) + M(X (x, t), u) = 0. Observations can be obtained from multiple sensors (only images in this application)

25 Variational assimilation Cost function : J (X (ɛ n, ɛ m )) = 1 2 tf t ɛ n 2 B tf t 0 Y H(X (ɛ m, ɛ n )) 2 R 1dt ɛ m 2 Q 1dt. B, Q, R : information (covariances) matrices and : X R 1 = X T R 1 X

26 Differentiation A minimizer X must cancel the directional derivative dj (X + βθ(x, t)) δj X (θ) = lim = 0 β 0 dβ The expression J (X + βθ(x, t)) reads : { 1 tf δj X (θ) = lim (Y H(X + βθ)) T R 1 (Y H(X + βθ)) dt β 0 2 t (X (t 0) + βθ(t 0 ) X 0 ) T B 1 (X (t 0 ) + βθ(t 0 ) X 0 ) tf t 0 ( X t ) T ( X + M(X + βθ) Q 1 t ) } + M(X + βθ) dt In practice, the estimation of such gradient is unfeasible (this requires to compute perturbations along all the components) Adjoint formulation of the problem [Lions 71]

27 Differentiation A minimizer X must cancel the directional derivative dj (X + βθ(x, t)) δj X (θ) = lim = 0 β 0 dβ The expression J (X + βθ(x, t)) reads : { 1 tf δj X (θ) = lim (Y H(X + βθ)) T R 1 (Y H(X + βθ)) dt β 0 2 t (X (t 0) + βθ(t 0 ) X 0 ) T B 1 (X (t 0 ) + βθ(t 0 ) X 0 ) tf t 0 ( X t ) T ( X + M(X + βθ) Q 1 t ) } + M(X + βθ) dt In practice, the estimation of such gradient is unfeasible (this requires to compute perturbations along all the components) Adjoint formulation of the problem [Lions 71]

28 Adjoint formulation (1/2) : simplify the previous relation Introduction of an adjoint variable λ λ = Q 1 ( X t ) + M(X + βθ) Definition of the linear tangent operator M X : dm(x + βθ) lim = M β 0 dβ X (θ) (same for H X ) Definition of the adjoint operator M = ( X M) by : (same for H = ( X H) ) (x, y) (V, O), < Ax, y > O =< x, A y > V = One can factorize the previous expression and perform an integration by part

29 Adjoint formulation (1/2) : simplify the previous relation Introduction of an adjoint variable λ λ = Q 1 ( X t ) + M(X + βθ) Definition of the linear tangent operator M X : dm(x + βθ) lim = M β 0 dβ X (θ) (same for H X ) Definition of the adjoint operator M = ( X M) by : (same for H = ( X H) ) (x, y) (V, O), < Ax, y > O =< x, A y > V = One can factorize the previous expression and perform an integration by part

30 Adjoint formulation (1/2) : simplify the previous relation Introduction of an adjoint variable λ λ = Q 1 ( X t ) + M(X + βθ) Definition of the linear tangent operator M X : dm(x + βθ) lim = M β 0 dβ X (θ) (same for H X ) Definition of the adjoint operator M = ( X M) by : (same for H = ( X H) ) (x, y) (V, O), < Ax, y > O =< x, A y > V = One can factorize the previous expression and perform an integration by part

31 Adjoint formulation (2/2) The cost function can be rewritten as : Z J (X + βθ(x, t)) lim = θ T (t f )λ(t f )dx β 0 dβ Ω Z Z tf» θ T dλ «+ ( X M) λ ( X H) R 1 (Y H(X )) dtdx Ω t 0 dt Z h i + θ T (t 0) B 1 (X (t 0) X 0) λ(t 0) dx = 0 Ω

32 Adjoint formulation (2/2) The cost function can be rewritten as : Z J (X + βθ(x, t)) lim = θ T (t f )λ(t f )dx β 0 dβ Ω Z Z tf» θ T dλ «+ ( X M) λ ( X H) R 1 (Y H(X )) dtdx Ω t 0 dt Z h i + θ T (t 0) B 1 (X (t 0) X 0) λ(t 0) dx = 0 Ω 3 relations to cancel

33 Adjoint formulation (2/2) Vanishing this adjoint relation yields to : 1 Compute the adjoint variables λ(t) by the backward equation : 8 < λ(t f ) = 0 : λ t (t) + M λ(t) = H R 1 (Y H(X ))(t) 2 Update the initial condition X (t 0) = Bλ(t 0) + X (t 0) 3 Compute the state variable X with the forward equation : X t + M(X ) = Qλ If Q = 0 (perfect dynamical model) : direct solving If Q 0 (imperfect dynamical model) : incremental formulation around X = X + dx

34 Adjoint formulation (2/2) Vanishing this adjoint relation yields to : 1 Compute the adjoint variables λ(t) by the backward equation : 8 < λ(t f ) = 0 : λ t (t) + M λ(t) = H R 1 (Y H(X ))(t) 2 Update the initial condition X (t 0) = Bλ(t 0) + X (t 0) 3 Compute the state variable X with the forward equation : X t + M(X ) = Qλ If Q = 0 (perfect dynamical model) : direct solving If Q 0 (imperfect dynamical model) : incremental formulation around X = X + dx

35 Adjoint formulation (2/2) Vanishing this adjoint relation yields to : 1 Compute the adjoint variables λ(t) by the backward equation : 8 < λ(t f ) = 0 : λ t (t) + M λ(t) = H R 1 (Y H(X ))(t) 2 Update the initial condition X (t 0) = Bλ(t 0) + X (t 0) 3 Compute the state variable X with the forward equation : X t + M(X ) = Qλ If Q = 0 (perfect dynamical model) : direct solving If Q 0 (imperfect dynamical model) : incremental formulation around X = X + dx

36 System to solve : alternatively w.r.t. the adjoint variables λ and the system s state X Algorithm 1 Forward integration of the crude solution X t (t) + M( X ) = 0 2 Find Adjoint variables λ(t) with the backward equation λ(t f ) = 0 ; λ t (t) + M λ(t) = H R 1 (Y H( X ))(t) 3 Update the initial condition : dx (t 0) = Bλ(t 0) + dx (t 0) 4 Find the state space : compute X (t) from X (t 0) and λ with the forward integration «dx M (t) + t X dx (t) = Qλ(t) (definition of the adjoint variable) 5 Update : X = X + dx 6 Loop to step 2 until convergence

37 Resume For the assimilation, one needs to define 1 The system state X 2 The dynamical operator M (and its adjoint) 3 The observation operator H (and its adjoint) 4 The covariances matrixes

38 Outline Introduction - Overview Dense motion estimation in geophysics Problems with frame-to-frame estimators Introduction of dynamical constraints Variational assimilation Introduction System definition Adjoint formulation System to solve Application : motion estimation for atmospherical flows 1 Assimilation of motion for «bidimensional» atmospherical flows 2 Assimilation of motion and pressure maps for different layers of the atmosphere in an perfect model 3 Assimilation of motion for different layers of the atmosphere in an imperfect model

39 Applications 1 Assimilation of motion for «bidimensional» atmospherical flows Imperfect dynamical model issued from Navier-Stokes Observation : image intensities

40 1 - System state Natural representation of a motion field : v = (u, v) T In this application : use of the Helhmoltz decomposition Decomposition of any motion field v(x) = (u(x), v(x)) T as the sum of a curl-free (or irrotational) component, a div-free (or solenoidal) component and a div-curl-free (or laminar) component : v = v irr + v sol + v lam = Total field = Irr part + Sol Part curl(v(x)) = ξ(x) = div(v(x)) = ζ(x) = v/ x u/ y = 0 u/ x + v/ y = 0

41 1 - System state Natural representation of a motion field : v = (u, v) T In this application : use of the Helhmoltz decomposition Decomposition of any motion field v(x) = (u(x), v(x)) T as the sum of a curl-free (or irrotational) component, a div-free (or solenoidal) component and a div-curl-free (or laminar) component : v = v irr + v sol + v lam = Total field = Irr part + Sol Part curl(v(x)) = ξ(x) = div(v(x)) = ζ(x) = v/ x u/ y = 0 u/ x + v/ y = 0

42 1 - System state Motion field representation The laminar global transportation component can be easily estimated (on the basis of an optical flow estimator associated to a high smoothing penalty) The motion field v is represented by the vorticity ξ and the divergence ζ with : v = v irr + v sol = G ζ + G ξ = H G (X ). G : 2D Green Kernel associated to the Laplacian operator (G = 1 2π ln( x )). = System state : X = (ξ, ζ) T.

43 1 - System state Motion field representation The laminar global transportation component can be easily estimated (on the basis of an optical flow estimator associated to a high smoothing penalty) The motion field v is represented by the vorticity ξ and the divergence ζ with : v = v irr + v sol = G ζ + G ξ = H G (X ). G : 2D Green Kernel associated to the Laplacian operator (G = 1 2π ln( x )). = System state : X = (ξ, ζ) T.

44 2 - Dynamical law Starting from the Shallow-Water vorticity-divergence equation Formulation of the 3D incompressible Navier-Stokes equation for a layer h k of the atmosphere ξ t + v ξ + ξζ = ν ξ, ζ t + v ζ + ζ 2 = 2det(J(u, v)) hk h k t + v h k + h k ζ = 0 ξ : vorticity ; ζ : divergence ρ 0 + ν ζ,

45 2 - Dynamical law Starting from the Shallow-Water vorticity-divergence equation Formulation of the 3D incompressible Navier-Stokes equation for a layer h k of the atmosphere ξ t + v ξ + ξζ = ν ξ, ζ t + v ζ + ζ 2 = 2det(J(u, v)) hk h k t + v h k + h k ζ = 0 ξ : vorticity ; ζ : divergence ρ 0 + ν ζ, Assumption : observed phenomena at only one layer of the atmosphere

46 2 - Dynamical law As ξ and ζ completely determine the underlying velocity field, one get the following imperfect dynamical model for the fluid motion field : [ ] v ξ + ξζ ν ξ M(ξ, ζ) = v ζ + ζ 2. 2det(J(u, v)) ν ζ The tangent linear operator is analytically obtained from this formulation The discrete version of the adjoint operator is the transpose of the tangent linear operator [Talagrand02]

47 2 - Dynamical law As ξ and ζ completely determine the underlying velocity field, one get the following imperfect dynamical model for the fluid motion field : [ ] v ξ + ξζ ν ξ M(ξ, ζ) = v ζ + ζ 2. 2det(J(u, v)) ν ζ The tangent linear operator is analytically obtained from this formulation The discrete version of the adjoint operator is the transpose of the tangent linear operator [Talagrand02]

48 3 - Observations : optical-flow constraint equation (OFCE) Optical Flow Constraint Equation ( ) E(x, t) K n + E(x, t) H G X (x, t) 0 (7) t }{{} de dt = the observation Y = H(X ) is : { Y = K n E(x,t) t H(X ) = (K n E) T H G X. (8) The adjoint reads X H = H G (K n E). (9)

49 4 - Covariances matrixes The matrix R and B associated to the observations and initial condition are defined as R = B = C min + (C max C min )(e E(x,t) /σ2 ) where C and σ are parameters to fix. If no observation is available, these matrixes are set to infinity. Matrix associated to the dynamical model Q : constant diagonal matrix Quantitative values : Q = 0.001, σ = 0.9 and C [50; 500]

50 Experiments on a real sequence Synthetic sequence

51 Real sequence : vince (1/4) Real cyclone (9 th october 2005) Infrared data Sequence of 116 images

52 Real sequence : vince (2/4) Real cyclone (9 th october 2005) Visible data Sequence of 116 images

53 Real sequence : vince (3/4) Real cyclone (9 th october 2005) Velocities obtained with the visible channel superimposed to IR data Sequence of 116 images

54 Real sequence : vince (4/4) Comparison with fluid dedicated optical-flow estimator (without temporal consistency) Assimilated vorticity No temporal consistency

55 Applications 1 Assimilation of motion for «bidimensional» atmospherical flows Imperfect dynamical model issued from Navier-Stokes Observation : image intensities 2 Assimilation of motion and pressure maps for different layers of the atmosphere Perfect dynamical model issued from Navier-Stokes Observation : pressure (and not the motion) 3 Assimilation of motion for different layers of the atmosphere Imperfect dynamical model issued from Navier-Stokes Observation : pressure

56 Goal Recover the whole motion fields and images of pressure differences Observations : k sparse pressure differences images h 1 obs h 2 obs h 3 obs Dynamical models available : based on the shallow-water approximation

57 Data : related to layers of the atmosphere Sparse pressure images are obtained using [Heas07] and EUMETSAT products [Schmetz93] ( Details here ). Assumption : N independent layers = valid if the limit of horizontal scales are much greater than vertical scales height. For layers of 1 km thickness, this hypothesis is valid for horizontal scales greater or equal to 100km = only large-scales will be estimated

58 Data : related to layers of the atmosphere Sparse pressure images are obtained using [Heas07] and EUMETSAT products [Schmetz93] ( Details here ). Assumption : N independent layers = valid if the limit of horizontal scales are much greater than vertical scales height. For layers of 1 km thickness, this hypothesis is valid for horizontal scales greater or equal to 100km = only large-scales will be estimated

59 Dynamical model Goal : define a dynamic adapted to images of pressure-differences based on the shallow-water approximation With satellite data like MSG, a pixel represents 3 3 km 2 = to obtain a valid dynamical model at a pixel grid, we filter the dynamical equations by a Gaussian Kernel K δx with δ x = 100δp 1

60 Shallow-water approximation Using the shallow-water approximation, the filtered horizontal momentum equations for atmospheric motion read : { dũ dt + px ρ 0 ṽf φ = ν T ũ dṽ dt + py ρ 0 + ũf φ = ν T ṽ (10) where p = K δx p (11) ṽ = (ũ, ṽ) T = K δx (u, v) T (12) with ρ 0 : local mean density f φ : Coriolis factor depending on latitude φ ν T turbulent viscosity produced at sub-grid scales [Frisch95]. In practice : Smagorinsky sub-grid model ν T = (Cδ x ) 2(ũ 2 2 x + ṽ 2 y + (ũ y + ṽ x ) 2 )

61 Shallow-water approximation After several manipulations : Expanding total derivatives in isobaric coordinates Using the fact that we have incompressible flows Vertical integration of the Shallow-water equations and continuity equation one obtain independent shallow-water equation systems for atmospheric layers k [1, K] : 8 < : h k t +div( q k ) = 0 ( q k ) t +div( 1 h k qk q k )+ 1 2ρ k xy( h k ) 2 +» 0 1 f φ q k =νt k ( q k ), 1 0 (13) with h k = p(s k ) p(s k+1 ) ; ṽ k = (ũ k, ṽ k ) = 1 h k R p(s k ) p(s k+1 ) ṽdp ; qk = h k ṽ k and div( 1 h k qk q k ) = " ( h k (ũ k ) 2 ) + ( h k ũ k ṽ k ) x y ( h k ũ k ṽ k ) + ( h k (ṽ k ) 2 ) x y #,

62 Shallow-water approximation After several manipulations : Expanding total derivatives in isobaric coordinates Using the fact that we have incompressible flows Vertical integration of the Shallow-water equations and continuity equation one obtain independent shallow-water equation systems for atmospheric layers k [1, K] : 8 < : h k t +div( q k ) = 0 ( q k ) t +div( 1 h k qk q k )+ 1 2ρ k xy( h k ) 2 +» 0 1 f φ q k =νt k ( q k ), 1 0 (13) with h k = p(s k ) p(s k+1 ) ; ṽ k = (ũ k, ṽ k ) = 1 h k R p(s k ) p(s k+1 ) ṽdp ; qk = h k ṽ k and div( 1 h k qk q k ) = " ( h k (ũ k ) 2 ) + ( h k ũ k ṽ k ) x y ( h k ũ k ṽ k ) + ( h k (ṽ k ) 2 ) x y #,

63 System State variables : X k = [ h k, q k ] T with a control of initial conditions j h k (t 0) = h k 0 + η k h q k (t 0) = q k 0 + η k q (14) Dynamical model : previous relations Observations { Y = Kδx h k obs H = [I d, 0]. (15) = the motion correction is achieved relying uniquely on pressure difference observations. Cost function : J k (η k h, η k q) = 1 2 B k h and Bk q : cov. matrices tf t 0 K δx h k obs h k (η k h, η k q) 2 R kdt ηk h 2 B k h η q k 2 B k q, (16)

64 System State variables : X k = [ h k, q k ] T with a control of initial conditions j h k (t 0) = h k 0 + η k h q k (t 0) = q k 0 + η k q (14) Dynamical model : previous relations Observations { Y = Kδx h k obs H = [I d, 0]. (15) = the motion correction is achieved relying uniquely on pressure difference observations. Cost function : J k (η k h, η k q) = 1 2 B k h and Bk q : cov. matrices tf t 0 K δx h k obs h k (η k h, η k q) 2 R kdt ηk h 2 B k h η q k 2 B k q, (16)

65 System State variables : X k = [ h k, q k ] T with a control of initial conditions j h k (t 0) = h k 0 + η k h q k (t 0) = q k 0 + η k q (14) Dynamical model : previous relations Observations { Y = Kδx h k obs H = [I d, 0]. (15) = the motion correction is achieved relying uniquely on pressure difference observations. Cost function : J k (η k h, η k q) = 1 2 B k h and Bk q : cov. matrices tf t 0 K δx h k obs h k (η k h, η k q) 2 R kdt ηk h 2 B k h η q k 2 B k q, (16)

66 System details Algorithm Adjoint variables λ k = [λ k h, λk q] T 8 tλ k h + ṽ k (ṽ k )λ k q h k div(λ k ϱ q)=(r k ) 1 (K >< k δx h k obs h k ),» 0 1 tλ k q (ṽ k )λ k q ( λ k q)ṽ k λ k h+ f φ λ k q =νt k (λ k 1 0 q), >: λ k h(t f ) = 0, λ k q(t f ) = 0. Covariance matrix (R k ) 1 is directly defined using the mask of observation C k : j (R k ) 1 αobs if s C k (s, s) = 0 else, (17) (18) Matrices (B k h ) 1 and (B k q) 1 are defined as : j (B k h ) 1 (s, s) = α h (B k q) 1 (s, s) = α q (19)

67 Synthetic Experiments on a real sequence Large scales Fine scales

68 Real MSG sequence Sequence of 10 images of top-of-clouds pressure Cloud classification into K = 3 broad layers 10 images of 1024 x 1024 pixels covering an area over the north Atlantic Ocean (of about 3000 km 2 )

69 Perfect model (assimilation of pressure and motions) #1 #2 #3 t=0 min t= 1h15min t= 2h30min

70 Perfect model : comparison with optical flow (div and curl) for t = 0min optic-flow reference estimates 1 ṽ 1 ζ 1 ζ assimilation estimates

71 Perfect model : comparison with optical flow (div and curl) for t = 1h15min optic-flow reference estimates 1 ṽ 1 ζ 1 ζ assimilation estimates

72 Resume Input : sparse sequences of pressure images Output : dense sequences of pressure maps and motion fields for large scales h 1 obs h 2 obs h 3 obs The perfect model enables to estimate the motion field without any observations on it.

73 Resume Input : sparse sequences of pressure images Output : dense sequences of pressure maps and motion fields for large scales h 1 obs h 2 obs h 3 obs The perfect model enables to estimate the motion field without any observations on it.

74 Experiments on a real sequence Large scales Fine scales

75 Assimilation of pressure and motions at finer scales Assumption : horizontal motion fields greater than vertical ones = When we look at finer scales, the term 1 ( h k ) 2 can locally dominate 2ρ k in areas where this assumption breaks and this creates instabilities. The filtering with the Gaussian Kernel K δx fundamental (with δ x = 100δp 1 ) is Extraction of large scales only, fine scales impossible

76 Applications 1 Assimilation of motion for «bidimensional» atmospherical flows Imperfect dynamical model issued from Navier-Stokes Observation : image intensities 2 Assimilation of motion and pressure maps for different layers of the atmosphere Perfect dynamical model issued from Navier-Stokes Observation : pressure (and not the motion) 3 Assimilation of motion for different layers of the atmosphere Imperfect dynamical model issued from Navier-Stokes Observation : pressure

77 Simplified dynamical law Idea : relax the layering assumption and use an imperfect modelling to extract information on fine scales = we consider that filtered horizontal winds (vertically averaged within layer C k ) ṽ k are equal to filtered horizontal winds on layer upper surfaces s k+1. { ξk t + ṽ k ξ k + ( ξ k + f φ ) ζ k = ν T ( ξ k ) ζ k t +ṽ k ζ k +( ζ k ) 2 2 J ρ 1 0 p(sk+1 )+f φ ξ k =ν T ( ζ k )

78 Simplified dynamical law Idea : relax the layering assumption and use an imperfect modelling to extract information on fine scales = we consider that filtered horizontal winds (vertically averaged within layer C k ) ṽ k are equal to filtered horizontal winds on layer upper surfaces s k+1. { ξk t + ṽ k ξ k + ( ξ k + f φ ) ζ k = ν T ( ξ k ) ζ k t +ṽ k ζ k +( ζ k ) 2 2 J ρ 1 0 p(sk+1 )+f φ ξ k =ν T ( ζ k ) Simplification of the divergence model ( Detail )

79 Simplified dynamical law Idea : relax the layering assumption and use an imperfect modelling to extract information on fine scales = we consider that filtered horizontal winds (vertically averaged within layer C k ) ṽ k are equal to filtered horizontal winds on layer upper surfaces s k+1. New model { ξk t + ṽ k ξ k + ( ξ k + f φ ) ζ k = ν T ξ k ζ k t +ṽ k ζ k + ( ζ k ) 2 = ν T ζ k. (20)

80 Simplified model Observations

81 Image conservation Instead of the usual OFCE, we prefer to rely on the continuity equation filtered with a Gaussian Kernel (to deal with the aperture problem) K δx ( h k obs t + h k obs ṽ k + h k obsdivṽ k ) 0, (21) = the observation Y = H(ṽ k ) is : { Y = K δx hk obs (x,t) t H(h k obs ) = ( K δx hobs) k T HG ( ). (22) K δx h k obs T H G The adjoint reads X H = H G ( Kδx h k obs) ( Kδx h k obs) HG. (23)

82 System details Algorithm Adjoint variables [λ k ξ, λ k ζ] T are obtained with λ k ξ(t f ) = 0, λ k ζ(t f ) = 0, [ ] [ λ k ξ ṽ + t λ k + ζ ][ ] ν T ξ +f φ λ k ξ ζ 0 ṽ + ζ ν T λ k ζ }{{} ( X M) = X kh R 1 (Y H(X)). Cov. matrix (R k ) 1 is defined using the mask of observation C k and the luminance function : ( (R k ) 1 α (s, s) = min + (α max α min)(e hk obs (s,t) /σ2 ) if s C k (24) 0 else, Matrices (Q k ξ) 1, (Q k ζ) 1, (B k ξ) 1 and (B k ζ) 1 are defined as constant diagonal matrixes.

83 Synthetic Experiments on a real sequence Large scales Fine scales

84 Imperfect model (assimilation of div and curl) at large scales (layer 1) Horizontal wind fields, curl and div estimated by assimilation with an imperfect model.

85 Imperfect model (assimilation of div and curl) at large scales (layer 3) Horizontal wind fields, curl and div estimated by assimilation with an imperfect model.

86 Experiments on a real sequence Large scales Fine scales

87 Imperfect model (assim. div & curl) at finer scales assimilation estimates optical flow

88 Comments on fine scale estimation Consistency with results obtained by the previous method and optical flow. Smaller structures can be characterized Are they in accordance with the physics and the ground truth? Much more stable than the perfect modeling approach

89 Conclusion

90 Conclusion The framework of variational assimilation plays as a physically consistent spatio-temporal smoother We proposed 3 applications 1 Assimilation of ξ and ζ for 2D flows from MSG images in an imperfect model 2 Assimilation of observed pressure images and (u, v) T (not observed) in a perfect dynamical model for large scales 3 Assimilation of ξ and ζ in a imperfect dynamical model for large and small scales from pressure images This framework is flexible to deal with perfect/imperfect models, sparse/poor observations and large state spaces.

91 For the future Combination with others sources of measurements Physics : use of the temperature for the motion estimation Border conditions : how to manage? (control) Use of more sophisticated minimization methods Validation

92 Synthetic sequence of particles (1/2) Synthetic sequence to assess the benefits of the technique 2D divergence free turbulence obtained through a direct numerical simulation of the Navier-Stokes equation [Carlier 05]. Sequence of 52 images Im. sequence True Vorticity Estim. vorticity

93 Synthetic sequence of particles (2/2) Quantitative comparisons between the real field, this approach and a fluid motion dedicated optical-flow scheme (without temporal consistency) [Corpetti02] Vorticity Root Mean Square Error Optical flow estimation Velocities assimilation DNS Assimilation (images) Optical flow Images Mean Square Error Vorticity spectrum = Very good results = Assimilated vorticity recover high frequencies. Back

94 Observations : layering of the atmosphere Decomposition using [Heas07] : the k-th layer corresponds to the volume lying in between an upper surface s k+1 and a lower surface s k Surfaces s k+1 : height of top of clouds belonging to the k-th layer = they are defined only in areas where there exists clouds. Top of clouds classifications maps : based on thresholds of top of cloud pressure and are routinely provided by the EUMETSAT consortium (based on top of clouds pressure images [Schmetz93]). Cloud pressure Classification

95 Observations : sparse images of pressure differences For a layer k, the pressure difference p(x, y, s k ) p(x, y, s k+1 ) is the quantity of interest In satellite images, clouds of height s k are always occluded = we approximate the underneath pressure p(s k ) by p(s k ). The observation at layer k is ( Back ). j h k p(x, y, s k ) p(x, y, s k+1 ) if (x, y) C k obs = 0 else Classification pressure diff. higher level pressure diff. med. level pressure diff. low level

96 Dynamics : dealing with divergence Assumption : the div is advected by the flow up to an unknown noise. It follows a stochastic process d(divṽ k (t)) = ṽ k (divṽ k (t))dt + 2ν T db(t) (25) = the div is known only up to an uncertainty that grows up linearly with time (B t is a Brownian motion). Using Ito formula and Kolmogorov s forward equation [Oksendal98] t D(t, x) + D(t, x)ṽ k + D(t, x)divṽ k ν T D(t, x) = 0, (26) with D(t, x) = E[divw(x t )] Assumption : the divergence of the flow is given by its expectation ( ζ k = divṽ k (x) D(t, x)(x)) { ζk t +ṽ k ζ k + ( ζ k ) 2 = ν T ζ k. Back (27)

97 Outline Algorithm details Back

98 Outline Discretization

99 Discretization of the vorticity-velocity equation : principles Must be done cautiously (in particular concerning the advective terms v ) Use of non-oscillatory schemes that prevent from inappropriate numerical error amplifications. Use of Total Variation Diminishing (TVD) scheme prevents from an increase of oscillations over time and enables to transport shocks. The ENO (Essentially non-oscillatory) or WENO (Weighted ENO) are the most used schemes of such family [Levy00,Shu98].

100 Time integration Use of a TVD explicit Runge-Kutta [Shu98] on X = (ξ, ζ) T : X t+1 = X m with X 0 = X t and for i = 1...m : i 1 ( ( ) ) M X i = α ik X k + tβ ik X (X k ). k=0 To be TVD, the latter scheme must verify min i,k (α ik /β ik ) = 1, (α ik > 0, β ik > 0). In practice, we use order 3 with α 10 = β 10 = 1, α 20 = 3/4, β 20 = 0, α 21 = β 21 = 1/4, α 30 = 1/3, α 31 = β 30 = β 31 = 0, α 32 = β 32 = 2/3.

101 Time integration Use of a TVD explicit Runge-Kutta [Shu98] on X = (ξ, ζ) T : X t+1 = X m with X 0 = X t and for i = 1...m : i 1 ( ( ) ) M X i = α ik X k + tβ ik X (X k ). k=0 To be TVD, the latter scheme must verify min i,k (α ik /β ik ) = 1, (α ik > 0, β ik > 0). In practice, we use order 3 with α 10 = β 10 = 1, α 20 = 3/4, β 20 = 0, α 21 = β 21 = 1/4, α 30 = 1/3, α 31 = β 30 = β 31 = 0, α 32 = β 32 = 2/3.

102 Time integration Use of a TVD explicit Runge-Kutta [Shu98] on X = (ξ, ζ) T : X t+1 = X m with X 0 = X t and for i = 1...m : i 1 ( ( ) ) M X i = α ik X k + tβ ik X (X k ). k=0 To be TVD, the latter scheme must verify min i,k (α ik /β ik ) = 1, (α ik > 0, β ik > 0). In practice, we use order 3 with α 10 = β 10 = 1, α 20 = 3/4, β 20 = 0, α 21 = β 21 = 1/4, α 30 = 1/3, α 31 = β 30 = β 31 = 0, α 32 = β 32 = 2/3.

103 Reconstruction of the scalar at the cell boundaries Second order accurate method based on a Min-Mod limiter on the regular spatial grid (i x, j y ) [Levy97] : ξ + i+ 1 2,j = ξ i+1,j x 2 (ξ x) i+1,j and ξ i+ 1 2,j = ξ i,j + x 2 (ξ x) i,j, ( with (ξ x ) i,j = Minmod 2 (ξ i,j ξ i 1,j ) x and Minmod(x 1,, x n ) =, ξ i+1,j ξ i 1,j, 2 (ξ ) i+1,j ξ i,j ), 2 x x inf i (x i ) if x i 0 i sup i (x i ) if x i 0 i 0 otherwise. idem for ξ + and ξ i,j+ 1 2 i,j+ 1 2

104 Advection-Diffusion scheme (1/2) Semidiscrete TVD central scheme [Kurganov00a,Kurganov00b] H x i+ t ξ i,j = 1,j(t) Hx 2 i 1,j(t) H y (t) H y (t) 2 i,j+ 1 2 i,j ν ξ D i,j, x y numerical convection flux derived from the monotone Lax-Friedricks flux : H x i+ 1,j(t) = u i+ 1,j(t) ] 2 [ξ +i ,j ξ i+ u i+ 1,j(t) ] 2 [ξ +i+ 12,j 2 12,j ξ i+ 12,j H y (t) = v i,j+ 1 (t) [ ] 2 ξ + + ξ v i,j+ 1 (t) [ ] 2 ξ + ξ. i,j i,j+ 1 2 i,j i,j+ 1 2 i,j+ 1 2

105 Advection-Diffusion scheme (2/2) The intermediate values of the velocities are computed with a fourth-order averaging : u i+ 1 2,j(t) = u i+2,j(t) + 9u i+1,j (t) + 9u i,j (t) u i 1,j (t) 16 v i,j+ 1 2 (t) = v i,j+2(t) + 9v i,j+1 (t) + 9v i,j (t) v i,j 1 (t). 16 The linear viscosity ξ is approximated by the fourth-order central differencing : D i,j (t) = ξ i+2,j(t) + 16ξ i+1,j (t) 30ξ i,j (t) + 16ξ i 1,j (t) ξ i 2,j (t) 12 2 x + ξ i,j+2(t) + 16ξ i,j+1 (t) 30ξ i,j (t) + 16ξ i,j 1 (t) ξ i,j 2 (t) y

106 Outline Synthetic sequence with perfect model Back

107 Synthetic sequence with perfect model Synthetic sequence with realistic initial conditions, blurred with noises (perfect model) Quantitative values : initial settings # Mask Noise h k obs (t 0) RMSE RMSE on ṽ k (t 0 ) RMSE on ṽ k (t) % (hpa) (pixel/frame) (pixel/frame) a 1 no a 2 no a 3 yes a 4 yes FIG.: Initial setup used for assimilation with a perfect model. Initial Root Mean Square Error (RMSE) on observations h k obs at initial time t 0 and on wind norm ṽ k at initial time t 0 or at all times t of the image synthetic sequences.

108 Synthetic sequence Synthetic sequence with realistic initial conditions, blurred with noises (perfect model) ground truth experiment a 2 experiment a 3

109 Synthetic sequence Synthetic sequence with realistic initial conditions, blurred with noises (perfect model) ground truth experiment a 2 experiment a 3

110 Synthetic sequence with perfect model Synthetic sequence with realistic initial conditions, blurred with noises (perfect model) Quantitative values : final results # Mask Noise h k (t 0 ) RMSE RMSE on ṽ k (t 0 ) RMSE on ṽ k (t) % (hpa) (pixel/frame) (pixel/frame) a 1 no a 2 no a 3 yes a 4 yes FIG.: Final estimation errors obtained by assimilation with a perfect model. RMSE on estimates h k and ṽ k at initial time t 0 or at all times t of the image synthetic sequences.

111 Synthetic sequence with perfect model Synthetic sequence with realistic initial conditions, blurred with noises (perfect model) Quantitative values : initial settings # Mask Noise h k obs (t 0) RMSE RMSE on ṽ k (t 0 ) RMSE on ṽ k (t) % (hpa) (pixel/frame) (pixel/frame) a 1 no a 2 no a 3 yes a 4 yes FIG.: Initial setup used for assimilation with a perfect model. Initial Root Mean Square Error (RMSE) on observations h k obs at initial time t 0 and on wind norm ṽ k at initial time t 0 or at all times t of the image synthetic sequences. Back

112 Outline Motion estimation at different layers of the atmosphere Variational assimilation Data : layered difference pressure images Dynamics : shallow-water equations Assimilation of pressure images and vector-fields in a sound dynamical model 1 Experiments on a synthetic sequence 2 Experiments on a real sequence at large scales 3 Experiments on a real sequence at finer scales Assimilation of divergence and vorticity in a simplified dynamical model 1 Experiments on synthetic sequence 2 Experiments on a real sequence at large scales 3 Experiments on a real sequence at finer scales Conclusion

113 Outline Synthetic sequence with imperfect model Back

114 Synthetic sequence with imperfect model Quantitative values : initial settings Mask Noise RMSE on ξ k (t) RMSE on ζ k (t) RMSE on ṽ k (t) % (1/frame) (1/frame) (pixel/frame) b 1 no b 2 no b 3 yes b 4 yes FIG.: Initial setup used for assimilation with an imperfect model. Root Mean Square Error (RMSE) on observations h k obs, on initial vorticity ξ k, divergence ζ k and wind norm ṽ k obtained at all times t of the image synthetic sequences.

115 Synthetic sequence Synt. sequence (imperfect model)

116 Synthetic sequence Synt. sequence (imperfect model)

117 Synthetic sequence with imperfect model Quantitative values : final results Mask Noise RMSE on ξ k (t) RMSE on ζ k (t) RMSE on ṽ k (t) % (1/frame) (1/frame) (pixel/frame) b 1 no b 2 no b 3 yes b 4 yes FIG.: Final estimation errors obtained by assimilation with an imperfect model. RMSE on estimates ξ k (t), ζ k (t) and v k (t) obtained at all times t of the image synthetic sequences.

118 Synthetic sequence with imperfect model Quantitative values : initial settings Mask Noise RMSE on ξ k (t) RMSE on ζ k (t) RMSE on ṽ k (t) % (1/frame) (1/frame) (pixel/frame) b 1 no b 2 no b 3 yes b 4 yes FIG.: Initial setup used for assimilation with an imperfect model. Root Mean Square Error (RMSE) on observations h k obs, on initial vorticity ξ k, divergence ζ k and wind norm ṽ k obtained at all times t of the image synthetic sequences. Back