Mass-conserving and positive-definite semi-lagrangian advection in NCEP GFS: Decomposition for massively parallel computing without halo

Transcription

1 Mass-conserving and positive-definite semi-lagrangian advection in NCEP GFS: Decomposition for massively parallel computing without halo Hann-Ming Henry Juang Environmental Modeling Center, NCEP NWS,NOAA,DOC,USA 1

2 Introduction The common elements for traditional semi- Lagrangian method are Iteration to find departure and/or mid-point values Interpolation from regular grid points to departure and/or mid-points Require halo grids in MPP Advantage of semi-lagrangian Method Allowable larger time step, saving time Use linear grid in spectral model; same grid-point with higher resolution of spectral truncation 2

3 q t + u q x + v q y = 0 Dq Dt = 0 q A n +1 = q D n 1 M A D 3

4 Starting from mid-point A M D No guessing and no iteration but one 2-D interpolation and one 2-D remapping 4

5 Proposed Method Splitting semi-lagrangian advection advection in one direction first then advection in another direction temporal and spatial splitting Advantage no guessing and no iteration 1-D interpolation and remapping possible no halo (with transpose) incremental implementation Possible to add mass conservation and positive definite advection 5

6 q t + u q x + v q y = 0 q t q t Dq Dt X direction X direction X direction + q t Y direction + u q x + q t + Dq Dt Y direction + u q x + v q y = 0 Y direction = 0 + v q y = 0 6

7 X D = X M U M Δt X A = X M + U M Δt q A *n +1 = q D n 1 Interpolation q D n 1 relocation q A *n +1 D M A remapping *n +1 q 7

8 relocation q A n +1 A q n +1 Y D = Y M V M Δt Y A = Y M + V M Δt q A n +1 = q D *n +1 M remapping Interpolation q D *n +1 D 8

9 Ay Ay Dx Mx Ax My My Dx Mx Ax Dy Dy 9

10 Gaussian 256 x 128 with time step of 1800 sec 10

11 Gaussian 256 x 128 with time step of 1800 sec Across north pole 11

12 Gaussian 256 x 128 with time step of 1800 sec Across south pole 12

13 at north pole 13

14 at equator 14

15 at south pole 15

16 one complete revolution 16

17 [ ] [ ] L 1 = I q q T I q T [ ] [ ] L = Max q q T Max q T L 2 = 1 I[ ( q q T ) 2 ] 2 1 I[ ( q T ) 2 ] 2 max = Max [ q ] Max q T Max[ q T ] Min q T [ ] [ ] min = Min [ q ] Min q T Max[ q T ] Min q T [ ] [ ] where I[A] = 1 Acosφdφdλ 4πa 2 φλ Max[A] = global_max_of _ A Min[A] = global _min_of _ A 17

18 Compare Errors min max L 1 L 2 L FFSL-5 128x64 FFSL-3 256x128 NISL 128x64 NISL 256x128 NISL 512x E E E E E

19 For mass conservation, let s start from continuity equation ρ t + ρu x + ρv y + ρζ ζ = 0 ρ t (X ) + ρu x + ρ t (Y ) + ρv y + ρ t (Z ) + ρζ ζ = 0 Consider 1-D and rewrite it in advection form, we have ρ t (X direction) + u ρ x = ρ u x Advection form is for semi-lagrangian, but it is not conserved if divergence is treated as force at mid-point, So divergence term should be treated with advection 19

20 Divergence term in Lagrangian sense is the change of the volume if mass is conserved, so we can write divergence form as u x Lagrangian _ sense = 1 Δ x dδ x dt Put it into the previous continuity equation, we have dρδ x dt ρδ x t X direction X direction = 0 + u ρδ x x = 0 which can be seen as ( ρδ ) x departure = ( ρδ ) x arrival 20

21 How about mass conservation for tracer? If we use tracer and continuity equation as following together q t + u q x + v q y + ζ q ζ = 0 ρ t + ρu x + ρv y + ρζ ζ = 0 Then density weighted tracer can be treated as conservation as ρq t + ρqu x + ρqv y + ρqζ ζ = 0 Combine it with continuity equation, we can have conserved tracer advection dρqδ dt = 0 dρδ dt = 0 21

22 We do X D L = X M L U M L Δt X D R = X M R U M R Δt Δ D = X D D R X L Interpolation ρ D n 1 X A L = X M L + U M L Δt X A R = X M R + U M R Δt Δ A = X A A R X L *n +1 relocation ρ A M L M R X D L D R M A L A R Δ D Δ A ρ D n 1 Δ D = ρ A n +1 Δ A 22

23 n-1 D L Δ D D R n X n+1 A L A R Δ A ρ D n 1 Δ D = ρ A n +1 Δ A 23

24 The given value can be presented piece-wisely by D R S n 1 D (x)dx = n S +1 A (x)dx D L ρ = S(x) so the previous mass equality can be replaced as following A R A L Also we want to make sure that total mass is conserved as S n 1 R (x)dx = S n 1 D (x)dx = n S +1 A (x)dx = n S +1 R (x)dx where subscript R is regular grid D is departure grid A is arrival grid for This implies that mass conservation should be used during interpolation from regular cell to departure cell and from arrival cell to regular cell. thus, we apply monotonic PPM for S(x). 24

25 ( total mass) initial total mass ( ) initial total mass ( )

26 Isochronal flow Any given point will return to its original location after a given period of time. Gaussian grid dimensioned 512 x 256. Rotate coordinates so equator goes through (39N,77W), thus giving flow over real poles. Apply non-divergent global wave number 4-20 perturbation displacement (of standard deviation size non-dimensional vorticity) using a random number generator. Set return period to 10 days. 26

27 27

28 28

29 29

30 30

31 31

32 Decomposition in NCEP GFS Current NCEP GFS uses 1D decomposition with MPI and thread with OpenMP. Transpose with MPI_AllToAllv is used to move between two sub-domains for spectral transform. First sub-domain has some given latitudes with all longitudes grids, which is for FFT in longitudinal direction. Second sub-domain has some given zonal wave numbers with all meridian wave numbers, which is for Legendre transform in meridian direction. Transpose between two sub-domains, thus there is no halo required. 32

33 Implement into NCEP GFS The same first sub-domain is used to compute semi- Lagrangian advection in any given latitudinal global circle. All departure and arrival points are in the same circle, so no halo is needed. Then transpose first sub-domain to another grid-point sub-domain, which has all Gaussian latitude points but some longitude points. Therefore semi- Lagrangian advection can be computed in any given longitude with all latitude points, so, again, no halo is required. The PPM mass conserving between reduced grid and full grid in any given latitude is also applied. 33

34 np. np.. <=> sp Transpose sp No halo is required No increasing memory with Increasing number of PE(cpu) np 34

35 1D halo Halo Exchange Extra memory is required, which may be as huge as computing grid while number of MPP cpu increases. 2D 35

36 Case test in NCEP GFS Arbitrary date is selected. Modify NCEP GFS IO into grid-point data. Negative tracers are replaced with zero value at the initial time. T126 L64 resolution is tested. Model physics is included. Two runs are compared; control: Spectral advection in horizontal, finite difference in vertical asoperational GFS. nislfv: Non-iteration mass conserving positive definite semi- Lagrangian on tracers. 36

37 24h nislfv 72h control 37

38 24 h nislfv 72 h control 38

39 control 06h fcst specific humidity at model layer 40 nislfv 39

40 control 12h fcst specific humidity at model layer 40 nislfv 40

41 control 24h fcst specific humidity at model layer 40 nislfv 41

42 control 72h fcst specific humidity at model layer 40 nislfv 42

43 control 6hr fcst cloud water at model layer 35 nislfv 43

44 control 12hr fcst cloud water at model layer 30 nislfv 44

45 control 24hr fcst cloud water at model layer 30 nislfv 45

46 control 72hr fcst cloud water at model layer 5 nislfv 46

47 control 6hr fcst precipitation nislfv 47

48 control 24hr fcst precipitation nislfv 48

49 control 72hr fcst precipitation nislfv 49

50 Conclusion & Future Work Modified traditional semi-lagrangian without iteration to locate mid-/departure-points, but require interpolation and remapping with temporal and spatial split computation. Mass conserving is included with consideration of semi- Lagrangian for divergence. Positive definite is applied with monotone piecewise parabolic method (PPM) for interpolation/remapping. Due to spatial split, no halo is required since all required data for computation are all in the partial domain through transpose. Since no halo, there is no extra memory request, but it may have more data in communication than the method with halo and small number of cpu. Implement all prognostic variable, not only tracers, to have larger model time step to save integration cost. 50