P.H. Lauritzen1, M.A. Taylor2, S. Goldhaber1, P.A. Ullrich3, J
P.H. Lauritzen1, M.A. Taylor2, S. Goldhaber1, P.A. Ullrich3, J
CAM-SE-CSLAM: Consistent finite-volume transport ! with spectral-element dynamics! 1 Lauritzen , 2 Taylor , 1 Goldhaber , 3, Ullrich 2 Overfelt , 1 Nair , 1 Kelly P.H. M.A. S. P.A. J. R.D. R. 1National Center for Atmospheric Research, Boulder 2 2 ! ! Sandia National Laboratories, New Mexico !!!! Univeristy of California, Davis ! ! NCAR is sponsored by the Foundation (NSF)! 302 Ramachandran D. Nair,National Michael N. LevyScience and Peter H. Lauritzen Introduction! Problem: Coupling! Results! Setup: Transport inert and passive tracers in Jablonowski & Williamson (2006) baroclinic wave Gaussian “ball”! sigma-pressure floating Lagrangian vertical coordinate. Zonally symmetric (smooth)! /4 (-1,+1) (+1,+1) Slotted cylinder (rough)! e x 1 Q (-1,-1) /4 Physical Domain x 2 (+1,-1) + /4 Computational Domain The continuity equation for air mass (2) is solved with CAM-SE and the continuity for tracers (3) is solved with CSLAM. We need the CSLAM solution (3) to reduce to the CAM-SE solution for air mass (2) when q=1: S e Fig. 9.22 A schematic diagram showing the mapping between each spherical tile (element) of the physical domain (cubed-sphere) S onto a planar element e on the computational domain C (cube). For a DG discretization each element on the cube is further mapped onto a unique reference element Q, which is dened by the Gauss-Lobatto-Legendre (GLL) quadrature points. The horizontal discretization of the HOMME dynamical cores relies on this grid system. ! ! ! ! ! ! Computational throughput for many-tracer applications The cubed-sphere has the attractive feature that the domain S is naturally deDiscretization is mimetic => mass-conservation and total is composed into non-overlapping quadrilateral elements (tiles) eS . This topology well-suited high-order element-based methods such as spectral element or DG energyfor conservation on element methods, and amenable to efcient parallel implementation. Each face of the cubedConserves angular momentum very well 2 sphere has Ne × Naxial e elements, thus Nelm = 6 Ne elements span the entire spherical Nelm S (Lauritzen et =al., 2014) domain such that S ∪ e=1 e ; in Fig. 9.22 Ne is 4. There exists a one-to-one corS and the planar element respondence between spherical element eS on e Supports staticthemesh-refinement and retains formal order on C as depicted in Fig. 9.22. The element-wise continuous mapping allows us to of accuracy! perform integrations on the sphere in a mapped (local) Cartesian geometry rather Hon ighly scalable than the surface of the sphere. The High-Order Method Modeling Environment (HOMME) developed atsimilar NCAR relies on this grid system (Dennis et al, 2005). AMIP-climate to current model CSLAM (Conservative semi-Lagrangian Multi-tracer scheme) is based on a cell-integrated semi-Lagrangian approach that, unlike CAM-SE, allows for long time-steps, is locally and globally conservative, has a linear correlation shape-preserving limiter/filter and is geometrically very flexible meaning that the method can accommodate any spherical grid constructed from great-circle arcs. Since geometric information computed for the transport of one trace species can be re-used for each additional tracer, the scheme (CSLAM) is termed multi-tracer efficient. CAM-SE-CSLAM CAM-SE ! ! ! ! ! Solution: flux-form ! ! ! ! ! ! ! !! (a) CAM-SE-CSLAM CAM-SEreference (b) - Compute air mass-flux through CSLAM control volume sides using method developed by Taylor et al. - Find swept fluxes (using Newton iteration) so that CSLAM swept flux matches CAM-SE flux to round-off: a-b: Find perpendicular (x and y) CSLAM fluxes that match CAM-SE fluxes c: flux areas define departure points d: add extra point to swept side and iterate so that 2D CSLAM flux match CAM-SE day 15! day 15! day 17! day 17! day 15! Throughput data produced on NCAR’s Yellowstone supercomputer. day 17! 1 degree configuration (NE30NP4NC3), 40 tracers ntask 256, 1 degree (NE30NP4NC3), Yellowstone computer 120 100 SE: Total tracers CSLAM: Total tracers CSLAM: fill halo CSLAM: reconstruction CSLAM: remap CSLAM: high-order weights CSLAM: iterate 100 80 60 40 20 0 0 20 40 60 80 100 120 140 Number of tracers Day 7.5! Day 7.5! Day 7.5! Day 9! Day 9! day 5! day 5! day 5! day 10! day 10! day 10! Day 9! CAM-SE ! ! ! ! ! Performance! Surface pressure evolution for CAM-SE and CAM-SECSLAM match to round-off every time-step (not shown) GLL Quadrature Grid Properties: Simulations are performed with 30 vertical levels with 1 degree CAM-SE (left column), CAM-SE-CSLAM (middle column) and reference solution is 0.25 degree CAM-SE. Time in seconds e S 160 180 200 10 1 CSLAM SE One element per processor! 1000 number of processors Summary! ! Acknowledgments! • CSLAM has been consistently coupled with spectral Weelement thank I. Güor laboratory assistance, Mary Juana for (SE) for dynamical core, i.e. CSLAM conserves seeds, Herbmass, Isside for care, and(monotone), M.I. Menter for tracer is greenhouse shape-preserving and questionable advice. Funding this project was preserves statistical a constant mixing ratiofordistribution (freeprovided the Swarthmore College Department of streamby preserving). Biology, a Merck stipend, and my that • CSLAM takes summer a 3x longer time-step for mom. tracers[Note than SE people’s titles omitted.] • CSLAM hasare been shown to be more accurate for tracers with steep gradients compared to SE and equally accurate for smooth tracer distributions • CSLAM preserves linear correlation even under forced conditions in `toy’ terminator chemistry test whereas SE does not • CSLAM is faster than SE when transporting more than approximately 18 tracers • Even though CSLAM needs a halo of 3 cell width, it scales to one element (3x3 CSLAM control volumes) per processor More information: contact [email protected] ; Home page: http://www.cgd.ucar.edu/cms/pel ! The terminator ‘toy’-chemistry test: A simple tool to assess errors in transport schemes (Lauritzen et al., 2015, GMD)! Transport 2 reactive species (Cl and Cl2) in Jablonowski and Williamson (2006) flow. The sources and sinks for Cl and Cl2 are given by a simple, but non-linear, "toy" chemistry that mimics photolysis-driven conditions near the solar terminator, where strong gradients in the spatial distribution of the species develop near its edge. Despite the large spatial variations in each species, the weighted sum Cly=2CL+Cl2 should always be preserved Total tracers time in seconds terms of local orthogonal Cartesian coordinates x 1 , x2 ∈ [− /4, /4], as shown in Fig. 9.22 . Thus C is essentially a union of six non-overlapping sub-domains (faces) and any point on C can be uniquely represented by the ordered triple (x 1 , x2 , ) where = 1, . . . , 6, is the cube-face or panel index. The projections and the logical orientation of the cube panels are described in Nair et al (2005b) and Lauritzen et al (2010). CAM-SE (Community Atmosphere – Spectral The equiangular central projection results in a Model uniform element width ( x 1 = x2 ) on C , which is an advantage for practical implementation. Figure 9.22 proElements) is based on a continuous Galerkin spectral finite vides a schematic diagram of thehorizontal mapping between the physical domain S (cubedelement method in the directions and a hybrid sphere) and the computational domain C (cube).