Post-processing CESM Model Output Using Python-based Utilities
Post-processing CESM model outputusing Python-based utilitiesSpring 2020 HPRC Short CourseApril 10, 2020ABISHEK GOPALAssistant Research ScientistiHESP, Texas A&M OceanographyTexas A&M High Performance Research ComputingPost-processing CESM model output using Python – HPRC Short Course – Spring 20201
Helpful HPRC resources Ada quick start guide– https://hprc.tamu.edu/wiki/Ada:QuickStart Introduction to the Ada and Terra clusters – Short course (Spring 2020)– ntro to HPRC clusters 2020 spring.pdfSubmit tickets to help@hprc.tamu.eduPost-processing CESM model output using Python – HPRC Short Course – Spring 20202
Upcoming HPRC short courses Introduction to Entos - QM Simulation Software for PCs, HPC, and the Cloud– Instructor: Thomas Miller (Professor at Caltech and Entos co-founder) and Daniel Smith(Entos)– Time: Friday, April 10, 1:30PM-4:00PM Searching for the Cure – COVID-19 Classical Molecular Dynamics with NAMD– Instructor: Dr. Lisa M. Perez– Time: Friday, April 17, 10:00AM-12:30PM Searching for the Cure – COVID-19 Drug Docking with AutoDock andSchrödinger– Instructor: Dr. Lisa M. Perez– Time: Friday, April 17, htmlPost-processing CESM model output using Python – HPRC Short Course – Spring 20203
Course structure Structure of CESM model output and netCDF formatInstalling and configuring the CESM postprocessing packageGenerating time-averages and climatology filesShort break (15 minutes)Generating diagnostics of CESM model runsPangeo for geosciencesLoading and visualizing datasets using xarray and cartopySpeeding up computations using daskPost-processing CESM model output using Python – HPRC Short Course – Spring 20204
Expectations for this course Learn about performance benefits of parallel python tools in generatingclimatology filesLearn about the available diagnostic plots setsGet some hands-on experience with the CESM diagnostics toolLearn about the future trends in using Python for geosciencesPost-processing CESM model output using Python – HPRC Short Course – Spring 20205
The Community Earth System Model sm2Post-processing CESM model output using Python – HPRC Short Course – Spring 20206
Publicly available CESM model output arthsystemgrid.org/Sample datasets for the tutorialavailable on Ada at/scratch/training/CESM post/datasetsPost-processing CESM model output using Python – HPRC Short Course – Spring 20207
Directories in esm1 tutorial 130619 full.pdfPost-processing CESM model output using Python – HPRC Short Course – Spring 20208
History vs Timeseries format History files contain a set of variables output at a given time slice in asingle file. Timeseries files contain single variable data for all times CESM writes model output in short-term archive as history files Variables with larger dimensions (Ex: TEMP) are usually output as alower frequency history files, than smaller variables (Ex: SST). Timeseries files are usually shared publicly. How to transpose history files to timeseries files?Post-processing CESM model output using Python – HPRC Short Course – Spring 20209
CESM Case naming conventionAdam Phillips, NCARPost-processing CESM model output using Python – HPRC Short Course – Spring 202010
NetCDF erial/2015/01/data mgmt epcc/netcdf.pdfPost-processing CESM model output using Python – HPRC Short Course – Spring 202011
Tools to process and visualize NetCDF files NetCDF Operators (NCO)Climate Data Operators (CDO)NCAR Command Language (NCL)MATLABInteractive Data Language (IDL)Panoply (from NASA GISS)PyNGL and software.htmlPost-processing CESM model output using Python – HPRC Short Course – Spring 202012
CESM Postprocessing toolPyAveragerHistory filesDiagnosticNCL scriptsDiagnosticPlotsOMWGAMWGPMWGLMWGCESM Modeloutput ng CESM model output using Python – HPRC Short Course – Spring 2020PyConformCMIPcompliantoutput13
Generated diagnosticsPost-processing CESM model output using Python – HPRC Short Course – Spring 202014
Diagnostic setsModel vs ObservationsModel vs ControlPost-processing CESM model output using Python – HPRC Short Course – Spring 2020Model time series15
Some ocean diagnostics plotsType of plotQuantity2D Surface flux fieldsSENH F, LWUP F, EVAP F, PREC F2D Surface fieldsSSH, SST, etc3D fields (Zonally-averaged)TEMP, SALT, IAGEFields at various depth levelsTEMP, SALT, UVEL, VVEL, WVELVector fields at various depth levelsvelocityMOCPost-processing CESM model output using Python – HPRC Short Course – Spring 202016
PyAverager performanceSheri Mickelson1 & Alice Bertini2, NCARPOP 0.1deg grid 15 times faster than NCOPost-processing CESM model output using Python – HPRC Short Course – Spring 202017
Obtaining/installing the CESM diagnostics toolhttps://github.com/abishekg7/CESM postprocessing Forked from NCAR’s github.Incorporates Ada-specific configurationcreated by Alper AltuntasSimplified installation processPost-processing CESM model output using Python – HPRC Short Course – Spring 202018
Creating a post-processing case on Ada1. Activate the PP virtual environmentml purgeml Miniconda2/4.3.21ml Python/2.7.15-foss-2018bexport POSTPROCESS PATH /scratch/training/CESM post/CESM postprocessingsource activate POSTPROCESS PATH/cesm-env22. Create a post-processing case directory in your scratch directorycreate postprocess -case [your-case-directory]Exercise: Try above commands on AdaPost-processing CESM model output using Python – HPRC Short Course – Spring 202019
Post-processing workflow1. Edit configuration filesØ env postprocess.xmlØ env diags ocn.xml, env diags atm.xml, etc2. Edit job submission scripts (ocn averages, ocn diagnostics, etc)Ø Change project account, num cores, logging level, etc3. Generate averaged filesbsub ocn averages4. Generate diagnosticsbsub ocn diagnosticsPost-processing CESM model output using Python – HPRC Short Course – Spring 202020
Variables in env postprocess.xmlVariablePurposeCASEROOTLocation of caseroot. Same as PP CASE PATH forstandalone postprocessingPP CASE PATHDisk location of postprocessing case directoryCASECESM Case nameDOUT S ROOTCESM model output directory. Input for post-processing COMP GRIDString specifying grid for component COMP GENERATE TIMESERIESIf True, converts history to timeseries filesGENERATE AVGS COMP If True, calls averaging script for component COMP GENERATE DIAGS COMP If True, calls diagnostics script for component COMP Post-processing CESM model output using Python – HPRC Short Course – Spring 202021
Sample values for env ining/CESM post/pp cases/pp controlPP CASE PATH/scratch/training/CESM post/pp cases/pp controlCASEb.e13.B1950C5.ne30 g16.ihesp24 1950cntl.001DOUT S ROOT/scratch/training/CESM post/datasets/b.e13.B1950C5.ne30 g16.ihesp24 1950cntl.001ATM GRIDne30OCN GRIDgx1v6ICE GRIDgx1v6LND GRIDne30Post-processing CESM model output using Python – HPRC Short Course – Spring 202022
Grid 0/cesm/cesm doc 1 0 4/a3714.htmlGrid descriptorType of grid1.9x2.5 or f19Regular lon/lat finite volume grids of approximately1.9 and 2.5 degree lon and lat respectively.T85Spectral lon/lat grids with 85 as the spectraltruncation value for the resolutionne30np4Cubed sphere resolutions where X and Y areintegers. The short name is generally ne[X]pt1Single pt gridgx1v6 or g16Displaced pole grid of approximately 1 degreeresolution. 6 denotes the grid version.tx0.1v2Tripole grid of approx 0.1 degree resolution. 2denotes the grid version.Post-processing CESM model output using Python – HPRC Short Course – Spring 202023
Variables in env diags ocn.xmlVariablePurposeOCNDIAG DIAGOBSROOT/scratch/training/CESM post/obs rootOCNDIAG TAVGDIR PP CASE PATH/ocn out/climo. OCNDIAG YEAR0. OCNDIAG YEAR1OCNDIAG WORKDIR PP CASE PATH/ocn out/diag work. OCNDIAG YEAR0 OCNDIAG YEAR1OCNDIAG netcdf formatnetcdfLarge (64-bit)OCNDIAG MODEL VS OBSTRUEOCNDIAG MODEL VS CONTROLFALSEOCNDIAG MODEL TIMESERIESTRUEPost-processing CESM model output using Python – HPRC Short Course – Spring 202024
After averaging step (check OCNDIAG WORKDIR)1. 12 Climatology files (averaged for each month)Ø CASE. YEAR0- YEAR1. MONTH climo.nc2. Yearly-averaged files for every yearØ CASE. YEAR0.nc3. Time average (Single file)Ø tavg. YEAR0- YEAR1.nc4. Monthly average (Concatenation of the climatology files)Ø mavg. YEAR0- YEAR1.ncPost-processing CESM model output using Python – HPRC Short Course – Spring 202025
Exercise – generating averages1. Create a new postprocessing case for the transient case2. Copy env postprocess.xml and env diags ocn.xml from the existingcontrol case configuration at/scratch/training/CESM post/pp cases/pp control and edit for thetransient case3. Submit ocn averagesPost-processing CESM model output using Python – HPRC Short Course – Spring 202026
Short break!(15 minutes)Post-processing CESM model output using Python – HPRC Short Course – Spring 202027
Submit diagnostics1. bsub ocn diagnosticsPost-processing CESM model output using Python – HPRC Short Course – Spring 202028
Generated ocean diagnosticsPost-processing CESM model output using Python – HPRC Short Course – Spring 202029
Generated ocean diagnosticsPost-processing CESM model output using Python – HPRC Short Course – Spring 202030
Exercise – generating diagnostics1. Turn on the correct diagnostic sets in env diags ocn.xml2. Check the ocn diagnostics script for correctness3. Submit ocn diagnosticsPost-processing CESM model output using Python – HPRC Short Course – Spring 202031
Limitations of the current diagnostics tool Patchwork of many different components. Harder to locate centralizedhelp/documentationAdding new diagnostic sets is a challengePorting to a new machine can be trickyPost-processing CESM model output using Python – HPRC Short Course – Spring 202032
Pangeo frameworkRyan AbernatheyPost-processing CESM model output using Python – HPRC Short Course – Spring 202033
essing CESM model output using Python – HPRC Short Course – Spring 202034
What is xarray An extension of the pandas library to workwith labelled, N-dimensional arrays xarray.Dataset is an in-memoryrepresentation of the netCDF file format xarray works seamlessly with the dasklibrary to enable parallel computationsmore easilyPost-processing CESM model output using Python – HPRC Short Course – Spring 202035
Dask task ing CESM model output using Python – HPRC Short Course – Spring 202036
Dask vs Numpy Numpy is faster than Dask for a smaller problem sizeFor larger problems, Dask achieves better scalabilityNumpy cannot load very large arrays, whereas Dask works just ng CESM model output using Python – HPRC Short Course – Spring 202037
Launch JupyterLab notebook Go to https://portal.hprc.tamu.edu/Choose Ada, then Interactive Apps - JupyterLab from the top paneEnter path to JupyterLab environment/scratch/training/CESM post/conda/envs/jupyterlabRequest 1 hour on 4 cores, 1 GB memoryper core (4 GB in total)Click LaunchPost-processing CESM model output using Python – HPRC Short Course – Spring 202038
Download tutorial notebooks ssh username @ada.tamu.educd into your scratch directoryClone the notebook repository from githubgit clone https://github.com/abishekg7/pangeo binders.gitPost-processing CESM model output using Python – HPRC Short Course – Spring 202039
Launch JupyterLab notebookPost-processing CESM model output using Python – HPRC Short Course – Spring 202040
Additional resources CESM diagnostics tool tutorials––– GeoCAT– u/pages/software.htmlPangeo g CESM model output using Python – HPRC Short Course – Spring 202041
Acknowledgements Fred Castruccio and Alper Altuntas, NCARSanjiv Ramachandran and Dapeng Li, iHESPThe HPRC teamPost-processing CESM model output using Python – HPRC Short Course – Spring 202042
Questions?Post-processing CESM model output using Python – HPRC Short Course – Spring 202043
Post-processing CESM model output using Python - HPRC Short Course - Spring 2020 3 Upcoming HPRC short courses Introduction to Entos - QM Simulation Software for PCs, HPC, and the Cloud - Instructor: Thomas Miller (Professor at Caltech and Entos co-founder) and Daniel Smith
American Legion 5th District 5th District Commander, Western Springs Cicero Post #96 DesPlaines Post #36 George L. Giles Post #87, Chicago Maywood Post #133, Melrose Park Morton Grove Post #134 Schiller Park Post #104 T.H.B. Post #187, Elmhurst Edward Feely Post #190, Brookfield Richard J. Daley Post #197, Chicago
Paper output cover is open. [1202] E06 --- Paper output cover is open. Close the paper output cover. - Close the paper output cover. Paper output tray is closed. [1250] E17 --- Paper output tray is closed. Open the paper output tray. - Open the paper output tray. Paper jam. [1300] Paper jam in the front tray. [1303] Paper jam in automatic .
CISM, the Community Ice Sheet Model, originates from the Glimmer and Glimmer{CISM projects (Rutt et al., 2009)1. The current name re ects the project's evolution from a stand-alone ice sheet model to a fully supported, coupled component of the Community Earth System Model, or CESM. CISM is a numerical model a collection of software libraries .
CISM, the Community Ice Sheet Model, originates from the Glimmer and Glimmer{CISM projects (Rutt et al., 2009)1. The current name re ects the project's evolution from a stand-alone ice sheet model to a fully supported, coupled component of the Community Earth System Model, or CESM. CISM is a numerical model a collection of software libraries .
To create a corner post, install the 900 profile over the face of the post, for the new run of fence, and insert a post infill over the exposed recess of the corner post. 2420mm post face to post centre HOUSE 1 HOUSE 2 5mm clearance minimum 90º 90 configuration Insert post infill Install 90º profile over the face of the mid-post.
Facebook Post 3,000 Per Post Twitter Post 2,000 Per Post Instagram Post 2,000 Per Post Youtube Post 1,000 Per Post Landing Page Images & text 3,000 Admin. Fee OFFLINE Cabin Branding Overhead Lockers 5,000 Per a/c & Month . ADVERTISING RATES COUNTRY TOTAL SUBSCRIBERS CPM ( ) COST ( ) GB 1,955,061 30 58,652 IT .
3N SOL VALVE PORT AIR inj9 Output PNP for port air 3O SOL VALVE DOUBLE THROTTLE LS9 Output PNP for D throttle cont 3P SOL VALVE REL 1 inj10 Output PNP for relief sol 1 2A N / U Spare Speed Input Available Switch input 2B Tachometer output LS7 Output Dedicated 2C AUTO ECU idle1 Output N/A 2D AUTO ECU BARO Output N/A
the topic of artificial intelligence (AI) in English law. AI, once a notion confined to science fiction novels, movies and research papers, is now making a tremendous impact on society. Whether we are aware of it or not, AI already pervades much of our world, from its use in banking and finance to electronic disclosure in large scale litigation. The application of AI to English law raises many .
