The Numerical Model Training 2023 for the ICON model took place from 27 to 31 March at the headquarters of the German Meteorological Service (DWD) in Offenbach, Germany, using resources on ECMWF’s Atos high-performance computing facility (HPCF).
ICON training
For more than 20 years, the Department for Numerical Modelling and Analysis of DWD has organised training courses for using its numerical weather prediction (NWP) models. It started as a two‑hour add-on of the User Seminar (now called ICCARUS) and soon extended to a full-week event with theoretical lessons on model components and practical exercises to install and run the models. Since 2019, the courses have only covered the ICON model, which can run in global and limited-area mode. At DWD, ICON has replaced all regional applications formerly operated with the COSMO model. All partners still running COSMO will migrate to ICON soon.
ICON applications range from NWP and regional climate simulations (CLM – Climate Limited-Area Model) to the prediction of trace substance dispersion with ICON-ART. Therefore, ICON users include national meteorological services, universities and research institutions, which form the target groups of the training course. After four years without any face-to-face training due to COVID, high demand for this course was expected, and three parallel exercise groups were planned from an early stage: NWP for universities (Academia), regional climate simulations (CLM), and NWP for national meteorological services (MetService).
Using ECMWF’s Atos HPCF
As the number of trainees exceeded DWD's training resources, we contacted ECMWF User Services several months ahead of the event. We asked for the possibility to run practical exercises on ECMWF's Atos HPCF, thus allowing the MetService training group to run parallel jobs across multiple nodes. Meanwhile, the German Climate Computing Centre (DKRZ) was asked to provide computing resources for the Academia and CLM parts of the course. Both ECMWF and DKRZ agreed to make their HPCFs available.
Participants were asked to bring their own laptops. Individual trainee accounts were created by DWD. They were set up for time-based one-time password (TOTP) login and customised with the help of oathtool. Access to the Atos HPCF was realised via the Virtual Desktop Infrastructure (VDI), which provides a Linux desktop running in ECMWF’s data centre. This allowed participants to compile programs, run and control jobs, and visualise the results of their ICON runs just through a web browser. To log in, most participants were using a TOTP generator on their smartphones.
A total of 71 participants from 19 countries took part in the training. They included representatives from nine African national meteorological and hydrological services, who could attend thanks to support by the World Meteorological Organization (WMO).
For the first time, tailored exercises with the ICON model were offered to the MetService group. These exercises and corresponding materials had been developed by a group of COSMO scientists: U. Schättler and J.-N. Welß (DWD); S. Dinicila, S. Gabrian and R. Dumitrache (Romanian National Meteorological Administration); W. Interewicz and D. Wojcek (Polish Institute of Meteorology and Water Management); and A. Shtivelman (Israel Meteorological Service). Several online meetings were held in the weeks before the course to coordinate and discuss the necessary work. Colleagues from Romania and DWD were present in Offenbach to guide the participants through the exercises.
The first task was to generate a grid and external parameters for their own model area using DWD’s grid generator web service. The pre-processing step (interpolation of global ICON data to a regional grid) and the execution of an ICON-LAM simulation were then performed by all trainees on a provided grid under the guidance of the tutors. These steps then had to be transferred to the self-generated grid in the individual participants’ target areas.
The provided grid was limited in size so that a 48‑hour forecast could be run in five minutes on one Atos node. Forecasts with self-generated grids could run on up to three nodes. To ensure a quick job turnaround, ECMWF reserved a set of Atos nodes for this activity. Exercises to visualise the model results rounded off the practical learning content.
During the last afternoon, Axel Bonet and Iain Russell (ECMWF) gave online presentations and a tutorial on ECMWF’s workflow package ecFlow, used to run a large number of programs with dependencies on each other and on time in a controlled environment. In the Jupyter-Notebook-based tutorial, ecFlow was used to automate the execution of all steps that were run manually during the training week. This is necessary for the operational implementation of an NWP suite at a meteorological service.
DWD and the COSMO partners would like to express sincere thanks to ECMWF and the DKRZ for the opportunity to use their HPCFs. Special thanks go to ECMWF User Services for their extraordinary assistance during planning and conducting the exercises. Also, the presentations and tutorial on ecFlow were highly appreciated.