The Israel Meteorological Service (IMS) has been operating a small high-performance computing (HPC) cluster of about 1,400 cores over the last decade. This local resource was sufficient for limited-area model high-resolution operational runs of the Eastern Mediterranean, but it did not allow any contribution to regional projects, extensive research in numerical weather prediction (NWP), or regional ensemble runs. In June 2019, ECMWF’s Council approved an Optional Programme for Co-operating States that cannot apply for full Member State status, which enables them to use some of ECMWF’s HPC resources. Over the last two years, IMS has used this option for research and the development of NWP for southeast Europe. It has also used it to contribute to international projects, such as the World Meteorological Organization’s South-East European Multi-Hazard Early Warning Advisory System (SEE-MHEWS-A).
High-resolution ensemble system
IMS is a member of the European Consortium for Small-scale Modelling (COSMO). However, due to the remote location of Israel from the other members, the COSMO-LEPS (COSMO Limited-Area Ensemble Prediction System) 7 km ensemble did not cover the region of Israel and its neighbouring countries. The Optional Programme allowed IMS to build a high-resolution ensemble forecasting system for the Eastern Mediterranean (25–39°E, 26–36°N), including Egypt, Cyprus, Lebanon, Israel and Jordan, on ECMWF’s HPC facility. This COSMO ensemble suite (CO-IL-EPS) runs twice daily at the convection-permitting resolution of 2.5 km (Figure 1).
Initial data and lateral boundary conditions are taken from the ECMWF ensemble (ENS) and the deterministic high-resolution forecast (HRES). CO-IL-EPS includes 20 ensemble members and the deterministic model CO-IL2.5. Each member runs on 10 nodes of ECMWF’s HPC facility for a total of 210 nodes for the entire suite. The spatial scale of geographic features in this region cannot be resolved in ECMWF’s current HRES, which has a horizontal grid spacing of about 9 km, and even less in the current ENS, which has a horizontal grid spacing of about 18 km. The Judea hills, for example, reach a height of more than 900 m above sea level, and the Dead Sea, at a distance of less than 15 km away, drops to more than 400 m below sea level. Some important meteorological processes are not described properly in ECMWF’s forecasts, such as convective precipitation patterns (which are the typical precipitation mechanisms of this region). Thus, high-resolution probabilistic products gradually became essential tools for the IMS forecasters. Some of these products are also available to the other countries of the region through the IMS public website.
The SEE-MHEWS-A deterministic system
Alongside the COSMO ensemble system, IMS has set up another NWP suite based on the 2.5 km resolution ICON-LAM model (ICON-IL2.5) to provide deterministic weather forecasts for the southeast Europe region (4–45.5°E, 25.5–53°N, see Figure 2), as a part of the WMO’s SEE-MHEWS-A project. This early warning system includes several ECMWF Member and Co-operating States and is intended to benefit 18 WMO Members of the region: Albania, Bosnia-Herzegovina, Bulgaria, Croatia, Cyprus, Greece, Hungary, Israel, Jordan, Lebanon, North Macedonia, Republic of Moldova, Montenegro, Romania, Serbia, Slovenia, Turkey, and Ukraine. The SEE-MHEWS-A project supports national meteorological and hydrological services in fulfilling their mandate to provide timely and accurate warnings to minimise the impacts on people, infrastructure, and industry of hazardous weather events. For this suite, initial and lateral boundary conditions are taken from the IFS HRES. The suite runs twice per day with the model utilising 57 compute nodes since September 2020.
CO-IL-EPS and ICON-IL2.5 are also used for hydrological modelling. ICON-IL2.5 forecasts were used as input for the hydrological models of the SEE-MHEWS-A project pilot phase for the Vrbas River in Bosnia and Herzegovina and Vardar/Axios River in North Macedonia and Greece (see ECMWF Newsletter No. 170 for details of ECMWF’s support for SEE-MHEWS‑A). Both CO-IL-EPS and ICON-IL2.5 precipitation forecasts are used to drive the hydrological models of the Israeli Water Authority. First experience shows that specific precipitation rate percentiles of CO-IL-EPS are significantly less noisy than the deterministic forecasts and hence are particularly valuable for hydrological forecasting.
In addition to CO-IL-EPS and ICON-IL2.5 operational suites, ECMWF’s HPC facilities are used by IMS for research purposes. New parametrizations and developments conducted in the framework of the COSMO consortium (https://www.cosmo-model.org/) are continuously being tested and verified. The operational suites are updated by improved versions once per year or even more frequently. Some of these scientific developments, which could not be conducted without ECMWF’s HPC facilities, are highlighted in the last section.
Access to products
CO-IL-EPS and ICON-IL2.5 products are visualised on several platforms. ICON-IL2.5 forecasts are presented on the Common Information Platform (CIP) of the SEE-MHEWS-A project. The reader is referred to ECMWF Newsletter 170 for more details. Both CO-IL-EPS and ICON-IL2.5 are visualised on internal forecasting platforms used by IMS forecasters. Additional products are being continuously developed and added to these platforms almost on a weekly basis.
Most common synoptic maps of CO-IL-EPS, ICON-IL2.5, and the IFS are presented to the public on the IMS website (https://ims.gov.il/en/ModelMaps). Figure 2 shows an example of an ICON-IL2.5 3-hourly precipitation forecast over southeast Europe. Additional domains can be selected, including the Balkans, for the convenience of SEE-MHEWS‑A countries.
CO-IL-EPS and ICON-IL2.5 raw data forecasts on a limited area are also freely accessible through IMS web services. This allows the private sector to develop products oriented for specific users. Moreover, ICON-IL2.5 forecasts are operationally distributed for the use of the Mediterranean cyclone tracking research as part of the COST Action CA19109 and for verification purposes as part of the COSMO consortium.
Visualisations using ECMWF tools
Utilisation of ECMWF’s HPC facilities enabled the development of probabilistic products for CO-IL-EPS using ECMWF’s meteorological software: the Metview meteorological workstation application, the Metview Macro language, and Magics meteorological plotting software.
Figure 3 shows an example of post-processing using these tools for probabilistic precipitation forecasting. Panel (a) shows the radar and rain gauge composite for a heavy precipitation event that occurred in Tel Aviv on 4 January 2020 between 06 and 12 UTC. During this short event, 74 mm of rain fell within 2 hours (50 mm within half an hour), causing the drowning of a young couple in a flooded elevator in south Tel Aviv. Panels (b)-(d) present the CO-IL2.5 and CO-IL-EPS forecasts from 00 UTC on 3 January 2020. Panel (b) presents the CO-IL2.5 deterministic forecast, while panels (c) and (d) present the CO-IL-EPS probabilities to exceed 10 and 20 mm/6 h, respectively. Since precipitation is uncertain in space, the probability forecasts were upscaled to boxes of 12x12 km2. This upscaling works as follows: if an ensemble member precipitation forecast passes a certain threshold at any grid point within such a box, the event is counted for the entire box. For estimating the probability of the event occurring within the box, the relative number of events is calculated. One can see that the probabilistic forecasts (unfortunately not operational during the event) predicted the location of the heavy precipitation better than CO-IL2.5.
Figure 4 shows the corresponding CO-IL-EPS precipitation meteogram for Tel Aviv, also obtained using ECMWF’s tools. As can be seen, the 90% percentile of the ensemble reaches only 26 mm/6 h, but there was an ensemble member that showed 74 mm/6 h for Tel Aviv. Note that this meteogram is not upscaled. In contrast to Figure 3c,d, it represents precipitation forecasts in a 2.5x2.5 km2 grid box over Tel Aviv.
Verification
Before becoming operational, CO-IL-EPS and ICON-IL2.5 underwent extensive tuning focused on precipitation (Khain et al., 2021), near-surface parameters and cloudiness. Today these two systems, together with IFS HRES, show the highest skill among the models used at IMS. Figure 5 shows precipitation verification over Israel for the period from 1 March 2021 to 13 January 2022. The verification of selected CO-IL-EPS percentiles, the deterministic COSMO member (CO-IL2.5), ICON-IL2.5, and the IFS HRES was performed versus the IMS radar and rain gauges composite. Fractional Skill Score (FSS) was averaged over the spatial radiuses of 10, 20, 30 and 50 km. The thresholds of 10 mm/6 h and 20 mm/6 h are shown (panels a and b, respectively). While for low precipitation thresholds the deterministic models have an advantage over CO-IL-EPS percentiles, for 20 mm/6 h the 75% CO-IL-EPS percentile is the undisputed winner. For high precipitation rates, the deterministic forecasts of the convection-permitting models are often noisy and uncertain in space and time. By contrast, this result shows that the 75% percentile of CO-IL-EPS is significantly less noisy and hence is an important tool for forecasts of severe weather and flooding.
The use of ECMWF’s HPC facility for research
In addition to operational runs, ECMWF’s HPC facility is used by IMS for its research and development efforts. Most of this research is conducted in the framework of the COSMO consortium and results in continuous improvement of CO-IL-EPS and ICON-IL2.5. Below we present several interesting examples.
Using CAMS forecasts of aerosols in NWP
Operational weather forecasting is in constant conflict between the desire for accuracy and the goal of delivering results in a timely manner. Computational resources are always limited, so usually expensive fully-coupled aerosol models, such as those provided by the EU-funded Copernicus Atmosphere Monitoring Service (CAMS) run by ECMWF and ICON-ART, are replaced by a monthly aerosol climatology. On the other hand, radiation fluxes and cloud microphysics are critically affected by aerosol content. Solar radiation is scattered and absorbed by aerosols and can be significantly reduced in a highly polluted environment. Liquid and ice formation in clouds is also influenced by hydrophilic and hydrophobic aerosol content in the atmosphere. IMS has conducted research which proposes an intermediate solution to these issues by coupling predicted aerosol fields taken from the CAMS aerosol model with the ICON model. The idea is that at each time step ICON will use the CAMS predicted aerosols, interpolated in time and space, without the need to advect them in space. Therefore, the runtime stays almost identical to the aerosol climatology setup. In a first stage, the ECMWF radiation scheme (ecRad), which was recently introduced in ICON, was coupled with 3D predicted aerosol mixing ratios of CAMS. In the next step, we plan to implement the coupling of these aerosols with ice nucleation and water droplet activation schemes in the ICON model. The method was tested for the entire year 2020, using ECMWF’s HPC facility.
Figure 6 shows the global radiation verification using 17 automatic radiation stations over Israel during the year 2020. Twelve relatively polluted days, i.e. days when the averaged PM2.5 measurements over Israel were more than three times higher than normal, were chosen for verification. The verification was performed during clear-sky conditions, where cloud cover both in models and observations was below 30%. Figure 6 compares the mean diurnal cycle of the global radiation in observations, the IFS HRES, CO-IL2.5, ICON-IL2.5 and the new version ICON-IL2.5-CAMS with CAMS aerosol coupling. The first 24 hours of each simulation were verified. For each model, the mean bias and the root-mean-square error (RMSE) are presented.
One can see that in ICON-IL2.5-CAMS the global radiation on polluted days was substantially improved. The 47.6 Wm-2 bias in ICON-IL2.5 was essentially removed, and the RMSE was reduced from 91.5 Wm-2 to 68.9 Wm-2. Preliminary results show that during non-polluted days the global radiation improvement is less significant. We plan to verify that the implementation of CAMS aerosols coupling in ICON-IL2.5 improves the radiation forecast over the entire southeast-Europe domain. The operational implementation of CAMS aerosols coupling in ICON-IL2.5 is planned for the summer of 2022 and will hopefully yield better forecasts for SEE-MHEWS-A project countries.
Modelling of clouds in ICON
A precise representation of clouds is important for NWP and climate models, and the aerosol impact on cloud microphysics is one of the key issues in current uncertainties in cloud modelling. Latent heat released in clouds is the main source of energy for many cloud-related phenomena as well as for global circulation. For the accurate modelling of cloud evolution, an accurate representation of cloud microphysics and consequently its interaction with aerosols is needed. The Spectral Bin Microphysics (SBM) scheme (Khain & Pinsky, 2018) is a state-of-the-art microphysical parametrization that is included in several models worldwide, such as the Weather Research and Forecasting (WRF) model, the Goddard Cumulus Ensemble model, the System for Atmospheric Modelling (SAM), etc. Microphysical processes in SBM are described by solving non-linear differential equations based on first principles. At each model grid point and time step, the particle size distributions (PSDs) are defined on mass grids containing several tens of bins. For each bin, various microphysical processes are calculated and the PSD is evolved with time. The main disadvantage of SBM is that this method is computationally time-consuming. This is because the microphysical equations are written for each mass bin, which means that SBM contains several hundred fields to be calculated.
Using ECMWF’s HPC facility, the introduction and testing of SBM in ICON became possible. So far, we have introduced only the warm-phase part of SBM in ICON, and the introduction of the mixed-phase (i.e. ice processes) is planned.
Idealised 1 km resolution ICON simulations of cumulonimbus evolution were conducted under pristine and polluted conditions and compared with the two-moment (2M) microphysical scheme that is currently used in several advanced ICON implementations. Figure 7 shows a vertical cross-section, through the core of the clouds, of the updraft velocity after 30 minutes of cloud evolution. Panels (a) and (b) show 2M and SBM under polluted conditions, while (c) and (d) show them under pristine conditions. In contrast to 2M, SBM indicates strong sensitivity to aerosol concentration. Pristine conditions lead to a lower droplet concentration and higher supersaturation, faster drop growth and earlier beginning of intense collisions and raindrop formation. Compared to polluted conditions, the time during which droplets grow is small and hence less latent heat is released. This may explain the smaller development of the cloud and the smaller updraft velocities in the SBM simulation with pristine conditions.
Assimilation of OPERA radar network
The ICON model enables the assimilation of radar data in order to improve the forecast during the first few hours of the run. The radar data can be assimilated in the ICON-LAM via latent heat nudging (LHN) (Stephan et al., 2008), without a separate data assimilation system. During the model run the simulated precipitation is compared with the observed. In locations where there is a deficit in simulated precipitation, the vertical profiles of temperature and humidity are adjusted to account for the underestimation of latent heat release, which occurred as the precipitation formed. In locations where there is overestimation, the profiles are adjusted in the opposite direction.
As part of IMS activities for SEE-MHEWS-A, output from OPERA, EUMETNET’s Operational Programme for the Exchange of Weather Radar Information, was assimilated. Using ECMWF’s HPC facility during the second half of 2021, we could analyse the influence of the OPERA radar network assimilation on ICON-IL2.5 forecasts.
Figure 8 shows the verification of ICON-IL2.5 with and without OPERA assimilation during a severe flooding event over central Europe (12–15 July 2021). The verification was performed with OPERA data using the Equitable Threat Score for the threshold of 1 mm/h. In order to analyse the ability of the model to assimilate the OPERA data correctly, the LHN was performed during the entire simulation time. Obviously, this cannot be done in real time since the ‘future’ radar data is not available. Figure 8 nicely shows the benefit of OPERA data assimilation. The assimilation of the OPERA radar network in ICON-IL2.5 became operational in November 2021. Recently a new tuning of the LHN scheme, related to the bright band detection, was suggested by Germany’s National Meteorological Service (DWD). It is planned to be tested in ICON-IL2.5.
Selection of driving ensemble members
Due to limitations of computer resources, high-resolution regional ensembles usually consist of a relatively small number of ensemble members. This number is often smaller than the size of the driving global ensemble. The specific selection of a subgroup of driving global ensemble members may influence the quality of the driven regional ensemble. IMS is currently performing research in the framework of the COSMO consortium on how to select a subgroup of the driving global ensemble members to optimise the performance of the driven regional ensemble. For initial and boundary conditions, the study uses ECMWF’s ENS consisting of 50 members. The regional ensemble used is CO-IL-EPS. The conclusions are validated using two methods for model physics perturbations, namely the Stochastically Perturbed Parametrization Tendencies (SPPT) also used at ECMWF, and the parameter perturbations (PP). Among 50 ENS members, 20 are chosen in different ways to drive the CO-IL-EPS. The corresponding 20 CO-IL-EPS members are then perturbed via SPPT or PP and verified against observations. Finally, the verification scores are compared, resulting in the optimal algorithm for the selection of 20 ENS members.
This research includes re-forecasting of several configurations of CO-IL-EPS over historical periods, which requires a substantial amount of computer resources. ECMWF’s HPC facility is being used for this research, which otherwise could not be conducted.
Conclusions
The Israel Meteorological Service (IMS) has taken advantage of the Optional Programme for Co-operating States, which makes it possible to use ECMWF’s HPC resources. IMS uses these resources to run a computationally demanding high-resolution COSMO ensemble system over the Eastern Mediterranean and a high-resolution ICON model over the southeast Europe region. Both suites are running operationally under time-critical option 2. These suites provide weather forecasts not only for use in IMS but also for the benefit of other weather services in the region. ICON forecasts are used within the WMO SEE-MHEWS-A project in southeast Europe, while COSMO ensemble forecasts are used as part of a collaboration with nearby countries. Both forecast systems are visualised on various platforms, and their products are being provided to the public and academia free of charge.
In addition to running operational systems, ECMWF’s HPC facilities enable IMS to perform intensive research in the framework of the COSMO consortium. In addition to continuous improvement of the operational systems, these activities make it possible for IMS’s research department to flourish. Many new collaborations have become possible, and scientific papers are being published.
Acknowledgements
Implementation of the CO-IL-EPS suite at ECMWF was supported by the weather service of the Italian Emilia-Romagna region, Arpae-SIMC, which runs COSMO-LEPS as a time-critical option 2 application at ECMWF and kindly offered to share its configuration with IMS.
The CO-IL-EPS and ICON-IL2.5 suites run at ECMWF are both using the FieldExtra post-processing software developed at the Federal Office of Meteorology and Climatology of Switzerland, MeteoSwiss, kindly provided to IMS in the framework of the COSMO consortium.
We acknowledge colleagues from Germany’s National Meteorological Service, DWD, for constructive discussions and support in the implementation of CAMS predicted aerosols in ICON and the assimilation of OPERA via LHN.
IMS gratefully thanks ECMWF’s Council for the opportunity to use ECMWF’s HPC facilities and warmly thanks ECMWF staff for their extraordinary support.
Further reading
Khain, P., Y. Levi, H. Muskatel, A. Shtivelman, E. Vadislavsky & N. Stav, 2021: Effect of shallow convection parametrization on cloud resolving NWP forecasts over the Eastern Mediterranean. Atmos. Res. 247, article id. 105213.
Khain, A.P. & M. Pinsky, 2018: Physical Processes in Clouds and Cloud Modeling. Cambridge University Press (CUP): Cambridge, UK.
Stephan, K., S. Klink & C. Schraff, 2008: Assimilation of radar derived rain rates into the convective scale model COSMO-DE at DWD. Quarterly Journal of the Royal Meteorological Society, 134, 1315–1326.