Use and Verification of ECMWF products in Member and Co-operating States (2019)

Each summer Member and Co-operating States report on the application and verification of ECMWF’s forecast products for the previous year, whilst ECMWF also gathers feedback in other fora. This report summarises the feedback collected between summer 2018 and summer 2019.

Uptake and use of ECMWF products and data appears to have increased this year across the full range of lead times (short range to seasonal). Encouragingly, there also appears to be a greater focus on probabilistic forecasting (as served by the ECMWF ensemble (ENS)), although some forecasters still don’t use ENS output.

Positive comments were made about some newly introduced products, notably vertical profiles, the “integrated vapour transport” (IVT) diagnostic and “point rainfall”. The ECMWF web-based workstation tool ecCharts was also praised, though some complained about slow speed (a new significantly faster version of ecCharts has since been implemented). Useful ideas for new ECMWF products were also received, such as using ENS-based front-densities to select a “most representative” ensemble member.

Multi-faceted post-processing of ECMWF output continues in many countries, mainly for specific sites, where calibrating observations exist. However, limited area model (LAM) output is increasingly being adopted as a form of “truth” in this sphere.

When performing verification, many compare ECMWF’s high resolution runs (HRES) with LAM output, and so usually centre on shorter ranges (up to ~48-60 h). A very common finding, for almost every sensible weather parameter, was that HRES forecast biases have a diurnal cycle, and annual cycles are also often present. Overall, relative and absolute skill levels vary in many ways - e.g. with parameter, geography, weather situation, lead time and model. For low level relative humidity HRES reportedly out-perform LAMs, and through physical linkages this also benefits HRES handling of other weather features, such as low cloud and surface-based convection. Precipitation biases in HRES, for both small and large totals (compared to point observations), were again shown to exceed those of LAMs, but note that the experimental “point rainfall” product introduced in April 2019 helps address this model-resolution-related issue, at least for the ENS.

Large HRES forecast errors for 2m temperatures, in relatively extreme hot or cold situations, are a concern for some member states, notably in Scandinavia where very cold winter nights are demonstrably not nearly cold enough in HRES forecasts. ECMWF continues to work on this model issue. Meanwhile, using the insightful approach of conditional verification (a growth area at ECMWF), Finland show how 2m temperature errors in winter tend to be lowest when the skin temperature (ordinarily of a snow surface) is ~0⁰C. This is because energy exchanges then involve the latent heat of fusion more than temperature change.

Ensemble-related verification results were limited, but a multi-parameter multi-pressure level comparison by Germany between the ICON and ECMWF ensembles showed ECMWF to be consistently better over time at 48h leads in the northern hemisphere, except in the stratosphere, where ICON was much better. This IFS stratosphere problem will be addressed with a formulation change in cycle 47r1 in 2020.

The pivotal importance of correctly predicting severe events was re-iterated in many reports. For example, extreme Mediterranean cyclones, around 28 September and 29 October 2018, were mentioned by several countries. These cases also appear in ECMWF’s severe event catalogue.