More than 100 users of ECMWF forecasts met at the Centre from 6 to 9 June 2016 to discuss new approaches to the evaluation of weather predictions. Using ECMWF’s Forecasts (UEF) is an annual event which attracts participants from ECMWF’s Member and Co-operating States, primarily from national meteorological services (NMSs), as well as from commercial weather companies and academia across the world. The idea of the meeting is to provide a forum where ECMWF data users can network and learn from each other. ECMWF is also keen to receive feedback from all its users on how they use its products and what they think of them.
This year’s topic, ‘Shaping future approaches to evaluating high-impact weather forecasts’, focused on verification aspects, with an emphasis on measuring the quality of forecasts in a manner that is relevant to users and for different time ranges.
Three key themes
High-impact weather is of particular interest to forecast users, and its prediction with a seamless ensemble analysis/forecast system is at the heart of ECMWF’s new ten-year Strategy. Weather can have a big impact by virtue of its extreme amplitude (e.g. tropical cyclones), its duration (e.g. heat waves) or its societal impacts on specific user communities. All these aspects present a particular challenge for forecasting systems and demand considerable development of diagnostic and verification capabilities. ECMWF is working with its Member and Co-operating States to review and develop its current set of metrics, including for the evaluation of forecast skill for high-impact weather at lead times of two to four weeks. UEF 2016 offered an opportunity to hear from ECMWF data users what verification metrics they use and how ECMWF data can provide added value.
UEF 2016 teased out ideas in three thematic areas:
- Assessing long-term improvements - Forecasting high-impact weather is central to the work of NMSs. The knowledge of the quality of model data and relevant ensemble-based products is essential to deliver high-value information.
- User-oriented verification - Measures of model performance need to be relevant for user communities in order to identify the extent to which the forecast is useful as a basis for decision-making in weather-sensitive activities.
- Seamless verification across different timescales - NWP models provide forecasts for various timescales. From the verification viewpoint, this poses a challenge as the skill at different timescales will need to be compared even though verification questions may be different.
Session highlights
The meeting offered a mix of activities aimed at giving information, networking and providing feedback. In a session on ‘Assessing the long-term improvement of high-impact weather forecasts’, Petra Friederichs from the University of Bonn, Germany, reminded the audience that “there is no unique definition of extremes, but extremes are generally associated with rare or severe events. Mathematically, extremes are defined as block maxima or exceedances above high thresholds.” She added that appropriate metrics are the first step towards improving the prediction of extremes using post-processing.
ECMWF’s model climate was highlighted in a number of contributions as an invaluable dataset that can be used to define extreme events without using location-specific thresholds. Speaking in a session on ‘User-oriented verification’, Cristina Primo Ramos from the German national weather service (DWD) stressed the importance of using proper scores and appropriate verification methods to correctly answer the different questions that users may have about forecast quality. Therefore, verification needs to be done by experts for optimal results and to reach appropriate conclusions.
Nicole Girardot (Météo-France) introduced the audience to new scoring methods for 'sensible weather' (weather we can observe using our senses, such as rain, wind, sun and cloud cover). She explained that those methods aim to take into account the user perception of the weather and of forecast error. Grant Elliott from Woodside Energy Ltd mentioned that from the private sector viewpoint there are a number of challenges: “For example, procedures are designed to minimise the risks to which the workforce is exposed. Therefore we need robust verification methods for rare events that allow us to demonstrate the compliance of procedures.”
Talks given at a session on ‘Seamless verification across timescales’ explored the ability of forecasts to predict heat waves, cold spells, droughts and precipitation anomalies. Tressa Fowler from the US National Center for Atmospheric Research explained how verification methods must follow numerical models into seamless temporal space. She mentioned that users require a fair comparison of forecasts at all timescales although their questions may not be identical. Matthew Wheeler from Australia’s Bureau of Meteorology showed the audience an innovative way to verify forecasts across timescales by averaging them over intervals comparable to their lead times. He demonstrated that, when the prediction skills of these averages are plotted as a function of the logarithmic scale of time, a fair comparison of skill can be made across a large range of scales.
User-oriented visualisation
Visualising the uncertainty of a forecast is of primary importance. ECMWF has developed the Ensemble Meteogram (ENS Meteogram), which is primarily a probabilistic representation, for a given location, of forecasts from an ensemble forecasting system. ECMWF is seeking ways to improve this very popular product. To this end, the UEF 2016 event saw the launch of a competition (‘The Challenge’) for innovative ideas. If you are interested in participating in The Challenge or you have a colleague who may want to submit a project, please visit: http://www.ecmwf.int/en/learning/workshops/challenge-uef2016
I would like to thank ECMWF staff who helped during the meeting in various ways. I would also like to extend my thanks to all the participants, who contributed actively to the activities and gave us constructive feedback.
