ECMWF maintains a comprehensive range of verification statistics to evaluate the accuracy of its forecasts. Each year, a summary of verification results is presented to ECMWF’s Technical Advisory Committee (TAC). Their views about last year’s performance of the operational forecasting system are given in the box.
The overall performance of the operational forecasts is summarised using a set of headline scores endorsed by the TAC, which highlight different aspects of forecast skill. Upper-air performance of the high-resolution forecast (HRES) in the northern hemisphere extratropics is monitored through the anomaly correlation of 500 hPa geopotential height. Upgrades to the Integrated Forecasting System (IFS) in May 2015 (Cycle 41r1) and in March 2016 (Cycle 41r2) have led to a substantial increase in medium-range forecast skill. The forecast range at which the anomaly correlation drops below 80% now exceeds the highest levels reached previously by 0.2 days. Note that part of the increase seen during 2015–2016 is due to a concurrent increase in atmospheric predictability in this period. However, comparison with forecasts from other global centres shows that ECMWF has consolidated its lead both for the HRES and for ensemble forecasts (ENS). Similar improvements are found for the southern hemisphere and for other upper-air skill metrics. The 500 hPa height root-mean-square error in both hemispheres has reached its lowest level so far. As shown in the second figure, ENS upper-air skill, as measured by the continuous ranked probability skill score of 850 hPa temperature, has now increased beyond the levels seen during the anomalously predictable period in 2010. Upper-air skill in the tropics has slightly improved when verified against observations but not against analyses due to increased analysis activity.
Surface skill has increased for 2 m temperature and humidity, 10 m wind speed, precipitation,
and total cloud cover, with reductions in errors on the order of 1–2% in the early medium range. However, there are still issues with seasonally and diurnally varying regional biases in 2 m temperature which need to be resolved. Improvements in the forecast of total cloud cover are seen both in the direct verification against SYNOP and in the verification of solar radiation fluxes against satellite observations.
Forecasts of tropical cyclones have been comparable in skill to the previous year in terms of position errors both for the HRES and ENS, while speed errors have been further reduced. Tropical cyclone intensity has improved in terms of bias, but mean absolute errors have been larger than in the previous year.
Assessment of ECMWF’s Technical Advisory Committee, 13–14 October 2016
With regard to its overall view of the ECMWF operational forecasting system, the Committee:
- in view of the overall performance level of its weather forecast system, congratulated ECMWF on the improved forecast skill as revealed by the headline scores;
- noted that this improvement was very clear relative to ERA-Interim benchmarking, but that some of the other centres have also made marked improvements over this period. However ECMWF still maintains its lead in the medium range;
- welcomed ECMWF’s continued focus on weather parameters that have an impact on users, such as 2 m temperature, rainfall, and wind; was pleased to note in particular the improvement in forecasting cloud cover, although several member states still noted occasional problems with low-level clouds;
- congratulated ECMWF on the clear impact of model changes on forecast performance, most notably 41r2 and associated gains in predictability (3 to 6 h gain according to the first two headline scores); noted that other parameters were also positively affected by this model and resolution change; suggested that e-suite headline score time series be plotted alongside o-suite scores to help document these gains;
- noted that improvements in the ENS resolution in the 10–15 day period have significantly improved the forecasting capability at that time range;
- congratulated ECMWF on the good communication with Member States (MSs) prior to the 41r2 implementation, and especially the provision of e-suite products on EC Charts in real time;
- noted that MSs renewed their appreciation for the provision of new parameters targeting high-impact weather such as precipitation type, visibility, and the EFI for CAPE shear; encouraged ECMWF to develop further these diagnostics such as lightning density in consultation with MSs and provide verification results accordingly;
- welcomed the continued development of EC Charts facilities, and encouraged ECMWF working with MSs in its efforts to improve the performance of the service;
- welcomed the ongoing development of facilities offered to MSs for reporting model forecast problems or document severe weather case studies; encouraged ECMWF to include in its report to TAC
a summary of the actions that have been taken; - noted that forecasters in MSs occasionally report negatively on ENS unphysical behaviour of some members, welcomed first results from research being done on new stochastic physics (SPP) that should improve on that issue.
Wave forecast skill has further increased with respect to both wave height and peak period, allowing ECMWF to maintain its lead compared to other global wave forecasting systems.
The presence of a strong El Niño enhanced predictability on the seasonal timescale. Positive anomalies of 2 m temperatures over North America and northern Eurasia during the northern-hemisphere winter 2015–16 were captured well by the seasonal forecast. The westernmost parts of Europe were influenced by a persistent cold anomaly over the North Atlantic, which was also predicted by the model. During the northern-hemisphere summer of 2016, sea-surface temperatures in the eastern tropical Pacific had reverted back from the previous El Niño to slightly negative anomalies. In terms of large-scale anomaly patterns, the seasonal 2 m temperature forecast captured the major features. However compared to the period December 2015 to February 2016, there was less skill on smaller, sub-continental scales.
Verification results provided by Member and Co-operating States focused on surface parameters in the shorter ranges, where they compared IFS forecasts with limited-area model (LAM) output. This feedback is greatly appreciated as it helps to improve future model versions. Among the issues reported were large 2 m temperature errors in stable situations, and a tendency to underestimate high wind speeds over mountainous areas. It was also noted by some states that over Europe there was generally too much convective activity over mountainous areas, but too little over some lowland areas. Meanwhile we also received reports of sporadic problems with sea-surface temperature and sea-ice cover within the IFS.
The complete set of annual results is available in ECMWF Tech. Memos No. 792 on the ‘Evaluation of ECMWF forecasts, including the 2016 resolution upgrade’ and No. 797 on the ‘Use and Verification of ECMWF products in Member and Co-operating States (2016)’. Both are downloadable from http://www.ecmwf.int/en/research/publications. These documents present recent verification statistics and evaluations of ECMWF forecasts (including weather, waves and severe weather events), whilst No. 792 also includes information about changes to the data assimilation/forecasting and post-processing systems. The performance of the monthly and seasonal forecasting systems is also assessed.
The following are other sources of information about verification and forecasting system changes:
- Verification pages on the ECMWF web server are regularly updated. They are accessible at http://www.ecmwf.int/en/forecasts/charts.
- Interactive plots showing inter-comparisons of global model forecast skill can be found on the WMO Lead Centre for Deterministic Forecast Verification (WMO-LCDNV) web page at http://apps.ecmwf.int/wmolcdnv/.
- All IFS forecasting system cycle changes since 1985 are described at http://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model.