Impact of GRIB compression on weather forecast data and data-handling applications

As part of its scientific ambitions, ECMWF plans to increase the resolution of its ensemble forecasts from 18 to 9 kilometres for IFS cycle Cy48r1. This single upgrade will be responsible for a four times increase in the volume of data produced (assuming all other factors constant), from roughly 30 TiB per cycle to 120 TiB. In aggregate, ECMWF’s forecasts will produce 480 TiB per day.

Data reduction strategies provide one approach to mitigating the impact of this expected growth. The bulk of ECMWF’s forecast output data is generated in GRIB format. As such, this report re-examines the GRIB packing methods available and investigates the use of compression algorithms to reduce the volume of data produced.

This report focuses on the grid_ccsds1 packing type supported in the WMO GRIB standard and grid_second_order developed by ECMWF, both implemented by the ecCodes library. We present estimates for the impact of these compression methods on current and future data volumes. We also look at the first-order impact on other components of the operational system, including the IFS, MARS and the FDB.

We demonstrate that the packing type grid_ccsds is particularly suitable for forecast data. Adopting this packing type for data that is already encoded in GRIB2 format would result in a 32% reduction in the volume of data produced. Further, a reduction of 42% is possible given the completion of the progressive migration from GRIB1 to GRIB2.

Although it appears as a one-shot solution, we also conclude that grid_ccsds would be a good investment, as the method also scales well and further improves its compression efficiency with future resolution increases.