On the use of a Huber norm for observation quality control in the ECMWF 4D-Var

Title
On the use of a Huber norm for observation quality control in the ECMWF 4D-Var
Technical memorandum
Date Published
11/2014
Secondary Title
ECMWF Technical Memoranda
Number
744
Author
C. Tavolato
L. Isaksen
Publisher
ECMWF
Abstract This paper describes a number of important aspects that need to be considered when designing and implementing an observation quality control scheme in an NWP data assimilation system. It is shown how careful evaluation of innovation statistics provides valuable knowledge about the observation errors and help in the selection of a suitable observation error model. The focus of the paper is on the statistical specification of the typical fat tails of the innovation distributions. In observation error specifications, like the one used previously at ECMWF (European Centre for Medium-Range Weather Forecasts), it is common to assume outliers to represent gross errors that are independent of the atmospheric state. The investigations in this paper show that this is not a good assumption for almost all observing systems used in today's data assimilation systems. It is found that a Huber norm distribution is a very suitable distribution to describe most innovation statistics, after discarding systematically erroneous observations. The Huber norm is a robust method, making it safer to include outlier observations in the analysis step. Therefore the background quality control can safely be relaxed. The Huber norm has been implemented in the ECMWF assimilation system for in-situ observations. The design, implementation and results from this implementation are described in this paper. The general impact of using the Huber norm distribution is positive, compared to the previously used variational quality control method that gave virtually no weight to outliers. Case studies show how the method improves the use of observations, especially for intense cyclones and other extreme events. It is also discussed how the Huber norm distribution can be used to identify systematic problems with observing systems.
URL https://www.ecmwf.int/en/elibrary/76606-use-huber-norm-observation-quality-control-ecmwf-4d-var
DOI 10.21957/58h6u3tru
Download citation