An Optimal Linear Transformation for Data Assimilation

Linear transformations are widely used in data assimilation for covariance modeling, for reducing dimensionality (such as averaging dense observations to form "superobs"), and for managing sampling error in ensemble data assimilation. Here we describe a linear transformation that is optimal in the sense that, in the transformed space, the state variables and observations have uncorrelated errors, and a diagonal gain matrix in the update step. We conjecture, and provide numerical evidence, that the transformation is the best possible to precede covariance localization in an ensemble Kalman filter. A central feature of this transformation in the update step are scalars, which we term canonical observation operators (COOs), that relate pairs of transformed observations and state variables and rank-order those pairs by their influence in the update. We show for an idealized problem that sample-based estimates of the COOs, in conjunction with covariance localization for the sample covariance, can approximate well the true values, but a practical implementation of the transformation for high-dimensional applications remains a subject for future research. The COOs also completely describe important properties of the update step, such as observation-state mutual information, signal-to-noise and degrees of freedom for signal, and so give new insights, including relations among reduced-rank approximations to variational schemes, particle-filter weight degeneracy, and the local ensemble transform Kalman filter.