Cellwise-robust grouped GMM for smoothed covariance estimation

• Institute of Statistics and Mathematical Methods in Economics, TU Wien, Vienna, Austria [patricia.puchhammer@tuwien.ac.at]

• Department of Quantitative Economics, Maastricht University, Maastricht, The Netherlands [i.wilms@maastrichtuniversity.nl]

• Institute of Statistics and Mathematical Methods in Economics, TU Wien, Vienna, Austria [peter.filzmoser@tuwien.ac.at]

Keywords: Multi-Group/Multi-Source Data – Gaussian Mixture Models – Cellwise Outliers

Many robust covariance estimation techniques assume one coherent data set while accounting for rowwise outliers or outlying cells. Yet, many applications involve multiple related data sets from different sources or non-coherent data sets whose observations can naturally be divided into multiple more coherent groups with shared global patterns. One example are spatial data where groups can be based on, e.g., spatial proximity. Even though existing methods for covariance estimation can be applied either globally or for each group individually, the former approach could result in misleading conclusions on the global level – since the data points are not coherent – and no conclusions for individual groups can be drawn. The latter approach can provide results representative for each group, but ignores any additional information from other groups or sources that could be leveraged to take shared patterns into account. Thus, grouped data should be analyzed jointly, and estimated covariance matrices should reflect shared patterns.

To this end we choose to model the data based on Gaussian Mixture Models (GMM). Let ${𝑿}_1,{𝑿}_2,\mathrm{\dots },{𝑿}_N$ be data sets from $N$ groups consisting of ${𝑿}_g={({({𝒙}_{g,1})}^{\prime },\mathrm{\dots },{({𝒙}_{g,n_g})}^{\prime })}^{\prime }\in {ℝ}^{n_g\times p}$ independent observations, for $g=1,\mathrm{\dots },N$, of the same $p$ variables. Assume that each observation ${𝒙}_{g,i}$ from group $g$ originates from a Gaussian mixture

$${𝒙}_{g,i}\sim 𝒩({𝝁}_k,{𝚺}_k)\text{ with probability }{\pi }_{g,k}\ge 0$$

for $k=1,\mathrm{\dots },N$, and where $\phi ({𝒙}_{g,i};{𝝁}_k,{𝚺}_k)$ is defined as the multivariate normal density for ${𝒙}_{g,i}$. Based on the assumption that each individual group is coherent, assume ${\pi }_{g,g}\ge 0.5$, thus each group has a main distribution but can share patterns by allowing the transition to other distributions. The resulting covariance matrix of a group $g$ is the one from distribution $g$ smoothed over the covariance matrices from the distributions with smoothing weights ${\pi }_{g,k}$,

$$\mathrm{Cov}[{𝒙}_g]=\sum _{k=1}^N{\pi }_{g,k}{𝚺}_k+\sum _{k=1}^N{\pi }_{g,k}({𝝁}_k-𝔼[{𝒙}_g]){({𝝁}_k-𝔼[{𝒙}_g])}^{\prime },$$

where $𝔼[{𝒙}_g]={\sum }_{k=1}^N{\pi }_{g,k}{𝝁}_k$ is the expected value of an observation ${𝒙}_g$ from group $g$.

Similar to the approach of Raymaekers and Rousseeuw [2023] and Zaccaria et al. [2024], cellwise robustness is included by using a missing value approach in the likelihood to set outlying cells to missings, indicated by the matrices $𝑾={\left({𝑾}_g\right)}_{g=1}^N$ consisting of binary vectors ${𝒘}_{g,i},i=1,\mathrm{\dots },n_g$. The objective function to minimize is

$$-2\sum _{g=1}^N\sum _{i=1}^{n_g}\left[\ln \left(\sum _{k=1}^N{\pi }_{g,k}\phi ({𝒙}_{g,i}^{({𝒘}_{g,i})};{𝝁}_k^{({𝒘}_{g,i})},{𝚺}_{reg,k}^{({𝒘}_{g,i})})\right)+\sum _{j=1}^p{q}_{g,ij}(1-w_{g,ij})\right]$$

subject to

${𝚺}_{reg,k}=(1-{\rho }_k){𝚺}_k+{\rho }_k{𝑻}_k$

${||{𝑾}_{g,.j}||}_0\ge h_g$ $\forall j=1,\mathrm{\dots },p,\forall g=1,\mathrm{\dots },N$

${\displaystyle \sum _{k=1}^N}{\pi }_{g,k}=1$ $\forall g=1,\mathrm{\dots },N$

${\pi }_{g,g}\ge \alpha \ge 0.5,$

where $q_{g,ij}$ are constants, ${.}^{(𝒘)}$ denotes the subset of observed variables, ${\rho }_k$ and ${𝑻}_k$ are regularizing factors and matrices for high-dimensional settings, and $h_g$ is the minimal number of cells observed per variable and group $g$.

Theoretical results include the breakdown point for an adaptation of the ideal setting described in Hennig [2004], Cuesta-Albertos et al. [2008] to the paradigm of cellwise outliers for multi-group data. The proposed methodology implemented via an EM-type algorithm provides good simulation results compared to the cellMCD [Raymaekers and Rousseeuw, 2023] and other methods in settings of low as well as high dimensional grouped data, and its potential is illustrated on Austrian weather data.

Acknowledgements: Co-funded by the European Union (SEMACRET, Grant Agreement no. 101057741) and UKRI (UK Research and Innovation). Ines Wilms is supported by a grant from the Dutch Research Council (NWO), research program Vidi under the grant number VI.Vidi.211.032.