Robust and Conjugate Gaussian Process Regression

Abstract

To enable closed form conditioning, a common assumption in Gaussian process (GP) regression is independent and identically distributed Gaussian observation noise. This strong and simplistic assumption is often violated in practice, which leads to unreliable inferences and uncertainty quantification. Unfortunately, existing methods for robustifying GPs break closed-form conditioning, which makes them less attractive to practitioners and significantly more computationally expensive. In this paper, we demonstrate how to perform provably robust and conjugate Gaussian process (RCGP) regression at virtually no additional cost using generalised Bayesian inference. RCGP is particularly versatile as it enables exact conjugate closed form updates in all settings where standard GPs admit them. To demonstrate its strong empirical performance, we deploy RCGP for problems ranging from Bayesian optimisation to sparse variational Gaussian processes. The code and datasets to reproduce all experiments are available at
https://github.com/maltamiranomontero/RCGP.

• Department of Statistical Science, University College London, London, UK [matias.altamirano.22@ucl.ac.uk]

• Department of Statistical Science, University College London, London, UK [f.briol@ucl.ac.uk]

• Department of Statistical Science, University College London, London, UK [j.knoblauch@ucl.ac.uk]

Keywords: Gaussian Processes – Robustness – Generalised Bayes

1 Abstract

GPs [Rasmussen and Williams, 2006] are one of the most widely used methods for Bayesian inference on latent functions, especially when uncertainty is required. They have numerous appealing properties, including that the prior is relatively interpretable and can be elicited through a choice of mean and covariance functions, as well as the fact that they have closed form posteriors under Gaussian likelihoods. Their convergence is also well understood, even under prior misspecification [Wynne et al., 2021]. Thanks to these advantages, GPs have found applications in diverse problems including single- and multi-output regression [Bonilla et al., 2007, Moreno-Muñoz et al., 2018], emulation of expensive simulators [Santner et al., 2018], Bayesian optimisation [Shahriari et al., 2015, Garnett, 2021] and Bayesian deep learning [Damianou and Lawrence, 2013, Salimbeni et al., 2019, Dutordoir et al., 2020]. Their use is enabled by a plethora of packages including GPflow [Matthews et al., 2017] GPyTorch [Gardner et al., 2018], BoTorch [Balandat et al., 2020], ProbNum [Wenger et al., 2021] and emukit [Paleyes et al., 2023].

By far the most common use of GPs is in regression. Here, the observations correspond to noisy realisations from an unknown latent function that is assumed to be drawn from a GP prior. To obtain a conjugate GP posterior distribution on the latent function, the observation noise is usually assumed to be Gaussian. While assuming Gaussian observation noise makes the posterior tractable, it also makes inferences non-robust. In particular, Gaussian noise makes GPs highly susceptible to extreme values, heterogeneities, and outliers. In many real-world applications and data sets, the presence of outliers is almost inevitable. They can occur for a variety of different reasons, including due to faulty measurements, broken sensors, extreme weather events, stock market sell-offs, or genetic mutations.

Existing Work

The lack of robustness in GPs is a well-known fundamental challenge for their widespread application, and a number of methods have been proposed to address this. Broadly, these fall into two categories. The first replaces the Gaussian measurement error with more heavy-tailed error distributions such as Student’s $t$ [Jylänki et al., 2011, Ranjan et al., 2016], Laplace [Kuss, 2006], Huber densities [Algikar and Mili, 2023], data-dependent noise [Goldberg et al., 1997], or mixture distributions [Naish-Guzman and Holden, 2007, Stegle et al., 2008, Daemi et al., 2019, Lu et al., 2023]. Heavy tails allow these distributions to better accommodate outliers, rendering them more robust to corruptions. Their main limitation lies in their computational cost, as abandoning Gaussian noise nullifies one key advantage of GPs: conjugacy. As a consequence, these techniques rely on approximations via variational methods or Markov chain Monte Carlo. This decreases their accuracy while increasing computational costs. The second set of approaches consists in removing outlying observations before using a standard GP with Gaussian noise [Li et al., 2021, Park et al., 2022, Andrade and Takeda, 2023]. While such approaches use conjugacy, it can be challenging to detect outliers in irregularly spaced data or higher dimensions. Outlier detection also tends to be computationally costly, and often requires estimating large numbers of parameters.

In this paper, we propose a new and third way to achieve robustness that uses generalised Bayesian inference [see e.g. Bissiri et al., 2016, Jewson et al., 2018, Knoblauch et al., 2022]. In doing so, we significantly improve upon an earlier attempt in this direction due to Knoblauch [2019] that was applicable only for variational deep GPs, lacked closed form solutions, and was based on hyperparameters that were difficult to choose. In line with the ideas of generalised Bayesian methods, we will not modify the Gaussian noise model. Instead, we change how information is assimilated, and leverage robust loss functions instead of robust error models.

Contributions

This paper proposes a novel robust and conjugate Gaussian process (RCGP) inspired by a generalised Bayesian inference scheme proposed in Altamirano et al. [2023]. The posteriors rely on a generalised form of score matching [Hyvärinen, 2006, Barp et al., 2019], which effectively down-weights outlying observations. The resulting inference resolves the trade-off between robustness and computation inherent in existing methods: it is robust in the sense of Huber [1981] while retaining closed form solutions for both its posterior and posterior predictive. Additionally—and unlike other robust GPs—RCGPs can easily be plugged into various GP techniques such as sparse variational GPs [Titsias, 2009, Hensman et al., 2013], deep GPs [Damianou and Lawrence, 2013], multi-output GPs [Bonilla et al., 2007], and Bayesian optimisation [Shahriari et al., 2015]. Finally, even in settings where robustness is not required, our experiments show that RCGPs performs as well as standard GPs—raising the possibility that RCGPs may become a preferred default choice over GPs in the future.