Bayesian Variable Selection in Generalized Linear Models

L. Filippozzi${}^{a,b}$, I. Urteaga${}^{c,d}$ and C. Agostinelli${}^a$

${}^a$University of Trento, ${}^b$Fondazione Bruno Kessler (FBK), ${}^c$Basque Center for Applied Mathematics (BCAM), ${}^d$Ikerbasque — Basque Foundation for Science

Variable or covariate selection is a crucial step aimed at identifying the most relevant predictors for explaining the response variable.

In this work, we propose BayesVS-GLM, a novel Bayesian covariate selection method for Generalized Linear Models (GLMs). BayesVS-GLM is based on a fully conjugate Bayesian hierarchical model that comes with theoretical posterior consistency guarantees. Specifically, we extend the standard GLM framework by introducing a binary vector $z$ to indicate which covariates are included in the generalized linear predictor. The regression coefficients $\beta$ are modeled conditionally on $z$, using conjugate priors for GLMs [4].

Although related method exists ([1], [2], [3]), our method is, to the best of our knowledge, the first that provides a unified framework in which: $(i)$ the formulation of the hierarchical GLM is fully conjugate; $(ii)$ the GLM likelihood is explicitly dependent on indicator variables $z$, enabling a regressor selection based uniquely on observed data; and $(iii)$ the posterior asymptotical accuracy of $z$ and posterior consistency of the regression coefficients $\beta$ are guaranteed. For posterior inference, we present an efficient Gibbs sampling algorithm, based on a fully conjugate Bayesian hierarchical model.

The BayesVS-GLM formulation is applicable to any distribution within the exponential family, and unifies a range of existing Bayesian variable selection perspectives within a single coherent hierarchical framework.

Keywords: Bayesian Variable Selection, Generalized Linear Models, Posterior model selection.