Bootstrapping Robust Nonparametric Regression Estimators

• Department of Mathematics, KU Leuven, Leuven, Belgium [Stefan.VanAelst@kuleuven.be]

• Department of Statistics, University of British Columbia, Vancouver, Canada [matias@stat.ubc.ca]

Keywords: Bootstrap – Nonparametric regression – Robustness

1 Robust Nonparametric Regression based on Splines

Consider the non-parametric regression model

$$Y=f(𝐱)+\sigma \epsilon ,$$

with $𝐱\in {ℝ}^p$ a vector of explanatory variables and $f$ an unknown regression function. We assume that the function $f$ is smooth (e.g. differentiable). The errors $\epsilon$ are assumed to be independent and identically distributed for some symmetric distribution with center zero and scale 1 such that the parameter $\sigma >0$ is the error scale.

In this talk we consider spline based estimators, that is, the unknown function $f$ is assumed to be of the form

$$f(𝐱)=\sum _{j=1}^K{\gamma }_j{b}_j(𝐱),$$

with $b_1(𝐱),\mathrm{\dots },b_K(𝐱)$ a spline basis, e.g. truncated polynomials or B-splines while the parameter $K$ controls the size of the spline space. The essential problem thus becomes estimating the parameter vector $𝜸={({\gamma }_1,\mathrm{\dots },{\gamma }_K)}^{\top }$.

Based on a sample $({𝐱}_1,Y_1),\mathrm{\dots },({𝐱}_n,Y_n)$ we can then consider estimators ${\widehat{𝜸}}_n$ which are a minimizer of a penalized optimization problem of the form

$$L_n(𝜸;{𝐱}_1,Y_1),\mathrm{\dots },({𝐱}_n,Y_n))+\lambda 𝜸{}^{\top }𝐃𝜸.$$

For M-type estimators the loss function $L_n$ is of the form

$$L_n(𝜸;{𝐱}_1,Y_1),\mathrm{\dots },({𝐱}_n,Y_n))=\frac{1}{n}\sum {}_{i=1}{}^n\rho (\frac{y_i-{𝜸}^{\top }{𝐛}_i}{{\widehat{\sigma }}_n}),$$

with ${𝐛}_i={(b_1({𝐱}_i),\mathrm{\dots },b_K({𝐱}_i))}^{\top }$ while for S-estimators $L_n={\widehat{\sigma }}_n^2(𝜸)$ where ${\widehat{\sigma }}_n^2(𝜸)$ solves

$$\frac{1}{n}\sum _{i=1}^n\rho \left(\frac{y_i-{𝜸}^{\top }{𝐛}_i}{{\widehat{\sigma }}_n(𝜸)}\right)=\delta ,$$

for some $\delta \in (0,1)$. The smoothness of the estimator is regularized by the penalty term and the form of the matrix $𝐃$. Note that such estimators can be rewritten in a penalized weighted least squares form whichis convenient to develop an iteratively weighted least squares algorithm for their computation. Several robust spline-based estimators fit within this framework, such as the M-type spline estimators of He and Shi [1995], Kalogridis and Van Aelst [2021, 2024], Kalogridis [2022, 2023] as well as the spline based S-estimator of Tharmaratnam et al. [2010].

2 Bootstrapping Robust Splines

For most robust spline-based estimators asymptotic results are not sufficiently developed to derive robust inference based on asymptotic theory. Therefore, we explore the use of bootstrap methods to construct robust inference. Application of the bootstrap for nonparametric regression often does not yield desirable results because the bias due to smoothing is not taken into account. See Kauermann et al. [2009] and references therein. In this talk we will adapt bootstrap methods to develop robust inference such as confidence bands based on robust spline-based estimators and investigate their performance.