Robust dimension reduction

• Department of Mathematics, Universidad del Litoral, Santa Fe, Argentina [anbergesio@gmail.com]

• Department of Mathematics, Universidad de Buenos Aires, Buenos Aires, Argentina [meszre@dm.uba.ar ]

• Department of Mathematics, Universidad de Buenos Aires, Buenos Aires, Argentina and CONICET, Argentina [victoryohai@gmail.com]

Keywords: Sufficient reduction, Principal Fitted Components and Reduced Rank regressions

Abstract

Non-parametric regression is a very flexible procedure to establish the relationship between a response $y$ variable and several covariables $𝐱=(x_1,\mathrm{\dots },x_p)$. Non-parametric regression techniques deal with fitting a model of the form $y=g(x_1,\mathrm{\dots },x_p)+\delta$ without assuming any predetermined parametric model for $g$. However the number of observations required for non-parametric regression increases exponentially with the number of covariables, and this number may be larger of what is generally available. This is usually known as the dimensionality curse problem.

Cook [2007] defined the concept of sufficient reduction statistics to overcome this problem. Suppose that $y$ is the response and $𝐱\in R^p$ is the vector of covariables. Let $𝐳=g(𝐱),$ $g:R^p\to R^d.$ Then $𝐳$ is a sufficient reduction statistics of size for $y$ if, $𝒟(y|𝐳)=𝒟(y|𝐱)$ $𝒟(y|𝐳).$ Therefore the vector $𝐳$ contains all the information that $𝐳$ has about $y$. This imply that $𝐱$ can be replaced by $𝐳$ in the non-parametric regression. To obtain a sufficient reduction statistics, Cook [2007] introduced the principal fitted components (PFC) model. The PFC model assumes that

$$𝐱={\mathrm{\Gamma }}_0{B}_0𝐯(𝐲)+{\mathrm{\Delta }}_0^{1/2}𝐮$$

where ${\mathrm{\Gamma }}_0$ is a $p\times d$ matrix, , $B_0$ is a $d\times r$ matrix where $r\ge d$ and $v:R\to R^r,$ ${\mathrm{\Gamma }}_0^T{\mathrm{\Delta }}^{-1}{\mathrm{\Gamma }}_0=I_d$, $E({𝐮}^T𝐮)=I_p$. Then, $E(𝐱)\in {𝒮}_d$, where ${𝒮}_d$ is the subspace of dimension $d$ generated by the columns of ${\mathrm{\Gamma }}_0$. For example $v(y)$ $=(1,y,y^2,\mathrm{\dots },y^{d-1}$ ). Calling $F_0={\mathrm{\Gamma }}_0{B}_0$, we can also write $𝐱=F_0𝐯(y)+𝜺$, where $\mathrm{rank}(F_0)=d$ and $E({𝜺}^T𝜺)={\mathrm{\Delta }}_0$. This imply that $𝐱$ follows a reduced rank multiple regression model of rank $d$ with regressor equal to $v(𝐲)$. Therefore the proposed estimators can also be used to estimate these models.

Cook [2007] and Cook and Forzani [2008] proved that under the PFC model where $𝜺$ is ${\mathrm{N}}_p(\mathrm{𝟎},{\mathrm{\Delta }}_0)$, a sufficient reduction statistics of dimension $d$ for $y$ is given by $𝐳={\mathrm{\Gamma }}_0^T{\mathrm{\Delta }}_0^{-1}𝐱$.

Let $({𝐱}_{1,},y_1),\mathrm{\dots },({𝐱}_n,y_n)$ be a random sample of the PFC model and consider the Mahalanobis distances between the predicted and observed values of ${𝐱}_i$

$$M{D}_i{(F,\mathrm{\Delta })}^2={({𝐱}_i-F𝐯(y_i))}^T{\mathrm{\Delta }}^{-1}({𝐱}_i-F𝐯(y_i))$$

Then the maximum likelihood estimator ${\mathrm{\Theta }}_0=({\mathrm{\Gamma }}_0,B_0,{\mathrm{\Delta }}_0)$ is given by

$$\widehat{\mathrm{\Theta }}=\arg \underset{\mathrm{\Theta }}{\min }det(\mathrm{\Delta })$$

subject to

$$\frac{1}{n}\sum _{i=1}^{n}M{D}_i{(F,\mathrm{\Delta })}^2=p,\mathrm{rank}(F)=d$$

As most of the ML estimators that assume normal errors, the ML estimator of the PFC model is very sensitive to a few outliers, even a single outlier may take the ML estimator beyond any limit. To overcome this problem a class of robust estimators for the PFC model based on a $\tau$-scale estimator, see Yohai and Zamar [1988], were proposed. The $\tau$-scale estimators are highly robust and may have breakdown point 0.5. Let $s_{\tau }$ be a $\tau -$scale estimator, and let $\kappa$ its asymptotic value for a sample of the ${\chi }^2$ distribution with $p$ degrees of freedom. Then a class of robust estimators for the PFC model (called $\tau$-PFC estimators) is given by

$${\widehat{\mathrm{\Theta }}}_{\tau }=\arg \underset{\mathrm{\Theta }}{\min }det(\mathrm{\Delta })$$

subject to

$$s_{\tau }^2(M{D}_1(𝚯),\mathrm{\dots },M{D}_n(𝚯))=\kappa ,\mathrm{rank}(F)=d.$$

These estimators are strongly consistent under general conditions. A computational procedure based on iterative weighted maximum likelihood estimators, where the weights penalize outlier observations is given. A procedure based on cross validation to determine $d,$ that is the dimension of the sufficient reduction statistics, is also proposed. A Monte Carlo study shows that the $\tau$-PFC estimators are simultaneously highly efficient and robust. We also give an example with real data where the non-parametric model using the reduction sufficient statistics obtained with the proposed robust sufficient reduction statistics works better than using the one obtained with the maximum likelihood estimator.