Quantile Integrated Depth and Applications

• Department of Public Health and Health Sciences, Northeastern University, Boston, USA [s.lopez-pintado@northeastern.edu]

• Herbert Wertheim School of Public Health and Human Longevity Science, University of California San Diego, San Diego, USA [maluo@ucsd.edu]

• Department of Probability and Mathematical Statistics, Charles University, Praque, Czech Republic [nagy@karlin.mff.cuni.cz]

• Department of Biostatistics, Columbia University, New York, USA [todd.ogden@columbia.edu]

1 Motivation and Summary

Functional data analysis involves data for which the basic unit of observation is a function or image. The development of robust exploratory tools and inferential methods is very much needed since few assumptions can be made about the generating process. Data depth, a well-known non-parametric tool for analyzing functional data, provides a rigorous method for ranking a sample of curves from the center outwards, allowing for robust inference and outlier detection. Several notions of depth for functional data have been introduced in the last few decades (e.g. Fraiman and Muniz, 2001, Lopez-Pintado and Romo, 2009, Narissty and Nair, 2016, among others). Here, we develop a new family of depths, termed quantile integrated depth ($QID$), that are based on integrating up to the $K$-th quantile of the univariate depths. We show that this new family of depths satisfies all the desirable properties established in Gijbels and Nagy (2017), including a type of invariance, maximality at the center, and monotonicity with respect to the deepest point. $QID$ generalizes the well-known integral and infimal depths and solves some of the drawbacks of these types of depths. In particular, since functional data are commonly observed with noise, we explore the effect of noise on different notions of depth. A visualization tool called the Spearman agreement depth (SAD) plot is introduced. The SAD plot compares depth measures of corresponding functional observations between two versions of a dataset, an original version of the data and a version of the data with additive noise. Compared to alternatives, the proposed $QID$ is shown to be robust and performs well with noisy functional data. We also illustrate the advantages of using $QI{D}_K$ as a function of $K$ to identify potential hard-to-detect shape outliers.

2 Quantile Integrated Depth

Let $X$ be a space of functions $x:S\to R$ for $S\subset R^d$ a set of positive finite Lebesgue measure, and $d\ge 1$. An example is the Banach space $X=𝒞(S)$ of continuous functions $S\to R$ equipped with the supremum distance. Let $P$ be a Borel probability measure on the space of function $X$. For $X\sim P$ we write $P_{X(s)}\equiv P_s$ for the marginal distribution of the random variable $X(s)\in R$, with $s\in S$.

Suppose that a univariate depth $D$ is given. For a function $x\in X$ and with probability $P$, to each $s\in S$ we attach the depth $D(x(s);P_s)$ of the functional value $x(s)$ with respect to the corresponding marginal distribution $P_s$. We obtain a mapping

$$Y_{x,P}:S\to [0,1]:s\mapsto D(x(s);P_s).$$

Consider now $S$ with its Borel subsets and a Lebesgue measure $\lambda$ on $S$ as a probability space. Without loss of generality, we may suppose that $\lambda (S)=1$, otherwise we just consider a properly normalized Lebesgue measure instead of $\lambda$. The map $Y_{x,P}$ induces a pushforward Borel probability measure in $[0,1]$. We denote that measure by $D_{x,P}$, and its distribution function by

$$F_{x,P}:[0,1]\to [0,1]:t\mapsto \lambda \left(Y_{x,P}\le t\right)=D_{x,P}([0,t]).$$

This measures the proportion of time that the point-wise depth of $x$ is below $t$. We also write

$$F_{x,P}^{-1}:[0,1]\to [0,1]:u\mapsto inf\{t\in [0,1]:F_{x,P}(t)\ge u\}$$

for the corresponding quantile function. For a given $K\in (0,1)$, the quantile integrated depth of $x\in 𝒳$ w.r.t. $P\in 𝒫\left(𝒳\right)$ is defined as

$$QID(x;P)={\int }_0^K{F}_{x,P}^{-1}(u)𝑑u.$$ (1)

Intuitively, $QI{D}_K$ measures the integral of the $K$ smallest pointwise depths of function $x$. The distinctive feature of the Quantile Integrated depth ($QID$) is the attention it gives to the shape of the left lower tail of the pointwise depth distribution $D_{x,P}$ and this makes $QID$ suitable for identifying possible shape functional outliers. Also, it can be shown that $QID$ satisfies desirable theoretical properties and behaves well in terms of robustness in the presence of noise.