Generalized Alpha-Beta Divergence, its Properties and Associated Entropy

Subhrajyoty Roy

{}^{1}

Supratik Basu

{}^{2}

Abhik Ghosh

{}^{3}

and Ayanendranath Basu

{}^{4}

${}^{1}$

Interdisciplinary Statistical Research Unit, Indian Statistical Institute, Kolkata, India [roysubhra98@gmail.com]
${}^{2}$

Duke University, NC, United States [supratik.basu@duke.edu]
${}^{3}$

Interdisciplinary Statistical Research Unit, Indian Statistical Institute, Kolkata, India [abhik.ghosh@isical.ac.in]
${}^{4}$

Interdisciplinary Statistical Research Unit, Indian Statistical Institute, Kolkata, India [ayanbasu@isical.ac.in]

Keywords: Statistical Divergence – Generalized Entropy – Breakdown Point Analysis

1 Abstract

In many applications of robust statistical inference, machine learning, pattern recognition, signal processing, statistical divergence measures play a key role. By using these divergence measures to quantify the distance between a family of model distributions (or densities) and the true data-generating distribution (or density), the minimum divergence estimators have become a popular tool in parametric inference, due to their simplicity and natural interpretability.

Within this realm, certain families of divergences stand out due to their efficiency in estimation and high robustness properties. Popular choices such divergences include squared Euclidean distance (also called $L^{2}$ distance), Cressie-Reed power divergence (PD) family comprising of Kullback Leibler (KL) divergence, Hellinger distance, Pearson’s chi-square, Neyman’s chi-square, etc., density power divergence family [Basu et al., 1998] (also known as Beta divergence), S-divergence family [Ghosh et al., 2017] (which is a subfamily of Alpha-Beta divergence), logarithmic S-divergence family [Maji et al., 2016] and $(\phi,\gamma)$ -divergence [Jones et al., 2001] among others. In this paper, we propose a very generalized form of the Alpha-Beta divergence family that encompasses all of the above as various special cases. For any two density functions $f$ and $g$ , we define this Generalized Alpha-Beta (GAB) divergence as

d_{GAB}^{(\alpha,\beta),\psi}(f,g)=\frac{\psi\left(\int f^{\alpha+\beta}\right% )}{\beta(\alpha+\beta)}-\frac{\psi\left(\int f^{\alpha}g^{\beta}\right)}{% \alpha\beta}+\frac{\psi\left(\int g^{\alpha+\beta}\right)}{\beta(\alpha+\beta)},

for hyperparameters $\alpha,\beta\in\mathbb{R}$ such that $\alpha\beta(\alpha+\beta)\neq 0$ and for a suitably chosen $\psi$ function. In our work, we first characterize the classes of generating function $\psi$ for which the above form remains nonnegative resulting in a valid measure of discrepancy between probability densities. This result helps us in constructing many new divergences and their associated entropy measures. Additionally, we illustrate various interesting properties like duality, convexity, generalized additivity and generalized Pythagorean identities for this general class of divergence and its associated family of entropy measures.

Another key result of our work is to provide a lower bound to the asymptotic breakdown point of the corresponding minimum GAB divergence estimator, when both $\alpha$ and $\beta$ are positive. This generalizes multiple results regarding the asymptotic breakdown point of minimum divergence estimators in the literature; see Roy et al. [2023] and the references therein. We also find that, in many setups, this lower bound remains free of the dimension of the data or the parameter space, and does not decay to $0$ , unlike the popular M-estimators. With different examples, we illustrate how this property can be extremely useful in analyzing high-dimensional data while ensuring robustness in the inference.

References

Basu et al. [1998] Ayanendranath Basu, Ian R. Harris, Nils L. Hjort, and M. C. Jones. Robust and Efficient Estimation by Minimising a Density Power Divergence. Biometrika, 85(3):549–559, 1998. ISSN 00063444. URL http://www.jstor.org/stable/2337385.
Ghosh et al. [2017] Abhik Ghosh, Ian R. Harris, Avijit Maji, Ayanendranath Basu, and Leandro Pardo. A generalized divergence for statistical inference. Bernoulli, 23(4A):2746–2783, 2017.
Jones et al. [2001] M. C. Jones, Nils Lid Hjort, Ian R. Harris, and Ayanendranath Basu. A comparison of related density-based minimum divergence estimators. Biometrika, 88(3):865–873, 2001. ISSN 00063444. URL http://www.jstor.org/stable/2673453.
Maji et al. [2016] Avijit Maji, Abhik Ghosh, and Ayanendranath Basu. The logarithmic super divergence and asymptotic inference properties. AStA Advances in Statistical Analysis, 100:99–131, 2016.
Roy et al. [2023] Subhrajyoty Roy, Abir Sarkar, Abhik Ghosh, and Ayanendranath Basu. Asymptotic breakdown point analysis for a general class of minimum divergence estimators. arXiv preprint arXiv:2304.07466, 2023.