Robust estimation of reliability coefficients with rating-scale data

• Department of Psychology, University of Zurich, Zurich, Switzerland [max.welz@uzh.ch]

• Department of Psychology, Harvard University, Cambridge, Massachusets, USA [mair@fas.harvard.edu]

• Department of Econometrics, Erasmus University, Rotterdam, The Netherlands [alfons@ese.eur.nl]

Keywords: Robust estimation, Survey data, Careless responding, Cronbach’s $\alpha$, McDonald’s $\omega$

1 Abstract

Psychometric scales are sets of survey items that are designed to jointly measure the same latent variable, such as psychological personality traits, consumer attraction of a product, or political opinions and attitudes. To assess if a psychometric scale’s measurement is of sufficiently high quality, its internal reliability is typically estimated using so-called reliability coefficients, such as Cronbach’s $\alpha$ or McDonald’s $\omega$. However, uninformative “contaminated” responses might be present in survey data, such as careless or inattentive responses, which is particularly a concern in online surveys. We show that already a very low prevalence of uninformative responses can suffice to produce arbitrarily poor results in commonly used estimators of reliability coefficients.

As a remedy, we propose a novel robust estimator of popular reliability coefficients for psychometric survey scales, including Cronbach’s $\alpha$ and McDonald’s $\omega$. The estimator is based on robust estimation of a commonly used latent variable model for the correlation of rating-scale variables. This model is robustly fitted using the method of Welz, Mair, and Alfons [2024], which is in turn based on the theory of $C$-estimation developed by Welz [2024], being a general theory for robust estimation with categorical data. We show that our estimator is consistent, asymptotically normally distributed, and fully efficient in the absence of contamination, which follows directly from $C$-estimation theory derived in Welz [2024].

We demonstrate in simulation studies the robustness of our estimator against contamination in survey data, and show by means of an empirical application how it can be used to help identify careless or inattentive respondents.

As such, this paper complements Christmann and Van Aelst [2006], who propose a robustification of Cronbach’s alpha for continuous variables. In contrast, we offer a robustification for discrete rating-scale variables.

2 Implementation

The proposed methodology is implemented in the free open source R package robcat (for “ROBust CATegorical data analysis”), which is publicly available at https://github.com/mwelz/robcat and will soon be submitted to the Comprehensive R Archive Network (CRAN). To optimize computational speed and performance, package robcat is primarily developed in C++ and integrated to R via Rcpp [Eddelbuettel, 2013].