Unifying Weights and Loadings in Sparse Component Analysis via Equality and Cardinality Constraints

Hetvi Chaniyara ${}^a$ and Katrijn Van Deun${}^b$

${}^a$Technical University of Berlin, ${}^b$Tilburg University

As high-dimensional, low-sample-size (HDLSS) datasets become increasingly common, there is a growing reliance on sparse principal component analysis (SPCA) and sparse factor analysis (SFA) for dimension reduction and the exploration of underlying latent structures. Their primary objective is to obtain interpretable low-dimensional representations through sparse components. Existing approaches differ in how sparsity is imposed. Weight-based SPCA methods yield explicit component scores but often suffer from instability in variable selection, as markedly different sparse weight vectors can produce nearly identical scores, a problem aggravated in HDLSS settings. Loading-based SFA methods instead promote simple structure and typically yield stable loading patterns, yet lack uniquely defined component scores due to factor score indeterminacy. Hybrid formulations include both weights and loadings but retain ambiguity regarding which parameters should guide interpretation.

We propose a sparse component method that resolves this tension by imposing equality between weights and loadings within a single constrained optimization framework. This constraint eliminates interpretationpal ambiguity, ensures uniquely defined component scores, and improves stability of variable selection. To further promote simple structure, a cardinality constraint is imposed instead of shrinkage-based regularization, motivated by its lower estimation bias and favorable support recovery properties in sparse PCA settings.

Estimation is performed using an alternating optimization scheme combining the Alternating Direction Method of Multipliers (ADMM) with Majorization–Minimization, enabling efficient computation in high-dimensional settings. Simulation studies and an empirical application demonstrate that the proposed approach yields stable and interpretable sparse components while simultaneously providing well-defined component scores. The method is compared to well-known sparse PCA methods.

Keywords: Sparse PCA, Constrained optimization.