Addressing unobserved heterogeneity in multistate event history models with weighted survival time

Abstract

Conventional survival analysis statistical models assume that the baseline hazard and covariate values determine the hazard function, ignoring unobserved factors that may affect survival outcomes. However, other factors can impact survival time distributions, resulting in variation among seemingly comparable individuals. Frailty models can account for unobserved factors and help cluster survival data. This study applies frailty models to multistate event history data to demonstrate their capacity to handle unobserved heterogeneity. Individual-specific survival weights change survival times to better reflect unmeasured influences, a fundamental component of our approach. When data are biassed or typical survival models fail to capture the full influence of variables, weighted survival time is critical.We simulate the efficacy and performance of frailty models in the multistate framework by comparing regression coefficient mean, MSE, and bias with and without frailty. Our findings emphasise the importance of unobserved fluctuations in survival analysis, especially with multistate models and weighted survival periods.

• Department of Mathematics & Computing,Indian Institute of Technology Dhanbad, Dhanbad-826004, India

• Population Health and Genomics, School of Medicine,University of Dundee,United Kingdom

• Department of Mathematics & Computing,Indian Institute of Technology Dhanbad, Dhanbad-826004, India [21dr0005@mc.iitism.ac.in]

Keywords: Multistate models – Heterogeneity – Weighted survival time

1 Introduction

Applications of frailty models in multistate event history data are well-established, with numerous studies demonstrating their effectiveness in capturing the complex dynamics of survival processes Wienke [2003], Duchateau and Janssen [2008]. Aalen (1988) proposed a time-homogeneous Markov model with frailty, where the frailty term affects all transitions within the Markov process Aalen [1988]. Other works have extended this approach to explore random effects across transitions in various medical conditions, such as Bhattacharya and Klein’s work on event-state transitions and Yen et al.’s study on adenoma-carcinoma progression in the small bowel Bhattacharyya and Klein [2005], Yen et al. [2010].

To measure the impact of unobserved fluctuations, we propose a modified survival time with individual-level survival weights. Survival periods without heterogeneity or fragility may bias results. Frailty models alter survival time to account for shared frailty in survey research with groups or clusters (e.g., families, ethnic groups). For personalised survival analysis, the model incorporates frailty into survival time since weak people experience events earlier. We created a multistate dataset with exponential, Weibull, and Gompertz parametric baseline risks and analysed frailty models for different transitions using a gamma distribution to explain this scenario in practical data analysis for the simulation study. We show how frailty models improve survival predictions by accounting for random effects and clustering in multistate data by applying them to real dataset . This method reveals survival process dependencies and assumptions, as well as how unobserved variability affects survival results.

2 Methods

Frailty models are especially significant in the realm of Multistate Models (MSMs), because individuals can shift among several health states over time. MSMs enhance conventional survival models by monitoring not just the occurrence of a singular event but also a sequence of transitions between states (e.g., from ”healthy” to ”diseased” to ”dead”). In practical situations, the risk of moving from one state to another may rely not just on reported factors but also on unobserved frailty, which adds individual variability to these transitions.

The proportional hazards model accounted for frailty in the presence of covariates can be adjusted as:

$h(t|Z,𝐗)$ $=h_0(t)\exp (\beta X+\omega Z)$

$=Zh(t|𝐗=𝐱)$ (1)

Here, $X$ denotes the vector of observed covariates, while $\omega$ represents the unobserved random effects due to frailty. This extended formulation allows the model to incorporate both observed covariate effects and unobserved heterogeneity, improving its accuracy in predicting survival outcomes, especially in multistate settings.

The multi-state model, represents disease progression through three states: 1 (healthy), 2 (ill), and 3 (death). The hazard of transitioning between these states is modeled using a transition-specific Cox proportional hazards (Cox PH) model. This means that for each possible transition (e.g., healthy to ill, ill to death), we estimate a separate hazard function. When incorporating frailty, the hazard of transitioning from state $i$ to state $j$ (where $i\ne j$) is given by:

$$h_{ij}(t|𝐗=𝐱,Z_{ij})=h_{0ij}(t)\exp ({\beta }^T{𝐱}_{\mathrm{𝐢𝐣}}+\omega Z_{ij})=Z_{ij}{h}_{0ij}(t)\exp ({\beta }^T𝐱)$$ (2)

In Stochastic Markov Processes, the transition rate between states depends on the elapsed time in the current state, independent of the trajectory followed to go there Sahner et al. [1997]. This renders SMPs more adaptable and appropriate for modelling processes like disease progression, recovery, or relapse, when the duration in intermediate states affects subsequent results Asanjarani et al. [2022].

2.1 Weights on Survival Time in Multi-State Models

In many real-world applications, such as modeling disease progression, it is essential to account for the time individuals spend in each state before transitioning to the next. To define an SMP in a multistate model, consider a homogeneous Markov chain $J_n,n\ge 0$ with finite state space $S=\{1,2,3\}$. The state $J_n$ denotes the state of the system following $n$ transitions. The transition probability for the $n$-th jump from state $i$ to state $j$ is given by:

$$P_{ij}=P(J_n=j|J_{n-1}=i)$$ (3)

where $i,j=0,1,2,3$ and $i\ne j$.

Let $k=1,2,\mathrm{\dots },n$ represent the number of individuals undergoing transition $\{ij\}$. The weighted survival time for an individual is defined as:

$$\widehat{S_k}(t)=\sum _{i\ne j}{w}_{ij}^{{}^{\prime }}{S}_{ij}(t)$$ (4)

where $w_{ij}^{{}^{\prime }}$ is the weight assigned to a specific transition, and $S_{ij}(t)$ is the corresponding survival time. By adjusting for these individual and group-level factors, the model delivers better predictions and reflects the complexity of real-world transitions. In this context we have developed an R function weightedOS_msm to generate a MSM with weighted overall survival time.