Novel Distance Regression for Repeated Outcomes With Missing Data: Applications to Longitudinal and Crossover Studies of Microbiome Beta-Diversity

The human microbiome plays a crucial role in health, but understanding its dynamic relationship with the host requires regular monitoring. Beyond challenges such as high dimensionality and sparsity, additional complexities arise, particularly within-cluster correlation from repeated measures and pervasive missing data. To address these issues, we develop Edger, a novel distance regression method for modeling community-level beta-diversity dynamics and their interactions with treatment or host physiology. By focusing on beta-diversity, a distance metric between microbial profiles, Edger (Ensembled semiparametric distance-based generalized estimation for repeated outcomes) directly models these distances as repeated outcomes, yielding interpretable coefficients and enabling a covariate batching strategy to mitigate omitted variable bias. Our semiparametric inference framework eliminates the need for time-consuming permutation tests, distinguishes between-cluster heterogeneity from within-cluster fluctuations, and allows flexible specification of working correlation structures. To handle missing data, we assume a missing-at-random (MAR) mechanism and incorporate a between-subject propensity score in the repeated distance regression to provide seamless joint inference, ensuring robust variance estimation without casewise deletion. Additionally, we introduce an algorithm to generate synthetic data from real-world microbial counts while preserving their zero-inflated and correlated nature. Edger demonstrates superior inferential power and computational efficiency through our numerical studies and real-world applications, making it a valuable tool for uncovering microbiome-host interactions and advancing multi-omics data integration.