A Bayesian hierarchical model with spatially varying dispersion for reference-free cell type deconvolution in spatial transcriptomics

A major challenge in spatial transcriptomics (ST) is resolving cellular composition, especially in technologies lacking single-cell resolution. The mixture of transcriptional signals within spatial spots complicates deconvolution and downstream analyses. To uncover the spatial heterogeneity of tissues, we introduce SvdRFCTD, a reference-free spatial transcriptomics deconvolution method, which estimates the cell type proportions at each spot on the tissue. To fully capture the heterogeneity in the ST data, we combine SvdRFCTD with a Bayesian hierarchical negative binomial model with spatial effects incorporated in both the mean and dispersion of the gene expression, which is used to explicitly model the generative mechanism of cell type proportions. By integrating spatial information and leveraging marker gene information, SvdRFCTD accurately estimates cell type proportions and uncovers complex spatial patterns. We demonstrate the ability of SvdRFCTD to identify cell types on simulated datasets. By applying SvdRFCTD to mouse brain and human pancreatic ductal adenocarcinomas datasets, we observe significant cellular heterogeneity within the tissue sections and successfully identify regions with high proportions of aggregated cell types, along with the spatial relationships between different cell types.