Hybrid Attention-State Space Model Backbone for Cancer Pathology Vision Foundation Model with Superior Downstream Task Performance
Colorectal cancer (CRC) management increasingly depends on the precise histologic and molecular characterization of pathology slides—including subtyping, polyp and adenoma grading, microsatellite instability (MSI) prediction, biomarker assessment, and survival prediction. Digital histopathology vision foundation models (VFMs) trained on whole slide images (WSIs) have shown promise across these tasks, but current Vision Transformer (ViT)-based VFMs still underperform on clinically demanding endpoints, particularly those tied to molecular phenotypes where the relevant tissue features are subtle and lie beyond human perception. To address this, we propose MVHybrid, a hybrid vision backbone combining state space models (SSMs) with ViT layers, motivated by the observation that SSMs initialized with negative real eigenvalues exhibit a strong bias toward the lowfrequency morphological signals correlated with molecular states. We evaluate MVHybrid across the breadth of downstream tasks central to CRC pathology AI. MVHybrid matched or outperformed ViT baselines across nearly every downstream metric. The largest and most clinically meaningful gains were in gene expression (spatial transcriptomics) prediction—MVHybrid achieved roughly 57% higher correlation than that of the best-performing ViT. We also observed consistent gains of MVHybrid’s downstream performance in clinically significant Microsatellite Instability (MSI), polyp/adenoma, and tissue classification. Similar trends could also be found in downstream survival prediction and zero-shot retrieval tasks.