class: title background-image: url("figure/chimborazo-data.png") background-size: cover .pull-left[ <div id="title"> Ecosystem Modeling using Multimodal Data </div> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <div id="subtitle"> Microsoft AI for Social Good Lab <br/> Kris Sankaran <br/> 05 | July | 2023 <br/> https://github.io/krisrs1128/LSLab </div> ] --- ### Introduction These themes have recurred across almost all the projects that I have studied: * **Multimodality**: Data are gathered from complementary sensors or assays. * **Ecosystem Analysis**: We would like to understand the interactions across a landscape, not just the isolated components .pull-left[ <img width=400 src="figure/sensor_equipment.png"/> ] .pull-right[ <img width=380 src="figure/bacteria_ecosystem.png"/> ] --- ### Multimodality Setup .center[ <img width=850 src="figure/multimodal.png"/> ] --- ### Multimodal Dimensionality Reduction * Canonical correlation analysis looks for shared structure across sources, `\begin{align*} \arg\min_{u \in \mathbb{R}^{P_1}, v \in \mathbb{R}^{P_2}} &\operatorname{Cov}_{\mathbf{P}^{X_{1}X_{2}}}\left[z_i^{(1)}(u), z_i^{(2)}(v)\right] \\ \text { subject to } &\operatorname{Var}_{\mathbf{P}^X_{1}}\left(z_i^{(1)}(u)\right)= \operatorname{Var}_{\mathbf{P}^{X_{2}}}\left(z_i^{(2)}(v)\right)=1 \end{align*}` where `\(z_i^{(1)}(u)=u^T x^{(1)}_i\)` and `\(z_i^{(2)}(v)=v^T x^{(2)}_i\)` are linear feature extractors. * In modern settings, it's often useful to introduce sparsity [1; 2] or nonlinearity [3; 4]. --- ### Multimodal Regression * If the goal is to predict a response from several sources, it can be helpful to use dimensionality reduction that emphasizes shared structure. * For example, cooperative learning [5] solves: `\begin{align*} \arg\min_{\beta_1 \in \mathbb{R}^{P_{1}}, \beta_2 \in \mathbb{R}^{P_2}} &\frac{1}{2}\left\|y-X_{1} \beta_{1}-X_{2} \beta_{2}\right\|^2+\frac{\rho}{2}\left\|\left(X_{1} \beta_{1}-X_{2} \beta_{2}\right)\right\|^2 + \\ &\lambda_1\left\|\beta_{1}\right\|_1+\lambda_2\left\|\beta_{2}\right\|_1 . \end{align*}` The second term is similar to the CCA objective, highlighting shared structure across tables --- ### Non-Exchangeability We often have context about how samples are related to one another, and using this information can improve power. .pull-left[ - Spatial or temporal structure - Cohort or batch effects - Network structure ] .pull-right[ <img src="figure/nonexchangeability.png"/> ] --- ### Mosaics Often, the same measurements are not available across all samples. We need to integrate across sources without assuming uniform collection. .center[ <img width=750 src="figure/mosaic.png"/> ] --- background-color: #f7f7f7 background-image: url("figure/multimodal-glaciers-cover.png") background-size: contain .center[ ## Multimodality in Earth Observation ] --- ### Glacier Monitoring * Glaciers provide significant ecosystem resources, and their disappearance has a large impact on the communities around them. * Effective remote sensing data analysis could help generate maps over large areas much faster than human annotation could. .pull-left[ <img width=300 src="figure/Blue-logo-on-a-white-background.jpg"/> ] .pull-right[ <a href="https://news.microsoft.com/on-the-issues/2021/01/12/ai-open-data-glacial-melt-himalaya/"><img width=300 src="figure/glaciers-article.png"/></a> ] --- ### Glacier Monitoring Semantic segmentation models can be trained to distinguish different types of glaciers and estimate glacial lake areas [6; 7; 8] .pull-left[ <img src="figure/glacier_lakes_trends.png"/> <img src="figure/glacier_lakes_volcano.png" width=250/> ] .pull-right[ <img src="figure/glacier_lakes_errors.png"/> ] [Colab Notebook](https://colab.research.google.com/drive/1ZkDtLB_2oQpSFDejKZ4YQ5MXW4c531R6?usp=sharing#scrollTo=r9jkq9qYcX_-) --- ### Multimodal Data Data sources are always changing, and to map historical trajectories, we need to integrate them. * There is a trade-off between spatial and spectral resolution. * It is possible to include ground-level data, e.g., wildlife monitoring apps. .center[ <img width=500 src="figure/satellites-evolution.png"/> ] The dream would be to build models that automatically get better as new sources become available. --- ### Multimodal Data Data sources are always changing, and to map historical trajectories, we need to integrate them. * There is a trade-off between spatial and spectral resolution. * It is possible to include ground-level data, e.g., wildlife monitoring apps. .center[ <img width=600 src="figure/two-maps.png"/> ] The dream would be to build models that automatically get better as new sources become available. --- ### An Experiment <div id="credit"> Yuliang Peng <br> <img width=100 src="figure/portrait.png"> </div> .pull-left[ How robust is multimodal segmentation to modality missingness? * Does it depend on transformer architectures? * This is the remote sensing analog of [9]. ] .pull-right[ <img width=500 src="figure/study-design.jpeg"/> ] --- ### Study Design * We studied Segformer models [10] with early, middle, and late fusion. * We downloaded aligned S1 and S2 imagery using the planetary computer ([script](https://github.com/krisrs1128/lake_labeller/blob/main/download/helpers.py), [data](https://github.com/krisrs1128/lake_labeller/blob/main/download/data_paths.csv)) .center[ <img width=600 src="figure/transformer-types.jpeg"/> ] --- ### Results .pull-left[ | | Early fusion | | Late fusion | | |-------|----------------|-----------|----------------|-----------| | | Debris-covered | Clean ice | Debris-covered | Clean ice | | S1+S2 | 0.252 | 0.651 | 0.159 | 0.642 | | S1 | 0.002 | 0.087 | 0 | 0.061 | | S2 | 0.249 | 0.615 | 0 | 0.423 | ] .pull-right[ * The early fusion model uniformly outperformed the late fusion model - S2 is much more important than S1 * Caveat: This experiment used a small region with few debris-covered glaciers ] --- ### Results Example predictions from patches with the highest and lowest IoUs. <img src="figure/results-array-glaciers.png"/> --- background-image: url("figure/microbiome-header-2.png") .center[ <div id="microbiome-header"> Multimodality in Microbiome Studies </div> ] --- ### Bacterial Vaginosis .pull-left[ * Bacterial Vaginosis (BV) affects 20% of women worldwide, with elevated rates in Sub-Saharan Africa. It is a known risk factor for preterm birth and HIV transmission. * BV is a disease of imbalance -- it doesn't come from a single pathogen, but rather interactions across the microbial community / host immune system. ] .pull-right[ <img src="figure/bv_overview.jpg" width=500/> Figure from [11]. ] --- ### Bacterial Vaginosis * Several initiatives, including the Gates Foundation-funded Vaginal Microbiome Research Consortium, are gathering multimodal molecular ("multi-omics") profiles to clarify the mechanisms behind disease development and recurrence * The goal is to use these multi-omics profiles to design precision treatments. * E.g., new probiotics, vaginal fluid/microbiome transplants --- ### Multimodal Data Different molecular assays are needed to capture different aspects of both the microbiome and the host immunological environment over time. * Human Host - Single Cell RNA-seq: How are genes from different host cell types expressed? - Cytokine Assays: Which immune system cells are present? - Survey Data: What host behaviors may influence disease trajectory? * Microbiome - 16S rRNA sequence: What are the bacterial species abundances? - Metagenomic Sequencing: What functions can those bacteria potentially carry out? - Metatranscriptomic Sequencing: Which genes are expressed? --- ### Integrated Learning Tools are needed across the entire data collection and analysis workflow: * Experimental design and power analysis. * Normalization, batch effect correction, missing data imputation. * Disease state/trajectory prediction. * Interpretation and visualization. --- ### Example: Intervention Analysis <div id="credit"> Pratheepa Jeganathan <br> <img width=100 src="figure/portrait-2.jpeg"> </div> What would longitudinal trajectories be like with or without specific interventions? We adapted ideas from the transfer function methodology. .pull-left[ `\begin{align} \mathbf{y}_{t} = \sum_{p = 1}^{P} A_{p} \mathbf{y}_{t - p} + \sum_{q = 0}^{Q - 1} B_{q}\mathbf{w}_{t - q} + \mathbf{\epsilon}_{t} \end{align}` * `\(\mathbf{y}_{t}\)` is a `\(J\)`-dimensional vector of molecular features at time `\(t\)` * `\(\mathbf{w}_{t}\)` is an intervention indicator * `\(A\)` and `\(B\)` track inter-feature and intervention effects, respectively ] .pull-right[ <img width=500 src="figure/transfer-diagram.png"/> ] --- ### Example: Intervention Analysis <div id="credit"> Pratheepa Jeganathan <br> <img width=100 src="figure/portrait-2.jpeg"> </div> What would longitudinal trajectories be like with or without specific interventions? We adapted ideas from the transfer function methodology. .pull-left[ `\begin{align} y_{t j}=f_j\left(\mathbf{y}_{(t-P-1):(t-1)^{\prime}} \mathbf{w}_{(t-Q+1) x}\right)+\epsilon_{j t} \end{align}` * `\(\mathbf{y}_{t}\)` is a `\(J\)`-dimensional vector of molecular features at time `\(t\)` * `\(\mathbf{w}_{t}\)` is an intervention indicator * `\(f_{j}\)` captures nonlinear effects across interventions and molecular features ] .pull-right[ <img width=500 src="figure/transfer-diagram.png"/> ] --- ### Example: Intervention Analysis .center[ <img src="https://krisrs1128.github.io/mbtransfer/articles/postpartum_files/figure-html/unnamed-chunk-15-1.png"/> ] [Manuscript](https://arxiv.org/abs/2306.06364), [Documentation](https://krisrs1128.github.io/mbtransfer/), [Demo](https://mybinder.org/v2/gh/krisrs1128/mbtransfer_demo/HEAD?urlpath=rstudio) --- .center[ ## Closing Thoughts ] --- ### Mount Chimborazo This is one of my favorite visualizations from the history of science. .center[ <img width=950 src="figure/Geographie_der_Pflanzen_in_den_Tropen-Ländern.jpg"/> ] --- ### Mount Chimborazo It is Alexander von Humboldt's *Tableau Physique*, and it shows how multiple data sources can be combined to create a unified view of the world. It is also remarkable how it anticipates many ideas from modern ecology. .center[ <img width=820 src="figure/Geographie_der_Pflanzen_in_den_Tropen-Ländern.jpg"/> ] <div id="von-humboldt"> <img src="figure/von-humboldt.jpeg"/> </div> --- ### Interdependence At a time when most science focused on taxonomy, the *Tableau* highlighted interdependence and unity within nature. .center[ <img width=600 src="figure/chimborazo-data.png"/> ] --- ### References [1] D. Y. Ding, S. Li, B. Narasimhan, et al. "Cooperative learning for multiview analysis". In: _Proceedings of the National Academy of Sciences of the United States of America_ 119 (2021). [2] D. M. Witten, R. Tibshirani, and T. J. Hastie. "A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis." In: _Biostatistics_ 10 3 (2009), pp. 515-34 . [3] C. Gao, Z. Ma, and H. H. Zhou. "Sparse CCA: Adaptive Estimation and Computational Barriers". In: _arXiv: Methodology_ (2014). [4] G. Andrew, R. Arora, J. A. Bilmes, et al. "Deep Canonical Correlation Analysis". In: _International Conference on Machine Learning_. 2013. --- ### References [5] W. Wang, R. Arora, K. Livescu, et al. "On Deep Multi-View Representation Learning". In: _International Conference on Machine Learning_. 2015. [6] S. Baraka, B. Akera, B. Aryal, et al. "Machine Learning for Glacier Monitoring in the Hindu Kush Himalaya". In: _ArXiv_ abs/2012.05013 (2020). [7] M. Zheng, X. Miao, and K. Sankaran. "Interactive Visualization and Representation Analysis Applied to Glacier Segmentation". In: _ISPRS Int. J. Geo Inf._ 11 (2021), p. 415. [8] A. Ortiz, W. Tian, T. C. Sherpa, et al. "Mapping Glacial Lakes Using Historically Guided Segmentation Models". In: _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ 15 (2022), pp. 9226-9240. --- ### References [9] C. M. Mitchell and J. M. Marrazzo. "Bacterial Vaginosis and the Cervicovaginal Immune Response". In: _American Journal of Reproductive Immunology_ 71 (2014), pp. 555 - 563. [10] M. Ma, J. Ren, L. Zhao, et al. "Are Multimodal Transformers Robust to Missing Modality?" In: _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2022), pp. 18156-18165. [11] E. Xie, W. Wang, Z. Yu, et al. "SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers". In: _ArXiv_ abs/2105.15203 (2021).