AIMI Research Meeting: Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains - Christian Bluethgen, PhD & Pierre Chambon, MSc

Abstract:

Multi-modal foundation models are typically trained on millions of pairs of natural images and text captions. Although such models depict excellent generative capabilities, they do not typically generalize well to specific domains such as medical images that have fundamentally shifted distributions compared to natural images. Building generative models for medical images that faithfully depict clinical context may help alleviate the paucity of healthcare datasets.

To investigate the capacity of a large pretrained latent diffusion model (Stable Diffusion) to generate medical domain-specific images, we explored the main components of the Stable Diffusion pipeline (the variational autoencoder, the U-Net and the text-encoder), to fine-tune the model to generate chest x-rays and evaluate the results on quantitative and qualitative levels. Our best-performing model can be text-conditioned to insert realistic-looking abnormalities like pleural effusions on synthetic radiological images, while maintaining a high accuracy on a classifier trained to detect the abnormality on real images.

About:

Christian Bluethgen, MD, MSc:

Christian is a physician-scientist, a radiologist with a clinical focus on thoracic imaging and currently a postdoctoral research fellow at the Stanford Center for Artificial Intelligence in Medicine and Imaging (AIMI). At the AIMI Center, he works on the application of deep learning for the detection, diagnosis and monitoring of interstitial lung disease and other thoracic pathologies, using large multi-modal imaging and EHR datasets. Before joining AIMI, he worked as a radiologist at the University Hospital Zurich, where he created computational simulations to visualize lung ultrasound wave propagation, developed NLP models for the classification of radiology reports, used radiomics for the differentiation of thymic neoplasms and deep learning for fracture detection and localization. In Zurich, he initiated the seminar series “Applied Machine Learning in Diagnostic Imaging” (currently in its fifth year), which brings together radiologists, researchers and clinicians from other medical specialties, and industry.

Pierre Chambon, MSc:

Pierre is an ML researcher, who recently graduated from a master at Stanford ICME and will soon graduate from a master in France at Ecole Centrale Paris, with a focus on mathematical and computational methods for machine learning and deep learning. He has been involved at the AIMI center for the last two years, where he developed NLP methods for domain-specific applications to radiology as well as multimodal tasks along vision models. As part of the MIDRC initiative, he worked on classification tasks in the data- and compute-constraint settings, as well as a text de-identification tool useful for the broad sharing of medical notes within and between institutions. More recently, he tackled both image-to-text and text-to-image tasks, leading to models that can generate synthetic radiology images and reports, hopefully further useful to other machine learning applications in radiology.