AIMI Research Meeting: Jason Fries, PhD & Dave Van Veen

Title: Model Hubs for Medical AI: How Far is Our "Hugging Face" Moment?

Abstract: Model hubs like Hugging Face have revolutionized AI research and enterprise applications by enabling rapid prototyping and adaptation of pretrained models. However, in medicine, significant challenges persist in easily sharing and adapting models for complex tasks, such as generating representations of longitudinal patient EHR data. In this talk, I'll highlight our recent work training and evaluating large-scale foundation models (CLMBR and MOTOR) for structured EHR data. Our methods surpass state-of-the-art approaches, drastically reducing training data requirements by up to 95% for new tasks and demonstrating improved temporal, subpopulation, and geographic robustness. Finally, we discuss our efforts to share our EHR frameworks (FEMR) and pretrained models with the Stanford community through the STAnford Research Repository (STARR). We hope to foster collaborative discussions within the AIMI community on how to better achieve a vision of a shared model hub for Stanford researchers and beyond.

About: Jason Fries is a computer scientist at Stanford University's Center for Biomedical Informatics Research. His work focuses on methods that enable domain experts to rapidly build and modify machine learning models in complex domains such as medicine where obtaining large-scale, expert-labeled training data is a significant challenge. His research focuses on weakly supervised machine learning, foundation models, and methods for data-centric AI. He previously completed his postdoc in computer science at Stanford where he collaborated on development of the weakly supervised framework Snorkel.

Title: Clinical Text Summarization: Adapting LLMs Can Outperform Human Experts

Abstract: Although large language models (LLMs) have shown immense promise in natural language processing (NLP) tasks, their efficacy across diverse clinical summarization tasks has not yet been rigorously examined. In this work, we employ domain adaptation methods on LLMs, spanning four distinct summarization tasks. Our quantitative assessment reveals trade-offs between models and adaptation methods in addition to instances where recent advances in LLMs may not lead to improved results. Further, in a clinical reader study with six physicians, we depict that summaries from the best adapted LLM are preferable to human summaries in terms of completeness and correctness. Our ensuing qualitative analysis delineates mutual challenges faced by both LLMs and human experts. Lastly, we correlate traditional quantitative NLP metrics with reader study scores to enhance our understanding of how these metrics align with physician preferences. Our research marks the first evidence of LLMs outperforming human experts in clinical text summarization across multiple tasks. This implies that integrating LLMs into clinical workflows could alleviate documentation burden, empowering clinicians to focus more on personalized patient care and other irreplaceable human aspects of medicine.

About: Dave Van Veen is a PhD candidate in Electrical Engineering (EE) advised by John Pauly and Akshay Chaudhari. His research threads include developing machine learning (ML) algorithms for computational imaging problems and applying large language models to improve clinical workflow. Prior to beginning his PhD, Dave spent two years as a research scientist at a Bay Area start-up and the AI in Medicine & Imaging (AIMI) center at Stanford. Previously he earned a MS in EE at the University of Texas conducting ML research, during which he also served as a Data Science for Social Good fellow in London. Before that Dave earned a BS in EE at the University of Wisconsin, where he created and led a 150-person organization to build a hyperloop pod for SpaceX.

Attendance is open to the Stanford community. If you would like to attend in-person or on Zoom, please contact the AIMI Center at aimicenter@stanford.edu.