AIMI Research Meeting: MedPerf Open and Standardized Benchmarking of Medical Artificial Intelligence – Roadmap and Call for Participation - Alexandros Karargyris, PhD & Renato Umeton, PhD

Abstract: AI has shown its potential to impact healthcare in unprecedented ways. However its clinical translation, from research to the real world, is held back by lack of robust validation in diverse patient populations. Repeated stories of AI trained on selected populations but failing in real world scenarios demonstrate the existing bias of healthcare AI, and more importantly, highlight the socioeconomic gap. In this talk we will be presenting MedPerf (http://medperf.org), an open source platform for neutral reproducible benchmarking of medical AI at global scale to help mitigate some of the current challenges. More specifically, we are going to be discussing current development and future directions of the platform. MedPerf is developed and maintained on a volunteering basis by a diverse group of industry and academic researchers offering a neutral approach to benchmarking. The platform is supported by MLCommons (http://mlcommons.org), a non-profit technical organization for benchmarking machine learning supported by industry and academia. We will take this opportunity also to invite your participation: we have worked with many researchers and companies and would love to see you involved too. Currently, MedPerf has collaborators across these companies, universities, and hospitals: A*STAR, Amazon, Brigham and Women's Hospital, Broad Institute of MIT and Harvard, Cisco, Dana-Farber Cancer Institute, Factored, Flower Labs, Fondazione Policlinico Universitario A. Gemelli IRCCS, German Cancer Research Center, Google, Harvard Medical School, Harvard T.H. Chan School of Public Health, Harvard University, Hugging Face, IBM Research, IHU Strasbourg, Intel, John Snow Labs, Landing.AI, Lawrence Livermore National Laboratory, MLCommons, Massachusetts Institute of Technology, Meta, Microsoft, NVIDIA, Nutanix, OctoML, Perelman School of Medicine, Red Hat, Rutgers University, Sage Bionetworks, Stanford University School of Medicine, Stanford University, Supermicro, Tata Medical Center, University of Cambridge, University of Heidelberg, University of Pennsylvania, University of Queensland, University of Strasbourg, University of Toronto, University of Trento, University of York, Vector Institute, Weill Cornell Medicine, Write Choice, cKnowledge, fast.ai.

Bio: Alexandros Karargyris leads the Medical working group at MLCommons. The group has a mission to provide neutral benchmarking and best practices for Medical AI in an effort to circumvent negative effects of AI. In its short life span the group has grown to support the largest federation for brain tumor segmentation in the world as well as to plan its first prospective AI studies. Previously he worked as a research lead at IHU Strasbourg in projects related to applications in the intersection of surgery and AI.  He also worked as a researcher at IBM and NIH for more than 10 years. His research interests lie in the space of medical imaging, machine learning and mobile health. He has contributed to healthcare commercial products and imaging solutions deployed in under-resourced areas. His research has been published in peer-reviewed journals and conferences.

Bio: Currently Renato serves as Director of Artificial Intelligence Operations and Data Science Services in the Informatics & Analytics department of Dana-Farber Cancer Institute, a teaching affiliate of Harvard Medical School. In this position, where he reports to the Chief Data and Analytics Officer, Renato created the departmental AI & data science horizontal, counting about 40 people (including temps) as of March 2023. He accrued 15 years of experience across artificial intelligence, data science, and big data working in  other hospitals, in academia, in consulting, and in industry, where he operated in roles spanning from postdoc to director. In those contexts, he worked on several scientific publications and patents, some of which were leveraged in clinical trials and others were licensed. As of March 2023, Renato co-authored 120+ scientific publications, 6+ patent applications, and he is currently affiliated also with MIT, Harvard T.H. Chan School of Public Health, and Weill Cornell Medicine. Renato also participates in various data science industry collaborations that aim at democratizing and expanding the reach of artificial intelligence and machine learning in healthcare and digital pathology.

In addition to his main responsibilities, he is/has been: (i) reviewer for various journals by Nature Publishing Group, Cell Press, IEEE, ACM, and Oxford Press among the others, (ii) invited speaker at venues ranging from a department of the US Federal Government to multiple Fortune 20 companies and numerous world-renowned academic medical centers, (iii) chair/organizer of several ML and AI communities/conferences (e.g., MedPerf.org and lod2023.icas.cc), (iv) mentor to 50+ mentees through programs that spanned from bootcamp to MD-PhD degree, and (v) manager for a global group of machine learning professionals counting 80,000+ members world-wide.

Outside of work he enjoys crossfit, a good hike, recreational spearfishing, the occasional cookout, traveling, and most of all he treasures spending time with his wife and children.

Attendance is open to the Stanford community. If you would like to attend on Zoom, please contact the AIMI Center at aimicenter@stanford.edu.