Skip to content Skip to navigation

BMIR Webinar: The Science Of Measurement in Machine Learning - Jacob Steinhardt, PhD

April 8, 2021 - 12:00pm to 1:00pm

Jacob Steinhardt, PhD

Assistant Professor
UC Berkeley

Thursday, April 8st, 2021 12:00 to 1:00 pm


Live Stream :

Webinar Passcode: 403428

In machine learning, we are obsessed with datasets and metrics: progress in areas as diverse as natural language understanding, object recognition, and reinforcement learning is tracked by numerical scores on agreed-upon benchmarks. However, other ways of measuring ML models are underappreciated, and can unlock important insights. In this talk, I'll discuss two important quantities beyond test accuracy: datapoint-level variation and similarity between representations, and robustness. A challenge in both cases is that interesting trends are often dominated by statistical noise. We address this issue, and make new discoveries:

* As neural network models get larger, while overall accuracy increases, many individual predictions get worse.

* Models of different depth appear to still make similar computations in similar orders.

Beyond these specific observations, I will tie measurement to historical trends in science, and draw lessons from the success of biology and physics in the mid-20th century.


Contact Email: