BMIR Webinar: The Science Of Measurement in Machine Learning - Jacob Steinhardt, PhD

Zoom

Jacob Steinhardt, PhD

Assistant Professor
UC Berkeley

Thursday, April 8^st, 2021 12:00 to 1:00 pm

BMIR WEBINAR

Live Stream : https://stanford.zoom.us/j/91316729197?pwd=RWJ6dnJOYU5vUzdKbXdHRHdDVVNXZz09

Webinar Passcode: 403428

Abstract:  
In machine learning, we are obsessed with datasets and metrics: progress in areas as diverse as natural language understanding, object recognition, and reinforcement learning is tracked by numerical scores on agreed-upon benchmarks. However, other ways of measuring ML models are underappreciated, and can unlock important insights. In this talk, I'll discuss two important quantities beyond test accuracy: datapoint-level variation and similarity between representations, and robustness. A challenge in both cases is that interesting trends are often dominated by statistical noise. We address this issue, and make new discoveries:

* As neural network models get larger, while overall accuracy increases, many individual predictions get worse.

* Models of different depth appear to still make similar computations in similar orders.

Beyond these specific observations, I will tie measurement to historical trends in science, and draw lessons from the success of biology and physics in the mid-20th century.