Jacob Steinhardt, PhD
Thursday, April 8st, 2021 12:00 to 1:00 pm
Webinar Passcode: 403428
In machine learning, we are obsessed with datasets and metrics: progress in areas as diverse as natural language understanding, object recognition, and reinforcement learning is tracked by numerical scores on agreed-upon benchmarks. However, other ways of measuring ML models are underappreciated, and can unlock important insights. In this talk, I'll discuss two important quantities beyond test accuracy: datapoint-level variation and similarity between representations, and robustness. A challenge in both cases is that interesting trends are often dominated by statistical noise. We address this issue, and make new discoveries:
* As neural network models get larger, while overall accuracy increases, many individual predictions get worse.
* Models of different depth appear to still make similar computations in similar orders.
Beyond these specific observations, I will tie measurement to historical trends in science, and draw lessons from the success of biology and physics in the mid-20th century.