Resources Test Page

Anytime-Valid PAC-Bayes for Industrial Applications

Companies often need to track a variety of key business and performance metrics and take action if they change significantly. It’s also important, though, to avoid unnecessary action because of purely random fluctuations; the standard approach to identify which changes are “real” is statistical hypothesis testing. Traditional methods, however, are designed for “looking” only once; checking continuously breaks their guarantees on false positive rates. A growing number of companies – including Adobe, Microsoft, Amazon, and Netflix – have now adopted safe "anytime-valid" inference tools which remain valid when tested continuously. Currently, the anytime-valid tools deployed by these companies allow experimenters to, for instance, continuously monitor simple low-dimensional linear models which are amenable to well understood statistical tests. But companies are becoming increasingly interested in going beyond these simple models, and to incorporate recent breakthroughs in deep learning and AI into their products and analyses (e.g., object recognition in autonomous cars). There is thus growing need for an anytime-valid metric for generalization performance, which can be used as a decision criterion for determining when to retrain and redeploy sophisticated, non-linear, and high-dimensional models. In collaboration with researchers at Carnegie Mellon University, this project aims to extend the anytime-valid inference theory into the PAC-Bayes setting, with a focus on developing novel theory for concrete applications to be used in an industrial setting.