Stability Properties of Empirical Risk Minimization Over Donsker Classes

Loading...
Thumbnail Image
Penn collection
Statistics Papers
Degree type
Discipline
Subject
empirical risk minimization
empirical processes
stability
Donsker classes
Statistics and Probability
Funder
Grant number
License
Copyright date
Distributor
Related resources
Author
Caponnetto, Andrea
Rakhlin, Alexander
Contributor
Abstract

We study some stability properties of algorithms which minimize (or almost-minimize) empirical error over Donsker classes of functions. We show that, as the number n of samples grows, the L2- diameter of the set of almost-minimizers of empirical error with tolerance x(n)=o(n-1/2 ) converges to zero in probability. Hence, even in the case of multiple minimizers of expected error, as n increases it becomes less and less likely that adding a sample (or a number of samples) to the training set will result in a large jump to a new hypothesis. Moreover, under some assumptions on the entropy of the class, along with an assumption of Komlos-Major-Tusnady type, we derive a power rate of decay for the diameter of almost-minimizers. This rate, through an application of a uniform ratio limit inequality, is shown to govern the closeness of the expected errors of the almost-minimizers. In fact, under the above assumptions, the expected errors of almost-minimizers become closer with a rate strictly faster than n-1/2.

Advisor
Date Range for Data Collection (Start Date)
Date Range for Data Collection (End Date)
Digital Object Identifier
Series name and number
Publication date
2006-12-01
Journal title
Journal of Machine Learning Research
Volume number
Issue number
Publisher
Publisher DOI
Journal Issue
Comments
At the time of publication, author Alexander Rakhlin was affiliated with Massachusetts Institute of Technology. Currently, he is a faculty member at the Statistics Department at the University of Pennsylvania.
Recommended citation
Collection