Improving Classifier Confidence using Lossy Label-Invariant Transformations

Loading...
Thumbnail Image
Penn collection
Departmental Papers (CIS)
Degree type
Discipline
Subject
CPS Machine Learning
Computer Engineering
Computer Sciences
Funder
Grant number
License
Copyright date
Distributor
Related resources
Contributor
Abstract

Providing reliable model uncertainty estimates is imperative to enabling robust decision making by autonomous agents and humans alike. While recently there have been significant advances in confidence calibration for trained models, examples with poor calibration persist in most calibrated models. Consequently, multiple techniques have been proposed that leverage label-invariant transformations of the input (i.e., an input manifold) to improve worst-case confidence calibration. However, manifold-based confidence calibration techniques generally do not scale and/or require expensive retraining when applied to models with large input spaces (e.g., ImageNet). In this paper, we present the recursive lossy label-invariant calibration (ReCal) technique that leverages label-invariant transformations of the input that induce a loss of discriminatory information to recursively group (and calibrate) inputs – without requiring model retraining. We show that ReCal outperforms other calibration methods on multiple datasets, especially, on large-scale datasets such as ImageNet.

Advisor
Date of presentation
2021-04-01
Conference name
Departmental Papers (CIS)
Conference dates
2023-05-18T01:23:43.000
Conference location
Date Range for Data Collection (Start Date)
Date Range for Data Collection (End Date)
Digital Object Identifier
Series name and number
Volume number
Issue number
Publisher
Publisher DOI
Journal Issue
Comments
24th International Conference on Artificial Intelligence and Statistics (AISTATS)(https://aistats.org/aistats2021/) 2021, April 13-15 2021, Virtual. PMLR: Volume 130 (http://proceedings.mlr.press/v130/)
Recommended citation
Collection