Poster
in
Workshop: Bayesian Decision-making and Uncertainty: from probabilistic and spatiotemporal modeling to sequential experiment design
Incremental Uncertainty-aware Performance Monitoring with Labeling Intervention
Alexander Koebler · Thomas Decker · Ingo Thon · Volker Tresp · Florian Buettner
Keywords: [ optimal transport ] [ Uncertainty ] [ Temporal Distribution Shifts ] [ Model Monitoring ]
We study the problem of monitoring machine learning models under temporal distribution shifts, where circumstances change gradually over time, often leading to unnoticed yet significant declines in accuracy. We propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates model performance by modeling time-dependent shifts using optimal transport. IUPM also quantifies uncertainty in performance estimates and introduces an active labeling strategy to reduce this uncertainty. We further showcase the benefits of IUPM on different datasets and simulated temporal shifts over existing baselines.