Skip to yearly menu bar Skip to main content


Poster

SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation

Misha Khodak · Lester Mackey · Miro Dudik · Alexandra Chouldechova

[ ]
Wed 11 Dec 4:30 p.m. PST — 7:30 p.m. PST

Abstract:

Disaggregated evaluation---estimating the performance of a machine learning model on different subpopulations---is an important task in the performance and group-fairness assessment of AI systems.However, evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, gender) are often tiny. Today, it is common for multiple clients to procure the same AI model from a model developer, and the challenge of disaggregated evaluation is faced by each customer individually. This gives rise to what we call the \textit{multi-task disaggregated evaluation problem}, wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task). In this work we develop a disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations. SureMap's efficiency gains come from (1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g. from the AI system creator or from their other clients. The method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE).We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong baselines.

Live content is unavailable. Log in and register to view live content