Workshop: Consequential Decisions in Dynamic Environments

Niki Kilbertus, Angela Zhou, Ashia Wilson, John Miller, Lily Hu, Lydia T. Liu, Nathan Kallus, Shira Mitchell

2020-12-12T08:00:00-08:00 - 2020-12-12T15:50:00-08:00
Abstract: Machine learning is rapidly becoming an integral component of sociotechnical systems. Predictions are increasingly used to grant beneficial resources or withhold opportunities, and the consequences of such decisions induce complex social dynamics by changing agent outcomes and prompting individuals to proactively respond to decision rules. This introduces challenges for standard machine learning methodology. Static measurements and training sets poorly capture the complexity of dynamic interactions between algorithms and humans. Strategic adaptation to decision rules can render statistical regularities obsolete. Correlations momentarily observed in data may not be robust enough to support interventions for long-term welfaremits of traditional, static approaches to decision-making, researchers in fields ranging from public policy to computer science to economics have recently begun to view consequential decision-making through a dynamic lens. This workshop will confront the use of machine learning to make consequential decisions in dynamic environments. Work in this area sits at the nexus of several different fields, and the workshop will provide an opportunity to better understand and synthesize social and technical perspectives on these issues and catalyze conversations between researchers and practitioners working across these diverse areas.

Video

Chat

Chat is not available.

Schedule

2020-12-12T08:00:00-08:00 - 2020-12-12T08:10:00-08:00
Welcome and introduction
2020-12-12T08:10:00-08:00 - 2020-12-12T08:30:00-08:00
Invited Talk 1: What do we want? And when do we want it? Alternative objectives and their implications for experimental design.
Maximilian Kasy
This talk will be based, in particular, on the following two papers: Adaptive treatment assignment in experiments for policy choice (joint with Anja Sautmann) Forthcoming, Econometrica, 2020 Manuscript: https://maxkasy.github.io/home/files/papers/adaptiveexperimentspolicy.pdf An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan (joint with Stefano Caria, Grant Gordon, Soha Osman, Simon Quinn and Alex Teytelboym) Working paper, 2020 Manuscript: https://maxkasy.github.io/home/files/papers/RefugeesWork.pdf
2020-12-12T08:30:00-08:00 - 2020-12-12T08:50:00-08:00
Invited Talk 2: Country-Scale Bandit Implementation for Targeted COVID-19 Testing
Hamsa Bastani
In collaboration with the Greek government, we use machine learning to manage the threat of COVID-19. With tens of thousands of international visitors every day, Greece cannot test each visitor to ensure that they are not a carrier of COVID-19. We developed a bandit policy that balances allocating scarce tests to (i) continuously monitor the dynamic infection risk of passengers from different locations (exploration), and (ii) preferentially target risky tourist profiles for testing (exploitation). Our solution is currently deployed across all ports of entry to Greece. I will describe a number of technical challenges, including severely imbalanced outcomes, batched/delayed feedback, high-dimensional arms, port-specific testing constraints, and transferring knowledge from (unreliable) public epidemiological data. Joint work with Kimon Drakopoulos, Vishal Gupta and Jon Vlachogiannis.
2020-12-12T08:50:00-08:00 - 2020-12-12T09:00:00-08:00
Q&A for invited talks 1&2
2020-12-12T09:00:00-08:00 - 2020-12-12T10:00:00-08:00
Poster Session 1
2020-12-12T10:00:00-08:00 - 2020-12-12T10:20:00-08:00
Break 1
2020-12-12T10:20:00-08:00 - 2020-12-12T10:30:00-08:00
Introduction of invited speakers 3, 4
2020-12-12T10:30:00-08:00 - 2020-12-12T10:50:00-08:00
Invited Talk 3: Modeling the Dynamics of Poverty
Rediet Abebe
The dynamic nature of poverty presents a challenge in designing effective assistance policies. A significant gap in our understanding of poverty is related to the role of income shocks in triggering or perpetuating cycles of poverty. Such shocks can constitute unexpected expenses -- such as a medical bill or a parking ticket -- or an interruption to one’s income flow. Shocks have recently garnered increased public attention, in part due to prevalent evictions and food insecurity during the COVID-19 pandemic. However, shocks do not play a corresponding central role in the design and evaluation of poverty-alleviation programs. To bridge this gap, we present a model of economic welfare that incorporates dynamic experiences with shocks and pose a set of algorithmic questions related to subsidy allocations. We then computationally analyze the impact of shocks on poverty using a longitudinal, survey-based dataset. We reveal insights about the multi-faceted and dynamic nature of shocks and poverty. We discuss how these insights can inform the design of poverty-alleviation programs and highlight directions at this emerging interface of algorithms, economics, and social work.
2020-12-12T10:50:00-08:00 - 2020-12-12T11:10:00-08:00
Invited Talk 4: From Moderate Deviations Theory to Distributionally Robust Optimization: Learning from Correlated Data
Daniel Kuhn
We aim to learn a performance function of the invariant state distribution of an unknown linear dynamical system based on a single trajectory of correlated state observations. The function to be learned may represent, for example, an identification objective or a value function. To this end, we develop a distributionally robust estimation scheme that evaluates the worst- and best-case values of the given performance function across all stationary state distributions that are sufficiently likely to have generated the observed state trajectory. By leveraging new insights from moderate deviations theory, we prove that our estimation scheme offers consistent upper and lower confidence bounds whose exponential convergence rate can be actively tuned. In the special case of a quadratic cost, we show that the proposed confidence bounds can be computed efficiently by solving Riccati equations.
2020-12-12T11:10:00-08:00 - 2020-12-12T11:20:00-08:00
Q&A for invited talks 3, 4
2020-12-12T11:20:00-08:00 - 2020-12-12T11:25:00-08:00
Contributed Talk 1: Fairness Under Partial Compliance
Jessica Dai, Zachary Lipton
2020-12-12T11:25:00-08:00 - 2020-12-12T11:30:00-08:00
Contributed Talk 2: Better Together? How Externalities of Size Complicate Notions of Solidarity and Actuarial Fairness
Kate Donahue, Solon Barocas
2020-12-12T11:30:00-08:00 - 2020-12-12T11:35:00-08:00
Contributed Talk 3: Algorithmic Recourse: from Counterfactual Explanations to Interventions
Amir Karimi, Bernhard Schölkopf, Isabel Valera
2020-12-12T11:35:00-08:00 - 2020-12-12T11:45:00-08:00
Q&A for contributed talks 1,2,3
2020-12-12T11:45:00-08:00 - 2020-12-12T12:20:00-08:00
Break 2
2020-12-12T12:20:00-08:00 - 2020-12-12T12:30:00-08:00
Introduction of invited speakers 5, 6, 7
2020-12-12T12:30:00-08:00 - 2020-12-12T12:50:00-08:00
Invited Talk 5: What are some hurdles before we can attempt machine learning? Examples from the Public and Non-Profit Sector
Mitsue Iwata
Machine learning and predictive analytics are more accessible to the public and nonprofit space now more than ever. Local government and nonprofits strive to leverage these new technologies to improve outcomes, performance, and operations. While a willingness to collaborate and connect on common goals though a shared understanding of data needs serves to build towards a stronger culture around data, the complexities around defining critical terms in dynamic environments pose significant hurdles to be able to scale any machine learning for large cross-departmental initiatives in service of the public. I will share examples from my professional work in NYC government, and probe into challenges with data-driven processes, consensus-based motivations and outcomes.
2020-12-12T12:50:00-08:00 - 2020-12-12T13:13:00-08:00
Invited Talk 6: Unexpected Consequences of Algorithm-in-the-Loop Decision Making
Yiling Chen
The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an “algorithm-in-the-loop” process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decision making. In this talk, I’ll discuss results from a set of controlled experiments on algorithm-in-the-loop human decision making in two contexts (pretrial release and financial lending). For example, when presented with algorithmic risk assessments, our study participants exhibited additional bias in their decisions and showed a change in their decision-making process by increasing risk aversion. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact. This talk is based on joint work with Ben Green.
2020-12-12T13:13:00-08:00 - 2020-12-12T13:35:00-08:00
Invited Talk 7: Prediction Dynamics
Moritz Hardt
2020-12-12T13:35:00-08:00 - 2020-12-12T13:50:00-08:00
Q&A for invited talks 5, 6, 7
2020-12-12T13:50:00-08:00 - 2020-12-12T14:20:00-08:00
Break 3
2020-12-12T14:20:00-08:00 - 2020-12-12T14:25:00-08:00
Contributed Talk 4: Strategic Recourse in Linear Classification
Yatong Chen, Yang Liu
2020-12-12T14:25:00-08:00 - 2020-12-12T14:30:00-08:00
Contributed Talk 5: Performative Prediction in a Stateful World
Shlomi Hod
2020-12-12T14:30:00-08:00 - 2020-12-12T14:35:00-08:00
Contributed Talk 6: Do Offline Metrics Predict Online Performance in Recommender Systems?
Karl Krauth, Sarah Dean, Wenshuo Guo, Benjamin Recht, Michael Jordan
2020-12-12T14:35:00-08:00 - 2020-12-12T14:45:00-08:00
Q&A for contributed talks 4, 5, 6
2020-12-12T14:45:00-08:00 - 2020-12-12T15:45:00-08:00
Poster Session 2
2020-12-12T15:45:00-08:00 - 2020-12-12T15:50:00-08:00
Wrap up