Workshop
Workshop on Distribution Shifts: New Frontiers with Foundation Models
Rebecca Roelofs · Fanny Yang · Hongseok Namkoong · Masashi Sugiyama · Jacob Eisenstein · Pang Wei Koh · Shiori Sagawa · Tatsunori Hashimoto · Yoonho Lee
Room R06-R09 (level 2)
Fri 15 Dec, 7 a.m. PST
Tagline: This workshop focuses on distribution shifts in the context of foundation models.Distribution shifts---where a model is deployed on a data distribution different from what it was trained on---pose significant robustness challenges in real-world ML applications. Such shifts are often unavoidable in the wild and have been shown to substantially degrade model performance in a wide range of applications. For example, models can systematically fail when tested on patients from different hospitals or people from different demographics. Training models that are robust to such distribution shifts is a rapidly growing area of interest in the ML community, and the goal of our workshop is to foster discussions and further research on distribution shifts. In the context of distribution shifts, our workshop this year focuses on foundation models: large pretrained models that can be adapted for a wide range of tasks. Foundation models open up an exciting new frontier in the study of distribution shifts, raising open research questions such as how pre-training improves robustness, how to finetune foundation models for increased robustness, how to leverage foundation models’ generative capabilities for robustness, and how to handle discrepancies between standard pre-training distributions and downstream distributions of interest. We aim to facilitate discussions around these topics by bringing together researchers working on distribution shifts and foundation models.
Schedule
Fri 7:00 a.m. - 7:10 a.m.
|
Opening Remarks
(
Opening Remarks
)
>
|
🔗 |
Fri 7:10 a.m. - 7:35 a.m.
|
Invited Talk 1
(
Invited Talk
)
>
SlidesLive Video |
Peng Cui 🔗 |
Fri 7:35 a.m. - 8:00 a.m.
|
Invited Talk 2
(
Invited Talk
)
>
SlidesLive Video |
Kate Saenko 🔗 |
Fri 8:00 a.m. - 8:30 a.m.
|
Coffee Break
|
🔗 |
Fri 8:30 a.m. - 10:00 a.m.
|
Poster Session
(
Poster Session
)
>
|
🔗 |
Fri 10:00 a.m. - 11:15 a.m.
|
Lunch Break
|
🔗 |
Fri 11:15 a.m. - 11:25 a.m.
|
TiC-CLIP: Continual Training of CLIP Models
(
Oral
)
>
link
SlidesLive Video |
Saurabh Garg · Mehrdad Farajtabar · Hadi Pouransari · Raviteja Vemulapalli · Sachin Mehta · Oncel Tuzel · Vaishaal Shankar · Fartash Faghri 🔗 |
Fri 11:25 a.m. - 11:35 a.m.
|
LLM Routing with Benchmark Datasets
(
Oral
)
>
link
SlidesLive Video |
Tal Shnitzer · Anthony Ou · Mírian Silva · Kate Soule · Yuekai Sun · Justin Solomon · Neil Thompson · Mikhail Yurochkin 🔗 |
Fri 11:35 a.m. - 11:45 a.m.
|
Does CLIP’s generalization performance mainly stem from high train-test similarity?
(
Oral
)
>
link
SlidesLive Video |
Prasanna Mayilvahanan · Thaddäus Wiedemer · Evgenia Rusak · Matthias Bethge · Wieland Brendel 🔗 |
Fri 11:45 a.m. - 11:55 a.m.
|
Domain constraints improve risk prediction when outcome data is missing
(
Oral
)
>
link
SlidesLive Video |
Sidhika Balachandar · Nikhil Garg · Emma Pierson 🔗 |
Fri 11:55 a.m. - 12:05 p.m.
|
OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection
(
Oral
)
>
link
SlidesLive Video |
14 presentersJingyang Zhang · Jingkang Yang · Pengyun Wang · Haoqi Wang · Yueqian Lin · Haoran Zhang · Yiyou Sun · Xuefeng Du · Kaiyang Zhou · Wayne Zhang · Yixuan Li · Ziwei Liu · Yiran Chen · Hai Li |
Fri 12:05 p.m. - 12:15 p.m.
|
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
(
Oral
)
>
link
SlidesLive Video |
Sewon Min · Suchin Gururangan · Eric Wallace · Weijia Shi · Hannaneh Hajishirzi · Noah Smith · Luke Zettlemoyer 🔗 |
Fri 12:15 p.m. - 12:40 p.m.
|
Invited Talk 3
(
Invited Talk
)
>
SlidesLive Video |
Aditi Raghunathan 🔗 |
Fri 12:40 p.m. - 1:05 p.m.
|
Invited Talk 4
(
Invited Talk
)
>
SlidesLive Video |
Hoifung Poon 🔗 |
Fri 1:05 p.m. - 1:30 p.m.
|
Coffee Break
|
🔗 |
Fri 1:30 p.m. - 2:00 p.m.
|
Invited Talk 5
(
Invited Talk
)
>
SlidesLive Video |
Ludwig Schmidt 🔗 |
Fri 2:00 p.m. - 2:50 p.m.
|
Panel Dicsussion
(
Panel Discussion
)
>
SlidesLive Video |
🔗 |
Fri 2:50 p.m. - 3:00 p.m.
|
Closing Remarks
(
Closing Remarks
)
>
|
🔗 |
-
|
The SVHN Dataset Is Deceptive for Probabilistic Generative Models Due to a Distribution Mismatch
(
Poster
)
>
link
SlidesLive Video |
Tim Xiao · Johannes Zenn · Robert Bamler 🔗 |
-
|
Understanding Catastrophic Forgetting in Language Models via Implicit Inference ( Poster ) > link | Suhas Kotha · Jacob Springer · Aditi Raghunathan 🔗 |
-
|
Predicting the Performance of Foundation Models via Agreement-on-the-Line ( Poster ) > link | Rahul Saxena · Aman Mehra · Taeyoun Kim · Christina Baek · J. Zico Kolter · Aditi Raghunathan 🔗 |
-
|
Can Transformer Models Generalize Via In-Context Learning Beyond Pretraining Data? ( Poster ) > link | Steve Yadlowsky · Lyric Doshi · Nilesh Tripuraneni 🔗 |
-
|
Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
(
Poster
)
>
link
SlidesLive Video |
Xiaohan Hu · Yi Ma · Chenjun Xiao · YAN ZHENG · Jianye Hao 🔗 |
-
|
Probing the Equivariance of Image Embeddings ( Poster ) > link | Cyrus Rashtchian · Charles Herrmann · Chun-Sung Ferng · Ayan Chakrabarti · Dilip Krishnan · Deqing Sun · Da-Cheng Juan · Andrew Tomkins 🔗 |
-
|
Exploring Generalisability of Self-Distillation with No Labels for SAR-Based Vegetation Prediction ( Poster ) > link | Laura Martínez-Ferrer · Anna Jungbluth · Joseph Alejandro Gallego Mejia · Matthew Allen · Francisco Dorr · Freddie Kalaitzis · Raul Ramos-Pollán 🔗 |
-
|
AutoFT: Robust Fine-Tuning by Optimizing Hyperparameters on OOD Data ( Poster ) > link | Caroline Choi · Yoonho Lee · Annie Chen · Allan Zhou · Aditi Raghunathan · Chelsea Finn 🔗 |
-
|
Turn Down the Noise: Leveraging Diffusion Models for Test-time Adaptation via Pseudo-label Ensembling ( Poster ) > link | Mrigank Raman · Rohan Shah · Akash Kannan · Pranit Chawla 🔗 |
-
|
Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models
(
Poster
)
>
link
SlidesLive Video |
Amal Rannen-Triki · Jorg Bornschein · Razvan Pascanu · Alexandre Galashov · Michalis Titsias · Marcus Hutter · András György · Yee Whye Teh 🔗 |
-
|
Tackling Concept Shift in Text Classification using Entailment-style modeling
(
Poster
)
>
link
SlidesLive Video |
Sumegh Roychowdhury · Siva Rajesh Kasa · Karan Gupta · Prasanna Srinivasa Murthy · Alok Chandra 🔗 |
-
|
Reliable Test-Time Adaptation via Agreement-on-the-Line ( Poster ) > link | Eungyeup Kim · Mingjie Sun · Aditi Raghunathan · J. Zico Kolter 🔗 |
-
|
Reward Model Underspecification in Language Model Alignment ( Poster ) > link | Jacob Eisenstein · Jonathan Berant · Chirag Nagpal · Alekh Agarwal · Ahmad Beirami · Alexander D'Amour · Krishnamurthy Dvijotham · Katherine Heller · Stephen Pfohl · Deepak Ramachandran 🔗 |
-
|
Learning Causally-Aware Representations of Multi-Agent Interactions ( Poster ) > link | Yuejiang Liu · Ahmad Rahimi · Po-Chien Luan · Frano Rajič · Alexandre Alahi 🔗 |
-
|
Fusing Models with Complementary Expertise ( Poster ) > link | Hongyi Wang · Felipe Maia Polo · Yuekai Sun · Souvik Kundu · Eric Xing · Mikhail Yurochkin 🔗 |
-
|
On Mitigating Shortcut Learning for Fair Chest X-ray Classification under Distribution Shift ( Poster ) > link | Yuzhe Yang · Haoran Zhang · Dina Katabi · Marzyeh Ghassemi 🔗 |
-
|
Are all classes created equal? Domain Generalization for Domain-Linked Classes ( Poster ) > link | Kimathi Kaai · Saad Hossain · Sirisha Rambhatla 🔗 |
-
|
Discovering environments with XRM ( Poster ) > link | Mohammad Pezeshki · Diane Bouchacourt · Mark Ibrahim · Nicolas Ballas · Pascal Vincent · David Lopez-Paz 🔗 |
-
|
Data Filtering Networks ( Poster ) > link | Alex Fang · Albin Madappally Jose · Amit Jain · Ludwig Schmidt · Alexander Toshev · Vaishaal Shankar 🔗 |
-
|
Skill-Mix: A Flexible and Expandable Family of Evaluations for AI Models ( Poster ) > link | Dingli Yu · Simran Kaur · Arushi Gupta · Jonah Brown-Cohen · Anirudh Goyal · Sanjeev Arora 🔗 |
-
|
Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Domain Adaptation ( Poster ) > link | Dapeng Hu · Jian Liang · Xinchao Wang · Chuan Sheng Foo 🔗 |
-
|
Ask Your Shift if Pre-Training is Right for You ( Poster ) > link | Benjamin Cohen-Wang · Joshua Vendrow · Aleksander Madry 🔗 |
-
|
Maximum Likelihood Estimation is All You Need for Well-Specified Covariate Shift ( Poster ) > link | Jiawei Ge · Shange Tang · Jianqing Fan · Cong Ma · Chi Jin 🔗 |
-
|
Simplifying and Stabilizing Model Selection in Unsupervised Domain Adaptation ( Poster ) > link | Dapeng Hu · Romy Luo · Jian Liang · Chuan Sheng Foo 🔗 |
-
|
Context-Aware Meta-Learning ( Poster ) > link | Christopher Fifty · Dennis Duan · Ronald Junkins · Ehsan Amid · Jure Leskovec · Christopher Ré · Sebastian Thrun 🔗 |
-
|
Transfer Learning, Reinforcement Learning for Adaptive Control Optimization under Distribution Shift
(
Poster
)
>
link
SlidesLive Video |
Pankaj Rajak · Wojciech Kowalinski · Fei Wang 🔗 |
-
|
Context is Environment ( Poster ) > link | Sharut Gupta · David Lopez-Paz · Stefanie Jegelka · Kartik Ahuja 🔗 |
-
|
HePCo: Data-Free Heterogeneous Prompt Consolidation for Continual Federated Learning ( Poster ) > link | Shaunak Halbe · James S Smith · Junjiao Tian · Zsolt Kira 🔗 |
-
|
A Nearest Neighbor-Based Concept Drift Detection Strategy for Reliable Condition Monitoring
(
Poster
)
>
link
SlidesLive Video |
Nicolas Jourdan 🔗 |
-
|
Improving Domain Generalization in Contrastive Learning via Domain-Aware Temperature Control ( Poster ) > link | Robert Lewis · Katie Matton · Rosalind Picard · John Guttag 🔗 |
-
|
Stochastic linear dynamics in parameters to deal with Neural Networks plasticity loss ( Poster ) > link | Alexandre Galashov · Michalis Titsias · Razvan Pascanu · Yee Whye Teh · Maneesh Sahani 🔗 |
-
|
Can Transformers In-Context Learn Task Mixtures? ( Poster ) > link | Nilesh Tripuraneni · Lyric Doshi · Steve Yadlowsky 🔗 |
-
|
Evolving Domain Adaptation of Pretrained Language Models for Text Classification
(
Poster
)
>
link
SlidesLive Video |
Yun-Shiuan Chuang · Rheeya Uppaal · Yi Wu · Luhang Sun · Makesh Narsimhan Sreedhar · Sijia Yang · Timothy T Rogers · Junjie Hu 🔗 |
-
|
Robustness May be More Brittle than We Think under Different Degrees of Distribution Shifts ( Poster ) > link | Kaican Li · Yifan Zhang · Lanqing Hong · Zhenguo Li · Nevin L. Zhang 🔗 |
-
|
On selective classification under distribution shift
(
Poster
)
>
link
SlidesLive Video |
Luís Felipe Cattelan · Danilo Silva 🔗 |
-
|
Understanding subgroup performance differences of fair predictors using causal models ( Poster ) > link | Stephen Pfohl · Natalie Harris · Chirag Nagpal · David Madras · Vishwali Mhasawade · Olawale Salaudeen · Katherine Heller · Sanmi Koyejo · Alexander D'Amour 🔗 |
-
|
AutoVP: An Automated Visual Prompting Framework and Benchmark ( Poster ) > link | Hsi-Ai Tsao · Lei Hsiung · Pin-Yu Chen · Sijia Liu · Tsung-Yi Ho 🔗 |
-
|
Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift ( Poster ) > link | Mona Schirmer · Dan Zhang · Eric Nalisnick 🔗 |
-
|
Continual Learning with Low Rank Adaptation
(
Poster
)
>
link
SlidesLive Video |
Martin Wistuba · Prabhu Teja Sivaprasad · Lukas Balles · Giovanni Zappella 🔗 |
-
|
Adaptive Sharpness-Aware Pruning for Robust Sparse Networks ( Poster ) > link | Anna Bair · Hongxu Yin · Maying Shen · Pavlo Molchanov · Jose M. Alvarez 🔗 |
-
|
Bilevel Optimization to Learn Training Distributions for Language Modeling under Domain Shift ( Poster ) > link | David Grangier · Pierre Ablin · Awni Hannun 🔗 |
-
|
Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implications ( Poster ) > link | Jiashuo Liu · Jiayun Wu · Tianyu Wang · Hao Zou · Peng Cui 🔗 |
-
|
Channel Selection for Test-Time Adaptation Under Distribution Shift ( Poster ) > link | Pedro Vianna · Muawiz Chaudhary · An Tang · Guy Cloutier · Guy Wolf · Michael Eickenberg · Eugene Belilovsky 🔗 |
-
|
Better than Balancing: Debiasing through Data Attribution ( Poster ) > link | Saachi Jain · Kimia Hamidieh · Kristian Georgiev · Marzyeh Ghassemi · Aleksander Madry 🔗 |
-
|
Enhancing Robustness of Foundation Model Representations under Provenance-related Distribution Shifts
(
Poster
)
>
link
SlidesLive Video |
Xiruo Ding · Zhecheng Sheng · Brian Hur · Feng Chen · Serguei Pakhomov · Trevor Cohen 🔗 |
-
|
Towards Global, General-Purpose Pretrained Geographic Location Encoders ( Poster ) > link | Konstantin Klemmer · Esther Rolf · Caleb Robinson · Lester Mackey · Marc Rußwurm 🔗 |
-
|
HyperNetwork Approximating Future Parameters for Time Series Forecasting under Temporal Drifts
(
Poster
)
>
link
SlidesLive Video |
Jaehoon Lee · Chan Kim · Gyumin Lee · Haksoo Lim · Jeongwhan Choi · Kookjin Lee · Dongeun Lee · Sanghyun Hong · Noseong Park 🔗 |
-
|
LCA-on-the-Line: Benchmarking Out of Distribution Generalization with Class Taxonomies
(
Poster
)
>
link
SlidesLive Video |
Jia Shi · Gautam Rajendrakumar Gare · Jinjin Tian · Siqi Chai · Zhiqiu Lin · Arun Balajee Vasudevan · Di Feng · Francesco Ferroni · Shu Kong · Deva Ramanan 🔗 |
-
|
Towards General-Purpose In-Context Learning Agents ( Poster ) > link | Louis Kirsch · James Harrison · Daniel Freeman · Jascha Sohl-Dickstein · Jürgen Schmidhuber 🔗 |
-
|
Do Transformers Parse while Predicting the Masked Word? ( Poster ) > link | Haoyu Zhao · Abhishek Panigrahi · Rong Ge · Sanjeev Arora 🔗 |
-
|
Retrieval-based Language Models Using a Multi-domain Datastore ( Poster ) > link | Rulin Shao · Sewon Min · Luke Zettlemoyer · Pang Wei Koh 🔗 |
-
|
Continually Adapting Optimizers Improve Meta-Generalization ( Poster ) > link | Wenyi Wang · Louis Kirsch · Francesco Faccio · Mingchen Zhuge · Jürgen Schmidhuber 🔗 |
-
|
Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentations ( Poster ) > link | Helen Qu · Sang Michael Xie 🔗 |
-
|
Confidence-Based Model Selection: When to Take Shortcuts in Spurious Settings ( Poster ) > link | Annie Chen · Yoonho Lee · Amrith Setlur · Sergey Levine · Chelsea Finn 🔗 |
-
|
Two-stage LLM Fine-tuning with Less Specialization and More Generalization ( Poster ) > link | Yihan Wang · Si Si · Daliang Li · MICHAL LUKASIK · Felix Yu · Cho-Jui Hsieh · Inderjit Dhillon · Sanjiv Kumar 🔗 |
-
|
An Empirical Study of Uncertainty Estimation Techniques for Detecting Drift in Data Streams
(
Poster
)
>
link
SlidesLive Video |
Anton Winter · Nicolas Jourdan · Tristan Wirth · Volker Knauthe · Arjan Kuijper 🔗 |
-
|
Outlier-Robust Group Inference via Gradient Space Clustering ( Poster ) > link | Yuchen Zeng · Kristjan Greenewald · Luann Jung · Kangwook Lee · Justin Solomon · Mikhail Yurochkin 🔗 |
-
|
Towards Calibrated Robust Fine-Tuning of Vision-Language Models
(
Poster
)
>
link
SlidesLive Video |
Changdae Oh · Mijoo Kim · Hyesu Lim · Junhyeok Park · Euiseog Jeong · Zhi-Qi Cheng · Kyungwoo Song 🔗 |