Workshop
HAMLETS: Human And Model in the Loop Evaluation and Training Strategies
Divyansh Kaushik · Bhargavi Paranjape · Forough Arabshahi · Yanai Elazar · Yixin Nie · Max Bartolo · Polina Kirichenko · Pontus Lars Erik Saito Stenetorp · Mohit Bansal · Zachary Lipton · Douwe Kiela
Sat 12 Dec, 8:15 a.m. PST
Human involvement in AI system design, development, and evaluation is critical to ensure that the insights being derived are practical, and the systems built are meaningful, reliable, and relatable to those who need them. Humans play an integral role in all stages of machine learning development, be it during data generation, interactively teaching machines, or interpreting, evaluating and debugging models. With growing interest in such “human in the loop” learning, we aim to highlight new and emerging research opportunities for the ML community that arise from the evolving needs to design evaluation and training strategies for humans and models in the loop. The specific focus of this workshop is on emerging and under-explored areas of human- and model-in-the-loop learning, such as employing humans to seek richer forms of feedback for data than labels alone, learning from dynamic adversarial data collection with humans employed to find weaknesses in models, learning from human teachers instructing computers through conversation and/or demonstration, investigating the role of humans in model interpretability, and assessing social impact of ML systems. This workshop aims to bring together interdisciplinary researchers from academia and industry to discuss major challenges, outline recent advances, and facilitate future research in these areas.
Schedule
Sat 8:15 a.m. - 8:30 a.m.
|
Opening Remarks
(
Opening Remarks
)
>
|
Divyansh Kaushik · Bhargavi Paranjape · Douwe Kiela 🔗 |
Sat 8:30 a.m. - 8:55 a.m.
|
Invited Talk by Tom Mitchell (Carnegie Mellon)
(
Talk
)
>
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 8:55 a.m. - 9:00 a.m.
|
Break
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 9:00 a.m. - 9:25 a.m.
|
Invited Talk by Jenn Wortman Vaughan (Microsoft Research)
(
Talk
)
>
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 9:25 a.m. - 9:30 a.m.
|
Break
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 9:30 a.m. - 9:55 a.m.
|
Invited Talk by Sanjoy Dasgupta (UCSD)
(
Talk
)
>
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 9:55 a.m. - 10:00 a.m.
|
Break
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 10:00 a.m. - 10:25 a.m.
|
Invited Talk by Finale Doshi-Velez (Harvard)
(
Talk
)
>
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 10:25 a.m. - 11:30 a.m.
|
Q & A and Panel Session with Tom Mitchell, Jenn Wortman Vaughan, Sanjoy Dasgupta, and Finale Doshi-Velez
(
Discussion Panel
)
>
|
Tom Mitchell · Jennifer Wortman Vaughan · Sanjoy Dasgupta · Finale Doshi-Velez · Zachary Lipton 🔗 |
Sat 11:30 a.m. - 12:45 p.m.
|
Poster Session 1 ( GatherTown Posters ) > link | 🔗 |
Sat 12:45 p.m. - 1:55 p.m.
|
Break (Or meet on GatherTown) link | 🔗 |
Sat 1:55 p.m. - 2:00 p.m.
|
Break
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 2:00 p.m. - 2:25 p.m.
|
Invited Talk by Dan Weld (University of Washington)
(
Talk
)
>
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 2:25 p.m. - 2:30 p.m.
|
Break
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 2:30 p.m. - 2:55 p.m.
|
Invited Talk by Kristen Grauman (UT Austin, FAIR)
(
Talk
)
>
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 2:55 p.m. - 3:00 p.m.
|
Break
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 3:00 p.m. - 3:25 p.m.
|
Invited Talk by Scott Yih (FAIR)
(
Talk
)
>
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 3:25 p.m. - 3:30 p.m.
|
Break
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 3:30 p.m. - 3:55 p.m.
|
Invited Talk by Emma Brunskill (Stanford)
(
Talk
)
>
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 3:55 p.m. - 4:00 p.m.
|
Break
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 4:00 p.m. - 4:25 p.m.
|
Invited Talk by Alex Ratner (Snorkel, University of Washington)
(
Talk
)
>
SlidesLive Video |
Divyansh Kaushik 🔗 |
Sat 4:25 p.m. - 5:30 p.m.
|
Q & A and Panel Session with Dan Weld, Kristen Grauman, Scott Yih, Emma Brunskill, and Alex Ratner
(
Discussion Panel
)
>
|
Kristen Grauman · Wen-tau Yih · Alexander Ratner · Emma Brunskill · Douwe Kiela · Daniel S. Weld 🔗 |
Sat 5:30 p.m. - 6:45 p.m.
|
Poster Session 2 ( GatherTown Posters ) > link | 🔗 |
Sat 6:45 p.m. - 7:00 p.m.
|
Closing Remarks
(
Closing
)
>
|
🔗 |
Sat 7:00 p.m. - 8:00 p.m.
|
Social/Get-Together on GatherTown ( GatherTown ) > link | 🔗 |