Workshop
Human in the loop dialogue systems
Behnam Hedayatnia · Rahul Goel · Shereen Oraby · Abigail See · Chandra Khatri · Y-Lan Boureau · Alborz Geramifard · Marilyn Walker · Dilek Hakkani-Tur
Fri 11 Dec, 6:10 a.m. PST
Conversational interaction systems such as Amazon Alexa, Google Assistant, Apple Siri, and Microsoft Cortana have become very popular over the recent years. Such systems have allowed users to interact with a wide variety of content on the web through a conversational interface. Research challenges such as the Dialogue System Technology Challenges, Dialogue Dodecathlon, Amazon Alexa Prize and the Vision and Language Navigation task have continued to inspire research in conversational AI. These challenges have brought together researchers from different communities such as speech recognition, spoken language understanding, reinforcement learning, language generation, and multi-modal question answering.
Unlike other popular NLP tasks, dialogue frequently has humans in the loop, whether it is for evaluation, active learning or online reward estimation. Through this workshop we aim to bring together researchers from academia and industry to discuss the challenges and opportunities in such human in the loop setups. We hope that this sparks interesting discussions about conversational agents, interactive systems, and how we can use humans most effectively when building such setups. We will highlight areas such as human evaluation setups, reliability in human evaluation, human in the loop training, interactive learning and user modeling. We also highly encourage non-English based dialogue systems in these areas.
The one-day workshop will include talks from senior technical leaders and researchers to share insights associated with evaluating dialogue systems. We also plan on having oral presentations and poster sessions on works related to the topic of the workshop. Finally we will end the workshop with an interactive panel of speakers. As an outcome we expect the participants from the NeurIPS community to walk away with better understanding of human in the loop dialogue modeling as well as key areas of research in this field. Additionally we would like to see discussions around the unification of human evaluation setups in some way.
This workshop will consist of live QA sessions. Therefore, in order to get the most out of the workshop, it is recommended that you watch all the prerecorded talks before the workshop day. Additionally we have put Reserved blocks of time as an opportunity to watch the pre-recorded talks before the Q/A.
Schedule
Fri 6:10 a.m. - 6:20 a.m.
|
Welcome and Opening Remarks ( Welcome and Opening Remarks ) > link | Behnam Hedayatnia 🔗 |
Fri 6:20 a.m. - 7:50 a.m.
|
Reserved Block of time to watch Pre-recorded Invited Talks 1, 2, 3
(
Reserved Block
)
>
|
🔗 |
Fri 7:50 a.m. - 8:05 a.m.
|
Invited Talk 1 Q/A - Milica Gašić ( Invited Talk ) > link | Milica Gasic 🔗 |
Fri 8:05 a.m. - 8:20 a.m.
|
Invited Talk 2 Q/A - Larry Heck ( Invited Talk ) > link | Larry Heck 🔗 |
Fri 8:20 a.m. - 8:35 a.m.
|
Invited Talk 3 Q/A - Maxine Eskenazi ( Invited Talk ) > link | Maxine Eskenazi · Shikib Mehri 🔗 |
Fri 8:35 a.m. - 9:05 a.m.
|
Reserved Block of time to watch Pre-recorded Contributed Talks 1, 2
(
Reserved Block
)
>
|
🔗 |
Fri 9:05 a.m. - 9:15 a.m.
|
Contributed Talk 1 Q/A ( Contributed Talk ) > link | Jiun-Hao Jhan 🔗 |
Fri 9:15 a.m. - 9:25 a.m.
|
Contributed Talk 2 Q/A ( Contributed Talk ) > link | Thibault Cordier 🔗 |
Fri 9:25 a.m. - 10:30 a.m.
|
Poster Session Presentations ( Poster Session ) > link |
11 presentersNalin Chhibber · Weiyi Lu · Lina Rojas-Barahona · Katherine Stasaski · Dookun Park · Govindarajan Thattai · Alexandry Augustin · Mathilde Veron · Sahisnu Mazumder · Evgeny Krivosheev · Alessandro Bozzon |
Fri 10:30 a.m. - 11:00 a.m.
|
Breakout session: Human Evaluation ( Breakout Session ) > link | Behnam Hedayatnia 🔗 |
Fri 10:30 a.m. - 11:00 a.m.
|
Breakout session: Automatic Evaluation ( Breakout Session ) > link | Yang Liu 🔗 |
Fri 11:00 a.m. - 11:20 a.m.
|
Reserved Block of time to watch Pre-recorded Contributed Talks 3, 4
(
Reserved Block
)
>
|
🔗 |
Fri 11:20 a.m. - 11:30 a.m.
|
Contributed Talk 3 Q/A ( Contributed Talk ) > link | José David Águas Lopes 🔗 |
Fri 11:30 a.m. - 11:40 a.m.
|
Contributed Talk 4 Q/A ( Contributed Talk ) > link | Qiuyuan Huang · Kezhen Chen 🔗 |
Fri 11:40 a.m. - 12:50 p.m.
|
Reserved Block of time to watch Pre-recorded Invited Talks 4, 5, 6
(
Reserved Block
)
>
|
🔗 |
Fri 12:50 p.m. - 1:05 p.m.
|
Invited Talk 4 Q/A - Jason Weston ( Invited Talk ) > link | Jason E Weston 🔗 |
Fri 1:05 p.m. - 1:20 p.m.
|
Invited Talk 5 Q/A - Zhou Yu ( Invited Talk ) > link | Zhou Yu 🔗 |
Fri 1:20 p.m. - 1:35 p.m.
|
Invited Talk 6 Q/A - Gokhan Tür ( Invited Talk ) > link | Gokhan Tur 🔗 |
Fri 1:35 p.m. - 1:55 p.m.
|
Reserved Block of time to watch Pre-recorded Contributed Talks 5, 6
(
Reserved Block
)
>
|
🔗 |
Fri 1:55 p.m. - 2:05 p.m.
|
Contributed Talk 5 Q/A ( Contributed Talk ) > link | Nathan Ng 🔗 |
Fri 2:05 p.m. - 2:20 p.m.
|
Contributed Talk 6 Q/A ( Contributed Talk ) > link | Chien-Wei Lin 🔗 |
Fri 2:20 p.m. - 3:30 p.m.
|
Reserved Block of time to watch Pre-recorded Invited Talks 7, 8, 9
(
Reserved Block
)
>
|
🔗 |
Fri 3:30 p.m. - 3:45 p.m.
|
Invited Talk 7 Q/A - Ankur Parikh ( Invited Talk ) > link | Ankur Parikh 🔗 |
Fri 3:45 p.m. - 4:00 p.m.
|
Invited Talk 8 Q/A - Percy Liang ( Invited Talk ) > link | Percy Liang 🔗 |
Fri 4:00 p.m. - 4:15 p.m.
|
Invited Talk 9 Q/A - Alexander Rudnicky ( Invited Talk ) > link | Alexander Rudnicky 🔗 |
Fri 4:15 p.m. - 5:15 p.m.
|
Panel ( Panel ) > link | Maxine Eskenazi · Ankur Parikh · Govindarajan Thattai · Alexander Rudnicky · Jason E Weston 🔗 |
Fri 5:15 p.m. - 5:20 p.m.
|
Closing Remarks / Best Paper Award ( Closing Remarks ) > link | Behnam Hedayatnia 🔗 |
-
|
Invited Talk 1 Presentation - Milica Gašić - On the track of multi-domain dialogue models
(
Invited Talk Presentation
)
>
SlidesLive Video |
Milica Gasic 🔗 |
-
|
Invited Talk 2 Presentation - Larry Heck - Master-Apprentice Learning
(
Invited Talk Presentation
)
>
SlidesLive Video |
Larry Heck 🔗 |
-
|
Invited Talk 3 Presenation - Maxine Eskenazi - Human > User in the Loop
(
Invited Talk Presentation
)
>
SlidesLive Video |
Maxine Eskenazi 🔗 |
-
|
Invited Talk 4 Presentation - Jason Weston - (Towards) Learning from Conversing
(
Invited Talk Presentation
)
>
SlidesLive Video |
Jason E Weston 🔗 |
-
|
Invited Talk 5 Presentation - Zhou Yu - Augment Intelligence with Multimodal Information
(
Invited Talk Presentation
)
>
SlidesLive Video |
Zhou Yu 🔗 |
-
|
Invited Talk 6 Presentation - Gokhan Tür - Past, Present, Future of Conversational AI
(
Invited Talk Presentation
)
>
SlidesLive Video |
Gokhan Tur 🔗 |
-
|
Invited Talk 7 Presentation - Ankur Parikh - Towards High Precision Text Generation
(
Invited Talk Presentation
)
>
SlidesLive Video |
Ankur Parikh 🔗 |
-
|
Invited Talk 8 Presentation - Percy Liang - Semantic Parsing for Natural Language Interfaces
(
Invited Talk Presentation
)
>
SlidesLive Video |
Percy Liang 🔗 |
-
|
Invited Talk 9 Presentation - Alexander Rudnicky - Creating socialbots with human-like conversational abilities
(
Invited Talk Presentation
)
>
SlidesLive Video |
Alexander Rudnicky 🔗 |
-
|
Contributed Talk 1 Presentation - "CheerBots: Chatbots toward Empathy and Emotion using Reinforcement Learning"
(
Contributed Talk Presentation
)
>
SlidesLive Video |
Jiun-Hao Jhan 🔗 |
-
|
Contributed Talk 2 Presentation - "Diluted Near-Optimal Expert Demonstrations for Guiding Dialogue Stochastic Policy Optimisation"
(
Contributed Talk Presentation
)
>
SlidesLive Video |
Thibault Cordier 🔗 |
-
|
Contributed Talk 3 Presentation - "The Lab vs The Crowd: An Investigation into Data Quality for Neural Dialogue Models"
(
Contributed Talk Presentation
)
>
SlidesLive Video |
José David Águas Lopes 🔗 |
-
|
Contributed Talk 4 Presentation - "NICE: Neural Image Commenting Evaluation with an Emphasis on Emotion and Empathy"
(
Contributed Talk Presentation
)
>
SlidesLive Video |
Qiuyuan Huang 🔗 |
-
|
Contributed Talk 5 Presentation - "Improving Dialogue Breakdown Detection with Semi-Supervised Learning"
(
Contributed Talk Presentation
)
>
SlidesLive Video |
Nathan Ng 🔗 |
-
|
Contributed Talk 6 Presentation - "Dialog Simulation with Realistic Variations for Training Goal-Oriented Conversational Systems"
(
Contributed Talk Presentation
)
>
SlidesLive Video |
Chien-Wei Lin 🔗 |