Workshop
Crowd Science Workshop: Remoteness, Fairness, and Mechanisms as Challenges of Data Supply by Humans for Automation
Daria Baidakova · Fabio Casati · Alexey Drutsa · Dmitry Ustalov
Fri 11 Dec, 8 a.m. PST
Despite the obvious advantages, automation driven by machine learning and artificial intelligence carries pitfalls for the lives of millions of people: disappearance of many well-established mass professions and consumption of labeled data that are produced by humans managed by out of time approach with full-time office work and pre-planned task types. Crowdsourcing methodology can be considered as an effective way to overcome these issues since it provides freedom for task executors in terms of place, time and which task type they want to work on. However, many potential participants of crowdsourcing processes hesitate to use this technology due to a series of doubts (that have not been removed during the past decade).
This workshop brings together people studying research questions on
(a) quality and effectiveness in remote crowd work;
(b) fairness and quality of life at work, tackling issues such as fair task assignment, fair work conditions, and on providing opportunities for growth; and
(c) economic mechanisms that incentivize quality and effectiveness for requester while maintaining a high level of quality and fairness for crowd performers (also known as workers).
Because quality, fairness and opportunities for crowd workers are central to our workshop, we will invite a diverse group of crowd workers from a global public crowdsourcing platform to our panel-led discussion.
Workshop web site: [ protected link dropped ]
Schedule
Fri 8:00 a.m. - 8:15 a.m.
|
Introduction & Icebreakers
(
Live Intro
)
>
|
🔗 |
Fri 8:15 a.m. - 8:35 a.m.
|
Data Excellence: Better Data for Better AI (by Lora Aroyo)
(
Invited Talk
)
>
SlidesLive Video |
Lora Aroyo 🔗 |
Fri 8:35 a.m. - 8:45 a.m.
|
Q&A with Lora Aroyo "Data Excellence: Better Data for Better AI "
(
Live Q&A
)
>
|
🔗 |
Fri 8:45 a.m. - 9:00 a.m.
|
A Gamified Crowdsourcing Framework for Data-Driven Co-creation of Policy Making and Social Foresight (by Andrea Tocchetti and Marco Brambilla)
(
Talk
)
>
SlidesLive Video |
Andrea Tocchetti 🔗 |
Fri 9:00 a.m. - 9:05 a.m.
|
Q&A with Andrea Tocchetti and Marco Brambilla "A Gamified Crowdsourcing Framework for Data-Driven Co-creation of Policy Making and Social Foresight"
(
Live Q&A
)
>
|
🔗 |
Fri 9:05 a.m. - 9:20 a.m.
|
Conversational Crowdsourcing (by Sihang Qiu, Ujwal Gadiraju, Alessandro Bozzon and Geert-Jan Houben)
(
Talk
)
>
SlidesLive Video |
Ujwal Gadiraju · Alessandro Bozzon 🔗 |
Fri 9:20 a.m. - 9:25 a.m.
|
Q&A with Sihang Qiu, Ujwal Gadiraju, Alessandro Bozzon and Geert-Jan Houben "Conversational Crowdsourcing"
(
Live Q&A
)
>
|
🔗 |
Fri 9:25 a.m. - 9:35 a.m.
|
Coffee Break
(
Coffee Break
)
>
|
🔗 |
Fri 9:35 a.m. - 9:55 a.m.
|
Quality Control in Crowdsourcing (by Seid Muhie Yimam)
(
Invited Talk
)
>
SlidesLive Video |
Seid Muhie Yimam 🔗 |
Fri 9:55 a.m. - 10:05 a.m.
|
Q&A with Seid Muhie Yimam "Quality Control in Crowdsourcing"
(
Live Q&A
)
>
|
🔗 |
Fri 10:05 a.m. - 10:20 a.m.
|
What Can Crowd Computing Do for the Next Generation of AI Technology? (by Ujwal Gadiraju and Jie Yang)
(
Talk
)
>
SlidesLive Video |
Ujwal Gadiraju · 🔗 |
Fri 10:20 a.m. - 10:25 a.m.
|
Q&A with Ujwal Gadiraju and Jie Yang "What Can Crowd Computing Do for the Next Generation of AI Technology?"
(
Live Q&A
)
>
|
🔗 |
Fri 10:25 a.m. - 10:40 a.m.
|
Real-Time Crowdsourcing of Health Data in a Low-Income country: A case study of Human Data Supply on Malaria first-line treatment policy tracking in Nigeria (by Olubayo Adekanmbi, Wuraola Fisayo Oyewusi and Ezekiel Ogundepo)
(
Talk
)
>
SlidesLive Video |
Olubayo Adekanmbi · Wuraola Oyewusi 🔗 |
Fri 10:40 a.m. - 10:45 a.m.
|
Q&A with Olubayo Adekanmbi, Wuraola Fisayo Oyewusi and Ezekiel Ogundepo: "Real-Time Crowdsourcing of Health Data in a Low-Income country: A case study of Human Data Supply on Malaria first-line treatment policy tracking in Nigeria"
(
Live Q&A
)
>
|
🔗 |
Fri 10:45 a.m. - 11:00 a.m.
|
Coffee Break
(
Coffee Break
)
>
|
🔗 |
Fri 11:00 a.m. - 12:30 p.m.
|
Panel Discussion "Successes and failures in crowdsourcing: experiences from work providers, performers and platforms"
(
Panel Discussion
)
>
|
🔗 |
Fri 12:30 p.m. - 1:00 p.m.
|
Lunch Break
(
Lunch Break
)
>
|
🔗 |
Fri 1:00 p.m. - 1:20 p.m.
|
Modeling and Aggregation of Complex Annotations Via Annotation Distance (by Matt Lease)
(
Invited Talk
)
>
SlidesLive Video |
Matt Lease 🔗 |
Fri 1:20 p.m. - 1:30 p.m.
|
Q&A with Matt Lease: "Modeling and Aggregation of Complex Annotations Via Annotation Distance"
(
Live Q&A
)
>
|
🔗 |
Fri 1:30 p.m. - 1:45 p.m.
|
Active Learning from Crowd in Item Screening (by Evgeny Krivosheev, Burcu Sayin, Alessandro Bozzon and Zoltán Szlávik)
(
Talk
)
>
SlidesLive Video |
Evgeny Krivosheev · Burcu Sayin Günel · Alessandro Bozzon · Zoltan Szlavik 🔗 |
Fri 1:45 p.m. - 1:50 p.m.
|
Q&A with Evgeny Krivosheev, Burcu Sayin, Alessandro Bozzon and Zoltán Szlávik: "Active Learning from Crowd in Item Screening"
(
Live Q&A
)
>
|
🔗 |
Fri 1:50 p.m. - 2:05 p.m.
|
Human Computation Requires and Enables a New Approach to Ethics (by Libuse Veprek, Patricia Seymour and Pietro Michelucci)
(
Talk
)
>
SlidesLive Video |
Libuše Vepřek · · Pietro Michelucci 🔗 |
Fri 2:05 p.m. - 2:10 p.m.
|
Q&A with Libuse Veprek, Patricia Seymour and Pietro Michelucci: "Human computation requires and enables a new approach to ethics"
(
Live Q&A
)
>
|
🔗 |
Fri 2:10 p.m. - 2:20 p.m.
|
Coffee Break
(
Coffee Break
)
>
|
🔗 |
Fri 2:20 p.m. - 2:40 p.m.
|
Bias in Human-in-the-Loop Artificial Intelligence (by Gianluca Demartini)
(
Invited Talk
)
>
SlidesLive Video |
Gianluca Demartini 🔗 |
Fri 2:40 p.m. - 2:50 p.m.
|
Q&A with Gianluca Demartini: "Bias in Human-in-the-loop Artificial Intelligence"
(
Live Q&A
)
>
|
🔗 |
Fri 2:50 p.m. - 3:05 p.m.
|
VAIDA: An Educative Benchmark Creation Paradigm using Visual Analytics for Interactively Discouraging Artifacts (by Anjana Arunkumar, Swaroop Mishra, Bhavdeep Sachdeva, Chitta Baral and Chris Bryan)
(
Talk
)
>
SlidesLive Video |
Anjana Arunkumar · Swaroop Mishra · Chitta Baral 🔗 |
Fri 3:05 p.m. - 3:10 p.m.
|
Q&A with Anjana Arunkumar, Swaroop Mishra, Bhavdeep Sachdeva, Chitta Baral and Chris Bryan: " VAIDA: An Educative Benchmark Creation Paradigm using Visual Analytics for Interactively Discouraging Artifacts"
(
Live Q&A
)
>
|
🔗 |
Fri 3:10 p.m. - 3:30 p.m.
|
Achieving Data Excellence (by Praveen Paritosh)
(
Invited Talk
)
>
|
Praveen Paritosh 🔗 |
Fri 3:30 p.m. - 3:40 p.m.
|
Q&A with Praveen Paritosh "Achieving Data Excellence"
(
Live Q&A
)
>
|
🔗 |
Fri 3:40 p.m. - 4:00 p.m.
|
Closing
(
Closing
)
>
|
🔗 |