Workshop: Crowd Science Workshop: Remoteness, Fairness, and Mechanisms as Challenges of Data Supply by Humans for Automation
Daria Baidakova, Fabio Casati, Alexey Drutsa, Dmitry Ustalov
2020-12-11T08:00:00-08:00 - 2020-12-11T16:00:00-08:00
Abstract: Despite the obvious advantages, automation driven by machine learning and artificial intelligence carries pitfalls for the lives of millions of people: disappearance of many well-established mass professions and consumption of labeled data that are produced by humans managed by out of time approach with full-time office work and pre-planned task types. Crowdsourcing methodology can be considered as an effective way to overcome these issues since it provides freedom for task executors in terms of place, time and which task type they want to work on. However, many potential participants of crowdsourcing processes hesitate to use this technology due to a series of doubts (that have not been removed during the past decade).
This workshop brings together people studying research questions on
(a) quality and effectiveness in remote crowd work;
(b) fairness and quality of life at work, tackling issues such as fair task assignment, fair work conditions, and on providing opportunities for growth; and
(c) economic mechanisms that incentivize quality and effectiveness for requester while maintaining a high level of quality and fairness for crowd performers (also known as workers).
Because quality, fairness and opportunities for crowd workers are central to our workshop, we will invite a diverse group of crowd workers from a global public crowdsourcing platform to our panel-led discussion.
Workshop web site: https://research.yandex.com/workshops/crowd/neurips-2020
Paper submission portal: https://easychair.org/conferences/?conf=neurips2020crowd
All submissions must be in PDF format. The page limit is up to eight (8) pages maximum for regular papers and four (4) pages for work-in-progress/vision papers. These limits are for main content pages, including all figures and tables. Additional pages containing appendices, acknowledgements, funding disclosures, and references are allowed. You must format your submission using the NeurIPS 2020 LaTeX style file which includes a “preprint” option for non-anonymous preprints posted online. The maximum file size for submissions is 50MB. Submissions that violate the NeurIPS style (e.g., by decreasing margins or font sizes) or page limits may be rejected without further review.
As an author, you are responsible for anonymizing your submission. In particular, you should not include author names, author affiliations, or acknowledgements in your submission and you should avoid providing any other identifying information.
This workshop brings together people studying research questions on
(a) quality and effectiveness in remote crowd work;
(b) fairness and quality of life at work, tackling issues such as fair task assignment, fair work conditions, and on providing opportunities for growth; and
(c) economic mechanisms that incentivize quality and effectiveness for requester while maintaining a high level of quality and fairness for crowd performers (also known as workers).
Because quality, fairness and opportunities for crowd workers are central to our workshop, we will invite a diverse group of crowd workers from a global public crowdsourcing platform to our panel-led discussion.
Workshop web site: https://research.yandex.com/workshops/crowd/neurips-2020
Paper submission portal: https://easychair.org/conferences/?conf=neurips2020crowd
All submissions must be in PDF format. The page limit is up to eight (8) pages maximum for regular papers and four (4) pages for work-in-progress/vision papers. These limits are for main content pages, including all figures and tables. Additional pages containing appendices, acknowledgements, funding disclosures, and references are allowed. You must format your submission using the NeurIPS 2020 LaTeX style file which includes a “preprint” option for non-anonymous preprints posted online. The maximum file size for submissions is 50MB. Submissions that violate the NeurIPS style (e.g., by decreasing margins or font sizes) or page limits may be rejected without further review.
As an author, you are responsible for anonymizing your submission. In particular, you should not include author names, author affiliations, or acknowledgements in your submission and you should avoid providing any other identifying information.
Chat
To ask questions please use rocketchat, available only upon registration and login.
Schedule
2020-12-11T08:00:00-08:00 - 2020-12-11T08:15:00-08:00
Introduction & Icebreakers
2020-12-11T08:15:00-08:00 - 2020-12-11T08:35:00-08:00
Data Excellence: Better Data for Better AI (by Lora Aroyo)
Lora Aroyo
2020-12-11T08:35:00-08:00 - 2020-12-11T08:45:00-08:00
Q&A with Lora Aroyo "Data Excellence: Better Data for Better AI "
2020-12-11T08:45:00-08:00 - 2020-12-11T09:00:00-08:00
A Gamified Crowdsourcing Framework for Data-Driven Co-creation of Policy Making and Social Foresight (by Andrea Tocchetti and Marco Brambilla)
Andrea Tocchetti
2020-12-11T09:00:00-08:00 - 2020-12-11T09:05:00-08:00
Q&A with Andrea Tocchetti and Marco Brambilla "A Gamified Crowdsourcing Framework for Data-Driven Co-creation of Policy Making and Social Foresight"
2020-12-11T09:05:00-08:00 - 2020-12-11T09:20:00-08:00
Conversational Crowdsourcing (by Sihang Qiu, Ujwal Gadiraju, Alessandro Bozzon and Geert-Jan Houben)
Ujwal Gadiraju, Alessandro Bozzon
2020-12-11T09:20:00-08:00 - 2020-12-11T09:25:00-08:00
Q&A with Sihang Qiu, Ujwal Gadiraju, Alessandro Bozzon and Geert-Jan Houben "Conversational Crowdsourcing"
2020-12-11T09:25:00-08:00 - 2020-12-11T09:35:00-08:00
Coffee Break
2020-12-11T09:35:00-08:00 - 2020-12-11T09:55:00-08:00
Quality Control in Crowdsourcing (by Seid Muhie Yimam)
Seid Muhie Yimam
2020-12-11T09:55:00-08:00 - 2020-12-11T10:05:00-08:00
Q&A with Seid Muhie Yimam "Quality Control in Crowdsourcing"
2020-12-11T10:05:00-08:00 - 2020-12-11T10:20:00-08:00
What Can Crowd Computing Do for the Next Generation of AI Technology? (by Ujwal Gadiraju and Jie Yang)
Ujwal Gadiraju,
2020-12-11T10:20:00-08:00 - 2020-12-11T10:25:00-08:00
Q&A with Ujwal Gadiraju and Jie Yang "What Can Crowd Computing Do for the Next Generation of AI Technology?"
2020-12-11T10:25:00-08:00 - 2020-12-11T10:40:00-08:00
Real-Time Crowdsourcing of Health Data in a Low-Income country: A case study of Human Data Supply on Malaria first-line treatment policy tracking in Nigeria (by Olubayo Adekanmbi, Wuraola Fisayo Oyewusi and Ezekiel Ogundepo)
Olubayo Adekanmbi, Wuraola Oyewusi
2020-12-11T10:40:00-08:00 - 2020-12-11T10:45:00-08:00
Q&A with Olubayo Adekanmbi, Wuraola Fisayo Oyewusi and Ezekiel Ogundepo: "Real-Time Crowdsourcing of Health Data in a Low-Income country: A case study of Human Data Supply on Malaria first-line treatment policy tracking in Nigeria"
2020-12-11T10:45:00-08:00 - 2020-12-11T11:00:00-08:00
Coffee Break
2020-12-11T11:00:00-08:00 - 2020-12-11T12:30:00-08:00
Panel Discussion "Successes and failures in crowdsourcing: experiences from work providers, performers and platforms"
2020-12-11T12:30:00-08:00 - 2020-12-11T13:00:00-08:00
Lunch Break
2020-12-11T13:00:00-08:00 - 2020-12-11T13:20:00-08:00
Modeling and Aggregation of Complex Annotations Via Annotation Distance (by Matt Lease)
Matt Lease
2020-12-11T13:20:00-08:00 - 2020-12-11T13:30:00-08:00
Q&A with Matt Lease: "Modeling and Aggregation of Complex Annotations Via Annotation Distance"
2020-12-11T13:30:00-08:00 - 2020-12-11T13:45:00-08:00
Active Learning from Crowd in Item Screening (by Evgeny Krivosheev, Burcu Sayin, Alessandro Bozzon and Zoltán Szlávik)
Evgeny Krivosheev, Burcu Sayin Günel, Alessandro Bozzon, Zoltan Szlavik
2020-12-11T13:45:00-08:00 - 2020-12-11T13:50:00-08:00
Q&A with Evgeny Krivosheev, Burcu Sayin, Alessandro Bozzon and Zoltán Szlávik: "Active Learning from Crowd in Item Screening"
2020-12-11T13:50:00-08:00 - 2020-12-11T14:05:00-08:00
Human Computation Requires and Enables a New Approach to Ethics (by Libuse Veprek, Patricia Seymour and Pietro Michelucci)
Libuše Vepřek, , Pietro Michelucci
2020-12-11T14:05:00-08:00 - 2020-12-11T14:10:00-08:00
Q&A with Libuse Veprek, Patricia Seymour and Pietro Michelucci: "Human computation requires and enables a new approach to ethics"
2020-12-11T14:10:00-08:00 - 2020-12-11T14:20:00-08:00
Coffee Break
2020-12-11T14:20:00-08:00 - 2020-12-11T14:40:00-08:00
Bias in Human-in-the-Loop Artificial Intelligence (by Gianluca Demartini)
Gianluca Demartini
2020-12-11T14:40:00-08:00 - 2020-12-11T14:50:00-08:00
Q&A with Gianluca Demartini: "Bias in Human-in-the-loop Artificial Intelligence"
2020-12-11T14:50:00-08:00 - 2020-12-11T15:05:00-08:00
VAIDA: An Educative Benchmark Creation Paradigm using Visual Analytics for Interactively Discouraging Artifacts (by Anjana Arunkumar, Swaroop Mishra, Bhavdeep Sachdeva, Chitta Baral and Chris Bryan)
Anjana Arunkumar, Swaroop Mishra, Chitta Baral
2020-12-11T15:05:00-08:00 - 2020-12-11T15:10:00-08:00
Q&A with Anjana Arunkumar, Swaroop Mishra, Bhavdeep Sachdeva, Chitta Baral and Chris Bryan: " VAIDA: An Educative Benchmark Creation Paradigm using Visual Analytics for Interactively Discouraging Artifacts"
2020-12-11T15:10:00-08:00 - 2020-12-11T15:30:00-08:00
Secret Invited Talk
Praveen Paritosh
2020-12-11T15:30:00-08:00 - 2020-12-11T15:40:00-08:00
Q&A
2020-12-11T15:40:00-08:00 - 2020-12-11T16:00:00-08:00