Workshop
The pre-registration experiment: an alternative publication model for machine learning research
Luca Bertinetto · João Henriques · Samuel Albanie · Michela Paganini · Gul Varol
Fri 11 Dec, 6:15 a.m. PST
Machine learning research has benefited considerably from the adoption of standardised public benchmarks. In this workshop proposal, we do not argue against the importance of these benchmarks, but rather against the current incentive system and its heavy reliance upon performance as a proxy for scientific progress. The status quo incentivises researchers to “beat the state of the art”, potentially at the expense of deep scientific understanding and rigorous experimental design. Since typically only positive results are rewarded, the negative results inevitably encountered during research are often omitted, allowing many other groups to unknowingly and wastefully repeat the same negative findings. Pre-registration is a publishing and reviewing model that aims to address these issues by changing the incentive system. A pre-registered paper is a regular paper that is submitted for peer-review without any experimental results, describing instead an experimental protocol to be followed after the paper is accepted. This implies that it is important for the authors to make compelling arguments from theory or past published evidence. As for reviewers, they must assess these arguments together with the quality of the experimental design, rather than comparing numeric results. In this workshop, we propose to conduct a full pilot study in pre-registration for machine learning. It follows a successful small-scale trial of pre-registration in computer vision and is more broadly inspired by the success of pre-registration in the life sciences.
Schedule
Fri 6:15 a.m. - 6:30 a.m.
|
Opening Remarks
(
Opening Remarks
)
>
|
Luca Bertinetto 🔗 |
Fri 6:31 a.m. - 7:00 a.m.
|
Francis Bach - Where is Machine Learning Going?
(
Invited talk
)
>
SlidesLive Video |
Francis Bach 🔗 |
Fri 7:01 a.m. - 7:30 a.m.
|
Yoshua Bengio - Incentives for Researchers
(
Invited talk
)
>
SlidesLive Video |
Yoshua Bengio 🔗 |
Fri 7:31 a.m. - 7:36 a.m.
|
Contributed talk - Contrastive Self-Supervised Learning for Skeleton Action Recognition
(
Contributed talk
)
>
SlidesLive Video |
Shaoyi Du 🔗 |
Fri 7:36 a.m. - 7:41 a.m.
|
Contributed talk - PCA Retargeting: Encoding Linear Shape Models as Convolutional Mesh Autoencoders
(
Contributed talk
)
>
SlidesLive Video |
Eimear O' Sullivan 🔗 |
Fri 7:41 a.m. - 7:46 a.m.
|
Contributed talk - Testing the Genomic Bottleneck Hypothesis in Hebbian Meta-Learning
(
Contributed talk
)
>
SlidesLive Video |
Rasmus Berg Palm 🔗 |
Fri 7:46 a.m. - 7:51 a.m.
|
Contributed talk - Policy Convergence Under the Influence of Antagonistic Agents in Markov Games
(
Contributed talk
)
>
SlidesLive Video |
Chase Dowling 🔗 |
Fri 8:01 a.m. - 9:00 a.m.
|
Poster session (on gather.town) ( Poster session ) > link | 🔗 |
Fri 9:00 a.m. - 10:30 a.m.
|
Break 1
|
🔗 |
Fri 10:31 a.m. - 11:00 a.m.
|
Joelle Pineau - Can pre-registration lead to better reproducibility in ML research?
(
Invited talk
)
>
SlidesLive Video |
Joelle Pineau 🔗 |
Fri 11:01 a.m. - 11:06 a.m.
|
Contributed talk - Confronting Domain Shift in Trained Neural Networks
(
Contributed talk
)
>
SlidesLive Video |
Carianne Martinez 🔗 |
Fri 11:06 a.m. - 11:11 a.m.
|
Contributed talk - Unsupervised Resource Allocation with Graph Neural Networks
(
Contributed talk
)
>
SlidesLive Video |
Miles Cranmer 🔗 |
Fri 11:11 a.m. - 11:16 a.m.
|
Contributed talk - FedPerf: A Practitioners' Guide to Performance of Federated Learning Algorithms
(
Contributed talk
)
>
SlidesLive Video |
Tushar Semwal 🔗 |
Fri 11:16 a.m. - 11:21 a.m.
|
Contributed talk - On the low-density latent regions of VAE-based language models
(
Contributed talk
)
>
SlidesLive Video |
Ruizhe Li 🔗 |
Fri 11:31 a.m. - 12:00 p.m.
|
Jessica Zosa Forde - Build, Start, Run, Push: Computational Registration of ML Experiments
(
Invited talk
)
>
|
Jessica Forde 🔗 |
Fri 12:00 p.m. - 12:01 p.m.
|
Introduction to break 2
(
Introduction
)
>
|
🔗 |
Fri 12:01 p.m. - 12:30 p.m.
|
Break 2
|
🔗 |
Fri 12:31 p.m. - 1:00 p.m.
|
Kirstie Whitaker - The Turing Way: Transparent research through the scientific lifecycle
(
Invited talk
)
>
SlidesLive Video |
Kirstie Whitaker 🔗 |
Fri 1:01 p.m. - 2:30 p.m.
|
Open Discussion
(
Discussion panel
)
>
|
🔗 |
-
|
Closing remarks
(
Closing remarks
)
>
|
🔗 |