Datasets and Benchmarks
Dataset and Benchmark Poster Session 4
Joaquin Vanschoren · Serena Yeung
Abstract:
The Datasets and Benchmarks track serves as a novel venue for high-quality publications, talks, and posters on highly valuable machine learning datasets and benchmarks, as well as a forum for discussions on how to improve dataset development. Datasets and benchmarks are crucial for the development of machine learning methods, but also require their own publishing and reviewing guidelines. For instance, datasets can often not be reviewed in a double-blind fashion, and hence full anonymization will not be required. On the other hand, they do require additional specific checks, such as a proper description of how the data was collected, whether they show intrinsic bias, and whether they will remain accessible.
Schedule
-
|
MLPerf Tiny Benchmark
(
Poster
)
>
SlidesLive Video |
19 presentersColby Banbury · Vijay Janapa Reddi · Peter Torelli · Nat Jeffries · Csaba Kiraly · Jeremy Holleman · Pietro Montino · David Kanter · Pete Warden · Danilo Pau · Urmish Thakker · antonio torrini · jay cordaro · Giuseppe Di Guglielmo · Javier Duarte · Honson Tran · Nhan Tran · niu wenxu · xu xuesong |
-
|
Benchmark for Compositional Text-to-Image Synthesis
(
Poster
)
>
SlidesLive Video |
Dong Huk Park · Samaneh Azadi · Xihui Liu · Trevor Darrell · Anna Rohrbach 🔗 |
-
|
A Unified Few-Shot Classification Benchmark to Compare Transfer and Meta Learning Approaches
(
Poster
)
>
SlidesLive Video |
Vincent Dumoulin · Neil Houlsby · Utku Evci · Xiaohua Zhai · Ross Goroshin · Sylvain Gelly · Hugo Larochelle 🔗 |
-
|
HiRID-ICU-Benchmark --- A Comprehensive Machine Learning Benchmark on High-resolution ICU Data
(
Poster
)
>
SlidesLive Video |
Hugo Yèche · Rita Kuznetsova · Marc Zimmermann · Matthias Hüser · Xinrui Lyu · Martin Faltys · Gunnar Rätsch 🔗 |
-
|
ATOM3D: Tasks on Molecules in Three Dimensions
(
Poster
)
>
link
SlidesLive Video |
13 presentersRaphael Townshend · Martin Vögele · Patricia Suriana · Alex Derry · Alexander Powers · Yianni Laloudakis · Sidhika Balachandar · Bowen Jing · Brandon Anderson · Stephan Eismann · Risi Kondor · Russ Altman · Ron Dror |
-
|
MQBench: Towards Reproducible and Deployable Model Quantization Benchmark
(
Poster
)
>
SlidesLive Video |
Yuhang Li · Mingzhu Shen · Jian Ma · Yan Ren · Mingxin Zhao · Qi Zhang · Ruihao Gong · Fengwei Yu · Junjie Yan 🔗 |
-
|
TenSet: A Large-scale Program Performance Dataset for Learned Tensor Compilers
(
Poster
)
>
SlidesLive Video |
Lianmin Zheng · Ruochen Liu · Junru Shao · Tianqi Chen · Joseph Gonzalez · Ion Stoica · Ameer Haj-Ali 🔗 |
-
|
Revisiting Time Series Outlier Detection: Definitions and Benchmarks
(
Poster
)
>
SlidesLive Video |
Kwei-Herng Lai · Daochen Zha · Junjie Xu · Yue Zhao · Guanchu Wang · Xia Hu 🔗 |
-
|
A Large-Scale Database for Graph Representation Learning
(
Poster
)
>
SlidesLive Video |
Scott Freitas · Yuxiao Dong · Joshua Neil · Duen Horng Chau 🔗 |
-
|
Contemporary Symbolic Regression Methods and their Relative Performance
(
Poster
)
>
SlidesLive Video |
William La Cava · Patryk Orzechowski · Bogdan Burlacu · Fabricio de Franca · Marco Virgolin · Ying Jin · Michael Kommenda · Jason Moore 🔗 |
-
|
Personalized Benchmarking with the Ludwig Benchmarking Toolkit
(
Poster
)
>
link
SlidesLive Video |
Avanika Narayan · Piero Molino · Karan Goel · Willie Neiswanger · Christopher Ré 🔗 |
-
|
EEGEyeNet: a Simultaneous Electroencephalography and Eye-tracking Dataset and Benchmark for Eye Movement Prediction
(
Poster
)
>
SlidesLive Video |
Ard Kastrati · Martyna Plomecka · Damian Pascual Ortiz · Lukas Wolf · Victor Gillioz · Roger Wattenhofer · Nicolas Langer 🔗 |
-
|
DABS: a Domain-Agnostic Benchmark for Self-Supervised Learning
(
Poster
)
>
link
SlidesLive Video |
Alex Tamkin · Vincent Liu · Rongfei Lu · Daniel Fein · Colin Schultz · Noah Goodman 🔗 |
-
|
Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development
(
Poster
)
>
link
SlidesLive Video |
Kexin Huang · Tianfan Fu · Wenhao Gao · Yue Zhao · Yusuf Roohani · Jure Leskovec · Connor Coley · Cao Xiao · Jimeng Sun · Marinka Zitnik 🔗 |
-
|
Datasets for Online Controlled Experiments
(
Poster
)
>
SlidesLive Video |
Chak Hin Bryan Liu · Angelo Cardoso · Paul Couturier · Emma McCoy 🔗 |
-
|
SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
(
Poster
)
>
SlidesLive Video |
Robin Chan · Krzysztof Lis · Svenja Uhlemeyer · Hermann Blum · Sina Honari · Roland Siegwart · Pascal Fua · Mathieu Salzmann · Matthias Rottmann 🔗 |
-
|
Relational Pattern Benchmarking on the Knowledge Graph Link Prediction Task
(
Poster
)
>
SlidesLive Video |
Afshin Sadeghi · Hirra Malik · Diego Collarana · Jens Lehmann 🔗 |
-
|
Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks
(
Poster
)
>
SlidesLive Video |
16 presentersAndrey Malinin · Neil Band · Yarin Gal · Mark Gales · Alexander Ganshin · German Chesnokov · Alexey Noskov · Andrey Ploskonosov · Liudmila Prokhorenkova · Ivan Provilkov · Vatsal Raina · Vyas Raina · Denis Roginskiy · Mariya Shmatova · Panagiotis Tigas · Boris Yangel |
-
|
MIND dataset for diet planning and dietary healthcare with machine learning: Dataset creation using combinatorial optimization and controllable generation with domain experts
(
Poster
)
>
link
SlidesLive Video |
Changhun Lee · Soohyeok Kim · Sehwa Jeong · Chiehyeon Lim · Jayun Kim · Yeji Kim · Minyoung Jung 🔗 |
-
|
SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning
(
Poster
)
>
SlidesLive Video |
Christopher Yeh · Chenlin Meng · Sherrie Wang · Anne Driscoll · Erik Rozi · Patrick Liu · Jihyeon Lee · Marshall Burke · David Lobell · Stefano Ermon 🔗 |
-
|
FLIP: Benchmark tasks in fitness landscape inference for proteins
(
Poster
)
>
link
SlidesLive Video |
Christian Dallago · Jody Mou · Kadina Johnston · Bruce Wittmann · Nicholas Bhattacharya · Samuel Goldman · Ali Madani · Kevin Yang 🔗 |
-
|
HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML
(
Poster
)
>
SlidesLive Video |
Sebastian Pineda Arango · Hadi Jomaa · Martin Wistuba · Josif Grabocka 🔗 |
-
|
Neural Latents Benchmark ‘21: Evaluating latent variable models of neural population activity
(
Poster
)
>
link
SlidesLive Video |
16 presentersFelix Pei · Joel Ye · David Zoltowski · Anqi Wu · Raeed Chowdhury · Hansem Sohn · Joseph O'Doherty · Krishna V Shenoy · Matthew Kaufman · Mark Churchland · Mehrdad Jazayeri · Lee Miller · Jonathan Pillow · Il Memming Park · Eva Dyer · Chethan Pandarinath |
-
|
Benchmarking Data-driven Surrogate Simulators for Artificial Electromagnetic Materials
(
Poster
)
>
SlidesLive Video |
Yang Deng · Juncheng Dong · Simiao Ren · Omar Khatib · Mohammadreza Soltani · Vahid Tarokh · Willie Padilla · Jordan Malof 🔗 |
-
|
ClimART: A Benchmark Dataset for Emulating Atmospheric Radiative Transfer in Weather and Climate Models
(
Poster
)
>
link
SlidesLive Video |
Salva Rühling Cachay · Venkatesh Ramesh · Jason Cole · Howard Barker · David Rolnick 🔗 |
-
|
Graph Robustness Benchmark: Benchmarking the Adversarial Robustness of Graph Machine Learning
(
Poster
)
>
SlidesLive Video |
Qinkai Zheng · Xu Zou · Yuxiao Dong · Yukuo Cen · Da Yin · Jiarong Xu · Yang Yang · Jie Tang 🔗 |
-
|
A sandbox for prediction and integration of DNA, RNA, and proteins in single cells
(
Poster
)
>
SlidesLive Video |
38 presentersMalte Luecken · Daniel Burkhardt · Robrecht Cannoodt · Christopher Lance · Aditi Agrawal · Hananeh Aliee · Ann Chen · Louise Deconinck · Angela Detweiler · Alejandro Granados · Shelly Huynh · Laura Isacco · Yang Kim · Dominik Klein · BONY DE KUMAR · Sunil Kuppasani · Heiko Lickert · Aaron McGeever · Honey Mekonen · Joaquin Melgarejo · Maurizio Morri · Michaela Müller · Norma Neff · Sheryl Paul · Bastian Rieck · Kaylie Schneider · Scott Steelman · Michael Sterr · Daniel Treacy · Alexander Tong · Alexandra-Chloe Villani · Guilin Wang · Jia Yan · Ce Zhang · Angela Pisco · Smita Krishnaswamy · Fabian Theis · Jonathan M Bloom |
-
|
A Channel Coding Benchmark for Meta-Learning
(
Poster
)
>
SlidesLive Video |
Rui Li · Ondrej Bohdal · Rajesh K Mishra · Hyeji Kim · Da Li · Nicholas Lane · Timothy Hospedales 🔗 |
-
|
HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
(
Poster
)
>
SlidesLive Video |
Katharina Eggensperger · Philipp Müller · Neeratyoy Mallik · Matthias Feurer · Rene Sass · Aaron Klein · Noor Awad · Marius Lindauer · Frank Hutter 🔗 |
-
|
OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs
(
Poster
)
>
SlidesLive Video |
Weihua Hu · Matthias Fey · Hongyu Ren · Maho Nakata · Yuxiao Dong · Jure Leskovec 🔗 |
-
|
RobustBench: a standardized adversarial robustness benchmark
(
Poster
)
>
SlidesLive Video |
Francesco Croce · Maksym Andriushchenko · Vikash Sehwag · Edoardo Debenedetti · Nicolas Flammarion · Mung Chiang · Prateek Mittal · Matthias Hein 🔗 |
-
|
Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks
(
Poster
)
>
link
SlidesLive Video |
Neil Band · Tim G. J. Rudner · Qixuan Feng · Angelos Filos · Zachary Nado · Mike Dusenberry · Ghassen Jerfel · Dustin Tran · Yarin Gal 🔗 |
-
|
Chaos as an interpretable benchmark for forecasting and data-driven modelling
(
Poster
)
>
SlidesLive Video |
William Gilpin 🔗 |
-
|
Benchmarking the Robustness of Spatial-Temporal Models Against Corruptions
(
Poster
)
>
SlidesLive Video |
Chenyu Yi · SIYUAN YANG · Haoliang Li · Yap-peng Tan · Alex Kot 🔗 |
-
|
Whole Brain Vessel Graphs: A Dataset and Benchmark for Graph Learning and Neuroscience
(
Poster
)
>
SlidesLive Video |
12 presentersJohannes C. Paetzold · Julian McGinnis · Suprosanna Shit · Ivan Ezhov · Paul Büschl · Chinmay Prabhakar · Anjany Sekuboyina · Mihail Todorov · Georgios Kaissis · Ali Ertürk · Stephan Günnemann · Bjoern Menze |
-
|
FS-Mol: A Few-Shot Learning Dataset of Molecules
(
Poster
)
>
SlidesLive Video |
Megan Stanley · John Bronskill · Krzysztof Maziarz · Hubert Misztela · Jessica Lanini · Marwin Segler · Nadine Schneider · Marc Brockschmidt 🔗 |
-
|
WRENCH: A Comprehensive Benchmark for Weak Supervision
(
Poster
)
>
SlidesLive Video |
Jieyu Zhang · Yue Yu · · Yujing Wang · Yaming Yang · Mao Yang · Alexander Ratner 🔗 |
-
|
GraphGT: Machine Learning Datasets for Graph Generation and Transformation
(
Poster
)
>
link
SlidesLive Video |
Yuanqi Du · Shiyu Wang · Xiaojie Guo · Hengning Cao · Shujie Hu · Junji Jiang · Aishwarya Varala · Abhinav Angirekula · Liang Zhao 🔗 |
-
|
BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
(
Poster
)
>
SlidesLive Video |
Nandan Thakur · Nils Reimers · Andreas Rücklé · Abhishek Srivastava · Iryna Gurevych 🔗 |
-
|
WildfireDB: An Open-Source Dataset Connecting Wildfire Occurrence with Relevant Determinants
(
Poster
)
>
SlidesLive Video |
Samriddhi Singla · Ayan Mukhopadhyay · Michael Wilbur · Tina Diao · Vinayak Gajjewar · Ahmed Eldawy · Mykel J Kochenderfer · Ross Shachter · Abhishek Dubey 🔗 |
-
|
The Tufts fNIRS Mental Workload Dataset & Benchmark for Brain-Computer Interfaces that Generalize
(
Poster
)
>
link
SlidesLive Video |
zhe huang · Liang Wang · Giles Blaney · Christopher Slaughter · Devon McKeon · Ziyu Zhou · Robert Jacob · Michael Hughes 🔗 |
-
|
The CPD Data Set: Personnel, Use of Force, and Complaints in the Chicago Police Department
(
Poster
)
>
SlidesLive Video |
Thibaut Horel · Lorenzo Masoero · Raj Agrawal · Daria Roithmayr · Trevor Campbell 🔗 |