Skip to yearly menu bar Skip to main content


NeurIPS 2022 Datasets and Benchmarks Track

We are immensely grateful for the tremendous contributions of the 92 area chairs, 1064 reviewers and 39 ethics reviewers to make this new endeavor a success.

 

The Datasets and Benchmarks track serves as a novel venue for high-quality publications, talks, and posters on highly valuable machine learning datasets and benchmarks, as well as a forum for discussions on how to improve dataset development. Datasets and benchmarks are crucial for the development of machine learning methods, but also require their own publishing and reviewing guidelines. For instance, datasets can often not be reviewed in a double-blind fashion, and hence full anonymization will not be required. On the other hand, they do require additional specific checks, such as a proper description of how the data was collected, whether they show intrinsic bias, and whether they will remain accessible. 

The 2021 Datasets and Benchmarks track was a big success; you can view the accepted papers from 2021 NeurIPS Datasets and Benchmarks here, and the winners of the best paper awards for the track here.

CRITERIA. We are aiming for an equally stringent review as the main conference, yet better suited to datasets and benchmarks. Submissions to this track will be reviewed according to a set of criteria and best practices specifically designed for datasets and benchmarks, as described below. A key criterion is accessibility: datasets should be available and accessible, i.e. the data can be found and obtained without a personal request to the PI, and any required code should be open source. Next to a scientific paper, authors should also submit supplementary materials such as detail on how the data was collected and organized, what kind of information it contains, how it should be used ethically and responsibly, as well as how it will be made available and maintained. 

RELATIONSHIP TO NEURIPS. Submissions to the track will be part of the main NeurIPS conference, presented alongside the main conference papers. Accepted papers will be officially published in associated proceedings clearly linked to, yet separate from, the NeurIPS proceedings. The proceedings will be called Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks and they will be hosted on the NeurIPS website next to the main NeurIPS proceedings. We will maintain a page on the NeurIPS website with all accepted datasets and additional information. 

SUBMISSIONS. There will be one deadline this year. It is also still possible to submit datasets and benchmarks to the main conference (under the usual review process), but dual submission to both is not allowed (unless you retracted your paper from the main conference). Submission is single-blind, and the review process is open during the discussion phase. However, only accepted papers will remain visible after the review phase, and the datasets themselves can be released at a later date (see below). If it is possible to properly review the submission double-blind, i.e., reviewers do not need access to non-anonymous repositories to review the work, then authors can also choose to submit the work anonymously.

SCOPE. In addition to new datasets and benchmarks on new or existing datasets, we welcome submissions that detail advanced practices in data collection and curation that are of general interest even if the data itself cannot be shared. Data generators, reinforcement learning environments, or benchmarking tools are also in scope. Frameworks for responsible dataset development, audits of existing datasets, identifying significant problems with existing datasets and their use, or systematic analyses of existing systems on novel datasets that yield important new insight are also in scope.

Read our blog post from last year for more about why we started this track.

 

Important dates

  • Abstract submission deadline: Monday, June 6, 2022 01:00 PM PDT.
  • Full paper submission and co-author registration deadline: Thursday, June 9, 2022 01:00 PM PDT
  • Supplementary materials submission deadline: Thursday, June 16, 2022 01:00 PM PDT
  • End of the reviewing process - July 27, 2022 01:00 PM PDT
  • Start of Author discussions on OpenReview: Wednesday, August 2, 2022, 01:00 PM PDT
  • End of author/reviewer discussions on OpenReview: Monday, August 29th, 2022, 01:00 PM PDT (extended to allow more time to respond to ethics reviews)
  • Author notification: Friday, September 16, 2022
  • Camera-ready deadline: Wednesday, October 12, 2022 01:00 PM PDT
  • Video submission deadline:  October 22, 2022 01:00 PM PDT
  • Poster submission deadline: before the conference.

 

Note: The site will start accepting submissions on April 16, 2022.

 

FREQUENTLY ASKED QUESTIONS

Q: My work is in scope for this track but possibly also for the main conference. Where should I submit it?

A: This is ultimately your choice. Consider the main contribution of the submission and how it should be reviewed. If the main contribution is a new dataset, benchmark, or other work that falls into the scope of the track (see above), then it is ideally reviewed accordingly. As discussed in our blog post, the reviewing procedures of the main conference are focused on algorithmic advances, analysis, and applications, while the reviewing in this track is equally stringent but designed to properly assess datasets and benchmarks. Other, more practical considerations are that this track allows single-blind reviewing (since anonymization is often impossible for hosted datasets) and intended audience, i.e., make your work more visible for people looking for datasets and benchmarks. 

Q: How will paper accepted to this track be cited?

A: Accepted papers will appear in official proceedings hosted on the NeurIPS website, next to (yet separate from) the main conference proceedings. The official name will be Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. 

Q: Do I need to submit an abstract beforehand?

A: Yes, please check the important dates section for more information.

Q: My dataset requires open credentialized access. Can I submit to this track?

A: This will be possible on the condition that a credentialization is necessary for the public good (e.g. because of ethically sensitive medical data), and that an established credentialization procedure is in place that is 1) open to a large section of the public, 2) provides rapid response and access to the data, and 3) is guaranteed to be maintained for many years. A good example here is PhysioNet Credentialing, where users must first understand how to handle data with human subjects, yet is open to anyone who has learned and agrees with the rules. This should be seen as an exceptional measure, and NOT as a way to limit access to data for other reasons (e.g. to shield data behind a Data Transfer Agreement). Misuse would be grounds for desk rejection. During submission, you can indicate that your dataset involves open credentialized access, in which case the necessity, openness, and efficiency of the credentialization process itself will also be checked.

Q: What is the guidance on the design and resolution of the thumbnail to be uploaded?

A: Thumbnails are crucial for virtual poster sessions. Please follow the guidelines here for creating and uploading the thumbnails.

 

SUBMISSION INSTRUCTIONS

A submission consists of:

  • Submissions are limited to 9 content pages in NeurIPS format, including all figures and tables; additional pages containing the paper checklist, references, and acknowledgements are allowed. If your submission is accepted, you will be allowed an additional content page for the camera-ready version.

    • Please carefully follow the Latex template for this track when preparing proposals. We follow the NeurIPS format, but with the appropriate headings, and without hiding the names of the authors. Download the template as a bundle here.

    • Papers should be submitted via OpenReview

    • Reviewing is in principle single-blind, hence the paper should not be anonymized. In cases where the work can be reviewed equally well anonymously, anonymous submission is also allowed.

    • During submission, you can add a public link to the dataset or benchmark data. If the dataset can only be released later, you must include instructions for reviewers on how to access the dataset. This can only be done after the first submission by sending an official note to the reviewers in OpenReview. We highly recommend making the dataset publicly available immediately or before the start of the NeurIPS conference. In select cases, requiring solid motivation, the release date can be stretched up to a year after the submission deadline.

  • Submission introducing new datasets must include the following in the supplementary materials (as a separate PDF):

    • Dataset documentation and intended uses. Recommended documentation frameworks include datasheets for datasets, dataset nutrition labels, data statements for NLP, and accountability frameworks. 

    • URL to website/platform where the dataset/benchmark can be viewed and downloaded by the reviewers.

    • Author statement that they bear all responsibility in case of violation of rights, etc., and confirmation of the data license.

    • Hosting, licensing, and maintenance plan. The choice of hosting platform is yours, as long as you ensure access to the data (possibly through a curated interface) and will provide the necessary maintenance.

  • To ensure accessibility, we largely follow the NeurIPS guidelines for data submission, but also allowing more freedom for non-static datasets. The supplementary materials for datasets must include the following:

    • Links to access the dataset and its metadata. This can be hidden upon submission if the dataset is not yet publicly available but must be added in the camera-ready version. In select cases, e.g when the data can only be released at a later date, this can be added afterward (up to a year after the submission deadline). Simulation environments should link to open source code repositories.

    • The dataset itself should ideally use an open and widely used data format. Provide a detailed explanation on how the dataset can be read. For simulation environments, use existing frameworks or explain how they can be used.

    • Long-term preservation: It must be clear that the dataset will be available for a long time, either by uploading to a data repository or by explaining how the authors themselves will ensure this

    • Explicit license: Authors must choose a license, ideally a CC license for datasets, or an open source license for code (e.g. RL environments). An overview of licenses can be found here: https://paperswithcode.com/datasets/license 

    • Add structured metadata to a dataset's meta-data page using Web standards (like schema.org and DCAT): This allows it to be discovered and organized by anyone. A guide can be found here: https://developers.google.com/search/docs/data-types/dataset. If you use an existing data repository, this is often done automatically.

    • Highly recommended: a persistent dereferenceable identifier (e.g. a DOI minted by a data repository or a prefix on identifiers.org) for datasets, or a code repository (e.g. GitHub, GitLab,...) for code. If this is not possible or useful, please explain why.

  • For benchmarks, the supplementary materials must ensure that all results are easily reproducible. Where possible, use a reproducibility framework such as the ML reproducibility checklist, or otherwise guarantee that all results can be easily reproduced, i.e. all necessary datasets, code, and evaluation procedures must be accessible and documented.

  • For papers introducing best practices in creating or curating datasets and benchmarks, the above supplementary materials are not required.

  • For papers resubmitted after being retracted from another venue: a brief discussion on the main concerns raised by previous reviewers and how you addressed them. You do not need to share the original reviews.

 

REVIEWING AND SELECTION PROCESS

Reviewing will be single-blind. A datasets and benchmarks program committee will be formed, consisting of experts on machine learning, dataset curation, and ethics. We will ensure diversity in the program committee, both in terms of background as well as technical expertise (e.g., data, ML, data ethics, social science expertise). Each paper will be reviewed by the members of the committee. In select cases that are flagged by reviewers, an ethics review may be performed as well.

The review process will be open: papers and reviews will be publicly visible during the review phase to allow community feedback. They will be hidden again after the review phase, unless they are accepted or when authors opt-in. Authors can choose to keep the datasets themselves hidden until a later release date, as long as reviewers have access.

The factors that will be considered when evaluating papers include:

  • Utility and quality of the submission: Impact, originality, novelty, relevance to the NeurIPS community will all be considered. 

  • Completeness of the relevant documentation: For datasets, sufficient detail must be provided on how the data was collected and organized, what kind of information it contains, how it should be used ethically and responsibly, as well as how it will be made available and maintained. For benchmarks, best practices on reproducibility should be followed.

  • Accessibility and accountability: For datasets, there should be a convincing hosting, licensing, and maintenance plan.

Ethics and responsible use: Any ethical implications should be addressed and guidelines for responsible use should be provided where appropriate. Note that, if your submission includes publicly available datasets (e.g. as part of a larger benchmark), you should also check these datasets for ethical issues. You remain responsible for the ethical implications of including existing datasets or other data sources in your work.

 

ADVISORY COMMITTEE

The following committee will provide advice on the organization of the track over the coming years: Sergio Escalera, Isabelle Guyon, Neil Lawrence, Dina Machuve, Olga Russakovsky, Joaquin Vanschoren, Serena Yeung.

 

DATASETS AND BENCHMARKS CHAIRS

Anmol Kalia, Meta AI Research (Assistant to the PC)

Deepti Ghadiyaram, Meta AI Research

Joaquin Vanschoren, Eindhoven University of Technology


Contact: neurips-2022-datasets-benchmarks@googlegroups.com