Skip to yearly menu bar Skip to main content


NeurIPS 2025 Datasets & Benchmarks Track Call for Papers

 

The NeurIPS Datasets and Benchmarks track serves as a venue for high-quality publications on highly valuable machine learning datasets and benchmarks crucial for the development and continuous improvement of machine learning methods. Previous editions of the Datasets and Benchmarks track were highly successful and continuously growing (accepted papers 2021, 2002, and 2023, and best paper awards 2021, 2022, 2023 and 2024. Read our original blog post for more about why we started this track, and the 2025 blog post announcing this year's track updates. 

Dates and Guidelines

Please note that the Call for Papers of the NeurIPS2025 Datasets & Benchmarks Track this year will follow the Call for Papers of the NeurIPS2025 Main Track, with the addition of three track-specific points:

  • Single-blind submissions
  • Required dataset and benchmark code submission
  • Specific scope for datasets and benchmarks paper submission

Accepted papers will be published in the NeurIPS proceedings and presented at the conference alongside the main track papers. As such, we aim for an equally stringent review as in the main conference track, while also allowing for track-specific guidelines, which we introduce below. For details on everything else, e.g. formatting, code of conduct, ethics review, important dates, and any other submission related topics, please refer to the main track CFP. 

OpenReview

Submit at: https://openreview.net/group?id=NeurIPS.cc/2025/Datasets_and_Benchmarks_Track

The site will start accepting submissions on April 3, 2025 (at the same time as the main track).

Note: submissions meant for the main track should be submitted to a different OpenReview portal, as shown here. Papers will not be transferred between the main and the Datasets and Benchmarks tracks after the submission is closed. 

 

DB track specifics

The Datasets and Benchmarks track adds three changes in the submission and review guidelines that are better suited for the review of datasets and benchmarks. Below we briefly introduce them. For more details on how to host datasets, what metadata to use to describe them and what access requirements we have, please consult these guidelines.

  1. SINGLE-BLIND SUBMISSIONS

Datasets are often not possible to be reviewed in a double-blind fashion, and hence full anonymization will not be required for D&B paper submissions. Submissions to this track will be reviewed according to a set of criteria and best practices specifically designed for datasets and benchmarks, as described below. Authors can choose to submit either single-blind or double-blind. If it is possible to properly review the submission double-blind, i.e., if reviewers do not need access to non-anonymous repositories to review the work, then authors can also choose to submit the work anonymously. 

Note: NeurIPS does not tolerate any collusion whereby authors secretly cooperate with reviewers, ACs, or SACs to obtain favorable reviews.

   2. REQUIRED DATASET AND BENCHMARK CODE SUBMISSION

This year, we are introducing a more stringent set of criteria in the submission process regarding the hosting, accessibility and documentation of datasets and code at submission time:

  • Hosting: New datasets should be hosted at one of the hosting sites dedicated to ML datasets (Dataverse, Kaggle, Hugging Face, or OpenML) or at a bespoke hosting site if the dataset requires it. See the guidelines for data hosting for details. Code should be made accessible in an executable format via a hosting platform (e.g., GitHub, Bitbucket).  See the main track code guidelines for details.
  • Access: Datasets and code should be available and accessible to all reviewers, ACs and SACs at the time of submission. Data should be found and obtained without a personal request to the PI. Code should be documented and executable. Non-compliance justifies the desk rejection of the paper. 
  • Metadata: Authors should use the Croissant machine-readable format to document their datasets in a machine-readable way and include the Croissant file with their paper submission in OpenReview.
    • If your data is hosted on one of the dedicated ML data hosting sites (Kaggle, OpenML, Hugging Face, Dataverse) 
      • a Croissant metadata file is automatically generated for you
    • If you choose to host your data on a different hosting site 
      • you need to generate the Croissant metadata file yourself 

All accepted papers should have their code and datasets documented and publicly available by the camera-ready deadline. 

   3. SPECIFIC SCOPE FOR DATASETS & BENCHMARKS PAPER SUBMISSION

The NeurIPS 2025 D&B track welcomes all work on data-centric machine learning research (DMLR) that enable or accelerate ML research, covering ML datasets and benchmarks as well as algorithms, tools, methods, and analyses for working with ML data. The D&B track is proud to support the open source movement by encouraging submissions of open-source libraries and tools that enable or accelerate ML research. 

The scope includes but is not limited to:

  • New datasets, or carefully and thoughtfully designed (collections of) datasets based on previously available data.
  • Data generators and reinforcement learning environments.
  • Data-centric AI methods and tools, e.g. to measure and improve data quality or utility, or studies in data-centric AI that bring important new insight.
  • Advanced practices in data collection and curation that are of general interest even if the data itself cannot be shared. Private data sets, however, cannot be listed as contributions, and the methodology should also be validated on reproducible and publicly available data/tools.
  • Frameworks for responsible dataset development, audits of existing datasets, identifying significant problems with existing datasets and their use
  • Benchmarks on new or existing datasets, as well as benchmarking tools, including novel benchmarking methodologies and designs, e.g., approaches to mitigate LLM contamination.
  • In-depth analyses of machine learning challenges and competitions (by organisers and/or participants) that yield important new insight.
  • Systematic analyses of existing systems on novel datasets yielding important new insight.
  • We invite competition papers from prior NeurIPS competitions to be submitted at the Datasets and Benchmarks track. They will not be given any special reviewing policy, and will be reviewed with the same requirements, standards as all other track submissions. 

 

Contact

For any information you need help with, you can contact the DB chairs by choosing “Datasets and Benchmarks” in the official NeurIPS contact form here.