Skip to yearly menu bar Skip to main content


NeurIPS 2020 Code Submission Policy

NeurIPS 2020 will use the following Code and Data Submission Policy.

 

1. The policy only applies to papers that contribute and present experiments with a) a new algorithm (or a modification to an existing algorithm) or b) a new dataset. That is, a paper is not covered by this policy if:

      a. The paper is not claiming the contribution of any novel algorithm and it is tested on existing datasets.

      b. The paper presents a new algorithm but only analyzes it theoretically (i.e., no experimental results are presented).

2.  Code and data submission for papers covered by this policy is expected but not enforced. If authors submit their code, they must anonymize it and include it in a single file along with any additional supplementary material document. Small datasets can also be included in such zip file (it must be <100MB). Big datasets can instead be linked via an anonymous URL. The supplementary material deadline is on June 11, 2020 1PM PDT.

3. The policy accepts a reimplementation by the authors that isn't the code originally run to produce the results reported in the paper (what is instead requested is the equivalent of an official implementation of the paper's contribution).

4. The policy accepts code that isn't “executable” as is as it has dependencies going beyond the algorithm itself and that cannot be released. Such dependencies would include: 

      a. Dataset that cannot be released (e.g., for privacy reasons).

      b. Specialized hardware that might not be commonly accessible (e.g., specialized accelerators or robotic platforms).

      c. Non-open sourced or non-free libraries, which do not include the algorithm that is claimed as the scientific contribution of the paper (e.g., paid-for mathematical programming solvers, commercial simulators, MATLAB).

         The authors will be asked to explain what dependencies are not released and why.

5. The policy expects (but does not require) code for accepted papers to be submitted by the camera-ready deadline (October 22, 2020). At this time, code must be de-anonymized.

6. The policy strongly encourages authors reporting experiments on a new dataset to conform to the following rules:

  • A link to the dataset is provided in the paper (at submission time in anonymized format).
  • The dataset is deposited in a repository that ensures long term preservation of the data.
  • The dataset has a persistent identifier such as Digital Object Identifier or Compact Identifier.
  • The dataset adheres to Schema.org or DCAT metadata standards.
  • The license and/or any data access restrictions are described in the paper.
  • The paper includes a convincing justification of the special nature of the dataset that makes it impossible to conform to these suggestions.

7. To facilitate reproducibility, we encourage authors to submit code that contains as many of the following components as possible:

  • Specification of dependencies
  • Training code
  • Evaluation code
  • Table/Figure of the main result, with instructions to reproduce it
  • Pre-trained model

Templates for the Readme and additional information are available at: https://github.com/paperswithcode/releasing-research-code

8. Code submitted during the review period is kept in absolute confidentiality and used only for reviewing purposes.

 

After the camera-ready deadline, NeurIPS intends to measure the percentage of accepted papers for which code and data were not released, despite being covered by the policy.