Skip to yearly menu bar Skip to main content


2024 Reviewer Guidelines

Thank you for agreeing to serve as a reviewer for NeurIPS 2024!

This page provides an overview of reviewer responsibilities and key dates.

Contact Information

The Area Chair (AC) assigned to a paper should be your first point of contact for that paper. You can contact the AC by leaving a comment in OpenReview with the AC as a reader.  (SACs and PCs will also be listed as readers, but will not be notified.)

If you encounter a situation that you are unable to resolve with your AC, please contact the program chairs at pc2024@neurips.cc. Please refrain from writing to the program chairs at their own email addresses.

Important Dates

Dates may be subject to change

Reviewers bid on papers:  Thur, May 23  – Thur, May 30
Check paper assignments: Mon, June 10
First round reviewing: Mon, June 10 – Fri, July 12
Invite emergency reviewers: Fri, July 12 – Wed, July 17
Second round reviewing: Wed, July 17 - Fri, July 26
Reviews released to authors: Tues, July 30
Author Rebuttal: Tues, July 30 - Tues, Aug 6
Reviewer-Author Discussions: Tue, Aug 6 – Tue, Aug 13
Reviewer-AC Discussions: Tue, Aug 13  - Fri, Aug 23
Metareviews due:  Fri, Aug 30
SAC-AC Discussions: Fri, Aug 30 - Mon, Sep 9
SAC-PC Discussions: Tues, Sep 10 – Thu, Sep 19
Author Notifications: Wed, Sept 25

 

Main Tasks

Fulfilling your responsibilities as a reviewer in a high quality and timely manner is critical to the success of the review process. Here is a list of key dates and tasks for reviewers:

  1. Preparation:
    • Read and agree to abide by the NeurIPS code of conduct.
    • NeurIPS 2024 is using OpenReview. Please make sure that your OpenReview profile is up to date. If you have changed or plan to change your email address, please update the address set as “preferred” in your OpenReview profile and confirm it. It is crucial that we are able to reach you quickly. We will send most emails from OpenReview (noreply@openreview.net). Such emails are sometimes accidentally marked as spam (or classified as Updates in Gmail). Please check these folders regularly. If you find such an email in there, please whitelist noreply@openreview.net so that you do not miss future emails related to NeurIPS 2024.
    • Note that your assignments and tasks will appear at the reviewer console in OpenReview: https://openreview.net/group?id=NeurIPS.cc/2024/Conference/Reviewers
    • Read what constitutes conflict of interest for NeurIPS 2024.
  2. Bid on papers: Thur, May 23  – Thur, May 30
    • Your bids are an important input to the paper matching process. 
    • Unfortunately, in past years there have been a small number of reviewers who engage in deceptive bidding practices. If we have a reason to suspect that a reviewer is engaged in deceitful bidding to influence reviewing outcomes, we will request an ethics investigation, and malicious actors may be removed from future involvement in the program committee. 
  3. Check paper assignments: Monday, June 10.
    • As soon as you are notified of papers to review, you are expected to log in to OpenReview to check for conflicts and to check that papers fall within your area of expertise.
    • If you don’t feel qualified to review a paper that was assigned to you, please communicate this to your AC right away.
    • These assignments may change during the first week, as some reviewers and ACs request re-assignments. Please watch for notification email from Openreview.
  4. Write thoughtful reviews: Mon, June 10 – Fri, July 12
    • We know that serving as a reviewer for NeurIPS is time consuming, but the community depends on your high quality reviews to uphold the scientific quality of NeurIPS. 
    • Please make your review as informative and substantiated as possible; superficial, uninformed reviews are worse than no review as they may contribute noise to the review process.
    • You can see the review form questions and guidance on how to answer each question in the "Review Form" section below.
    • Make sure to flag any questionable papers for ethics review. These papers will be assigned ethics reviewers, who will effectively join the paper's assigned program committee. See the NeurIPS ethics guidelines.
    • Feel free to use the NeurIPS paper checklist included in each paper as a tool when preparing your review (some submissions may have the checklist as part of the supplementary materials) . Remember that answering “no” to some questions is typically not grounds for rejection. In general, authors should be rewarded rather than punished for being up front about the limitations of their work and any potential negative societal impact. You are encouraged to think through whether any critical points are missing and provide these as feedback for the authors.
    • Do not worry about minor violations of the required format (e.g., papers that exceed the page limit by a few lines), but immediately report any major violations that you notice to your AC.
    • When writing your review, please keep in mind that after decisions have been made, reviews and meta-reviews of accepted papers as well as your discussion with the authors will be made public (but reviewer and SAC/AC identities will remain anonymous). This year, authors of rejected papers will have the option to make this information public for their rejected papers as well.
    • Please note that the review form this year asks for some additional fields such as text fields for the strengths and weaknesses, and questions and suggestions for authors. There are also additional ratings on soundness, presentation and contribution, in addition to the overall rating. These are intended to make it easier for the AC and SAC to understand your rationale for the rating, and facilitate better discussions.
  5. Read author responses and discuss papers: Tue, Aug 6 – Fri, Aug 23
    • Authors will be given one week to respond to their reviews before the discussion period. At the start of the discussion period, please carefully read all other reviews and the author responses to all reviews for the papers assigned to you.
    • As you read each author response, please keep an open mind.  Even if the author response didn’t change your opinion about the paper, please acknowledge that you have read and considered it.  
    • To minimize the chance of misunderstandings during the reviewing process, we will allow for a rolling discussion with the authors during the discussion period.  If you need to communicate with the authors, you can make a comment visible to them on the paper’s page.
    • Participating in discussions is a critical part of your role as a reviewer.  The discussion period is especially important for borderline papers and papers for which the reviewers’ assessments differ, and we hope that you take discussions seriously.  If your evaluation of the paper has changed, please revise your review and explain the change.
    • When discussing a paper, remember that different people have different backgrounds and different points of view. Reviewer consensus is valuable—only rarely are unanimous assessments overruled—but it is not mandatory.
  6. ACs make initial accept/reject decisions with SACs:  Fri, Aug 30 - Mon, Sep 9
    • During this period, ACs will be making initial recommendations for each paper. Your workload during this period should be light, but if ACs come back to you with additional questions, please respond promptly.
  7. Notification: Wednesday September 25, 2024.

 

Review Form

Below is a description of the questions you will be asked on the review form for each paper and some guidelines on what to consider when answering these questions. Feel free to use the NeurIPS paper checklist included in each paper as a tool when preparing your review (some submissions may have the checklist as part of the supplementary materials). Remember that answering “no” to some questions is typically not grounds for rejection. When writing your review, please keep in mind that after decisions have been made, reviews and meta-reviews of accepted papers and opted-in rejected papers will be made public.

  1. Summary: Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.
  2. Strengths and Weaknesses: Please provide a thorough assessment of the strengths and weaknesses of the paper, touching on each of the following dimensions:
    a. Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?
    b. Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
    c. Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)
    d. Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
    You can incorporate Markdown and Latex into your review.  See https://openreview.net/faq.
  3. Questions: Please list up and carefully describe any questions and suggestions for the authors. Think of the things where a response from the author can change your opinion, clarify a confusion or address a limitation. This can be very important for a productive rebuttal and discussion phase with the authors.  
  4. Limitations: Have the authors adequately addressed the limitations and potential negative societal impact of their work? If not, please include constructive suggestions for improvement.
    In general, authors should be rewarded rather than punished for being up front about the limitations of their work and any potential negative societal impact. You are encouraged to think through whether any critical points are missing and provide these as feedback for the authors.
  5. Ethical concerns: If there are ethical issues with this paper, please flag the paper for an ethics review. For guidance on when this is appropriate, please review the NeurIPS ethics guidelines.
  6. Soundness: Please assign the paper a numerical rating on the following scale to indicate the soundness of the technical claims, experimental and research methodology and on whether the central claims of the paper are adequately supported with evidence.
    • 4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  7. Presentation: Please assign the paper a numerical rating on the following scale to indicate the quality of the presentation. This should take into account the writing style and clarity, as well as contextualization relative to prior work.
    • 4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  8. Contribution: Please assign the paper a numerical rating on the following scale to indicate the quality of the overall contribution this paper makes to the research area being studied. Are the questions being asked important? Does the paper bring a significant originality of ideas and/or execution? Are the results valuable to share with the broader NeurIPS community.
    • 4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  9. Overall: Please provide an "overall score" for this submission. Choices:
    • 10: Award quality: Technically flawless paper with groundbreaking impact on one or more areas of AI, with exceptionally strong evaluation, reproducibility, and resources, and no unaddressed ethical considerations.
    • 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI and excellent impact on multiple areas of AI, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
    • 8: Strong Accept: Technically strong paper with, with novel ideas, excellent impact on at least one area of AI or high-to-excellent impact on multiple areas of AI, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
    • 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate-to-high impact on more than one area of AI, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
    • 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
    • 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
    • 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
    • 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
    • 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
    • 1: Very Strong Reject: For instance, a paper with trivial results or unaddressed ethical considerations
  10. Confidence:  Please provide a "confidence score" for your assessment of this submission to indicate how confident you are in your evaluation.  Choices
    • 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
    • 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
    • 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
    • 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
    • 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
  11. Code of conduct acknowledgement. While performing my duties as a reviewer (including writing reviews and participating in discussions), I have and will continue to abide by the NeurIPS code of conduct (NeurIPS Code Of Conduct).

 

Other Roles

During the review process you will be working with:

  • Area Chairs (ACs). ACs are the principal contact for reviewers during the whole reviewing process. ACs are responsible for recommending reviewers for submissions, ensuring that all submissions receive quality reviews, facilitating discussions among reviewers, writing meta-reviews, evaluating the quality of reviews, and making decision recommendations.
  • Senior Area Chairs (SACs). Each SAC oversees the work of a small number of ACs, making sure that the reviewing process goes smoothly. SACs are also responsible for helping ACs find expert reviewers, calibrating decisions across ACs, discussing borderline papers, and helping the Program Chairs (PCs) make final decisions.
  • Ethics Reviewers. You may flag submissions for additional review by ethics reviewers. The comments from the ethics reviewers will be visible to all reviewers, the AC, and the authors. You may use their comments to inform your deliberations.

Best Practices

  • Be thoughtful. The paper you are reviewing may have been written by a first year graduate student who is submitting to a conference for the first time and you don't want to crush their spirits.
  • Be fair. Do not let personal feelings affect your review.
  • Be useful. A good review is useful to all parties involved: authors, other reviewers and AC/SACs. Try to keep your feedback constructive when possible.
  • Be specific. Do not make vague statements in your review, as they are unfairly difficult for authors to address. 
  • Be flexible. The authors may address some points you raised in your review during the discussion period. Make an effort to update your understanding of the paper when new information is presented, and revise your review to reflect this.
  • Be timely. Please respect the deadlines and respond promptly during the discussion.  If you cannot complete your review on time, please let the AC know as soon as possible.
  • If someone pressures you into providing a positive or negative review for a submission, please notify program chairs right away (pc2024@neurips.cc).
  • If you notice unethical or suspect behavior, please notify your AC right away.

FAQ

NOTE: This section may be updated periodically throughout the review cycle as we get questions.

Q: What if I need to speak to my AC?

A: You can communicate with your AC via OpenReview by writing a comment with visibility limited to your AC on the discussion page of the paper in question.  The comment will be visible to the SAC and Program Chairs by default, but it will not trigger a notification.

Q: What if I’ve seen similar work in a NeurIPS/ICML workshop?

A: We allow work that has been submitted to non-archival workshops to be submitted to NeurIPS. To maintain anonymity, do not mention the workshop paper in your review.

Q: What should I do if I suspect plagiarism?

A: You should reach out to your AC and explain the situation.

Q: What should I do if I am not able to meet a deadline due to an emergency?

A: Please reach out to your AC as soon as possible to provide a revised timeline (or to let them know that you cannot complete the assignment).

Q: Can I accept or reject all the papers in my stack?

A: Please accept and reject papers based on their own merits. You do not have to match the conference acceptance rate.

Q: Do I have to read the supplementary material?

A: You are not required to read it, but you are welcome to.

Q: Can I read the previous reviews of a paper if it is a resubmission?

A: Never break anonymity. And even if you were already aware of the paper, your review should be independent of previous reviews.

Q: What should I do if I have already reviewed this paper at another venue?

A: Do not assume that the paper hasn’t changed. Read the paper carefully, and make sure you write a high quality review.

Q: Can I invite a sub-reviewer to help with my reviews?

A: No, sub-reviewers are not allowed. Conflicts of interest cannot be properly checked unless reviewers are officially in the system, and sub-reviewers would not be able to participate in the discussion, which is a critical phase of the review process.

Policies

Please make sure to review the policies in the NeurIPS 2024 Call for Papers. 

Executing Code & Clicking on Links

Please remember that just like any other untrusted code, any submitted code may contain security vulnerabilities. If you are planning to run any submitted code, please make sure you are doing this in a secure environment because this code is not vetted by our submission system. We recommend running source code (1) inside a Docker container, or (2) a Virtual Machine image (using VirtualBox or VMWare), or (3) on a network-isolated cloud instance.

You may wish to also be cautious about accessing other web links provided from the paper, as these may contain vulnerabilities or may log visitor IP addresses.

Confidentiality

You must keep everything relating to the review process confidential. Do not use ideas, code, or results from submissions in your own work until they become publicly available. Do not talk about or share submissions with anyone without prior approval from the program chairs. Code submitted for reviewing cannot be distributed or used for any other purpose. 

Double-blind reviewing

The reviewing process is double blind at the level of reviewers and ACs (i.e., reviewers and ACs cannot see author identities) but not at the level of SACs and program chairs. Authors are responsible for anonymizing their submissions. Submissions may not contain any identifying information that may violate the double-blind reviewing policy.  This policy applies to any supplementary or linked material as well, including code.  If you are assigned a submission that is not adequately anonymized, please contact the corresponding AC. Please do not attempt to find out the identities of the authors for any of your assigned submissions (e.g., by searching on arXiv).  This would constitute an active violation of the double-blind reviewing policy.

Formatting instructions

As a reminder, submissions are limited to nine content pages, including all figures and tables, in the NeurIPS “submission” style; additional pages containing only references and the NeurIPS 2024 paper checklist are allowed. Any content beyond that can be reviewed at your discretion. This includes any additional content that was submitted as part of the main PDF as well as any supplementary material uploaded separately.  In general, we were lenient with minor formatting violations (e.g., a spillover to page 10), as long as these violations can be easily rectified in the final version.  If you find violations that are not easily rectified without causing other presentation issues, please flag them to your AC.

Some submissions may have included the NeurIPS 2024 checklist into their supplementary material by mistake, so you may find the checklist there (to be viewed at your discretion).

Dual submissions

NeurIPS does not allow submissions that are identical or substantially similar to papers that are in submission to, have been accepted to, or have been published in other archival venues. Submissions that are identical or substantially similar to other NeurIPS submissions fall under this policy as well; all NeurIPS submissions should be distinct and sufficiently substantial. Slicing contributions too thinly is discouraged, and may fall under the dual submission policy.  If you suspect that a submission that has been assigned to you is a dual submission or if you require further clarification, please contact the corresponding AC. For more information about dual submissions, please see the Call for Papers and FAQ.

Useful Links