Skip to yearly menu bar Skip to main content


AC & SAC Guidelines

Frequently asked questions

Frequently asked questions can be found here: [ link for ACs ] [ link for SACs ]

Contacting the program chairs

If you encounter a situation that you are unable to resolve on your own and which is a general question or bug report, please contact the program chairs at neurips2020pcs@gmail.com .

If you have specific questions related to the handling of a particular paper, please email directly your point of contact PC (for SACs) or SAC (for ACs).

Introduction

Thank you for agreeing to serve for NeurIPS 2020! In addition to the guidelines below, please familiarize yourself with the reviewer responsibilities and guidelines.  You will be interacting significantly with reviewers, and the quality and timeliness of their feedback is critical to the success of the peer-reviewing process, so please make sure you understand what is expected of them and how this has been communicated.

Key dates

  • Abstract Submission deadline: May 27 

  • Paper Submission deadline: June 5 

  • ACs nominate desk rejects: June 12 -- June 25

  • SACs cross-check desk rejects: June 26 -- July 2

  • Reviewers write reviews: July 6 -- July 24

  • Authors response period: August 7-- August 13

  • ACs & reviewers discussion period: August 14 -- August 24

  • ACs make initial decisions with SACs: August 25 -- September 3

  • PCs & SACs finalize decisions: September 4 -- September 17

  • Notification date: September 25  

General information

  • Please respect deadlines and respond to emails as promptly as possible!

  • It is crucial that we are able to reach you in a timely manner. We will send most emails from CMT (i.e., email@msr-cmt.org). Such emails are sometimes accidentally marked as spam. Please check your spam folder regularly and if you find such an email in there, please whitelist the CMT email address so that you will receive future emails from CMT.

  • If you have changed or plan to change your email address, please update CMT accordingly. We have no way of knowing whether an email sent to you from CMT has bounced, so it is crucial that you make sure that CMT has the correct email address for you at all times. You should also make sure that your domain conflicts in CMT are up to date; these are important for preventing conflicts during the review process.

  • Read what constitutes conflict of interest for NeurIPS 2020.

  • NeurIPS uses the Toronto Paper Matching System (TPMS) and OpenReview to assign submissions to ACs and reviewers. Please log into TPMS and OpenReview to make sure that your profile is up to date. Using a stale profile causes poor assignments, outside your area of expertise.

  • All participants must agree to abide by the NeurIPS code of conduct.

Responsibilities

  • Senior Area Chair (SAC). SACs work alongside the area chairs (ACs) and program chairs (PCs). Each SAC oversees the work of a small number of ACs (around 7), making sure that the reviewing process goes smoothly. SACs serve as the first port of call for ACs if they need assistance or guidance. The reviewing process is double blind at the level of ACs (i.e., ACs cannot see author identities). During the desk rejection phase, the process is also double blind for SACs, but afterwards it is single blinded. SACs are responsible for finalizing the list of desk rejected papers, identifying conflicts of interest (and other unusual activity, such as suspicious bidding patterns) and re-assigning submissions to ACs or reviewers accordingly. SACs are also responsible for helping ACs chase late reviewers, calibrating decisions across ACs, and discussing borderline papers. During the final decision-making phase, SACs will discuss all proposed decisions with the PCs. There is no physical SAC/AC meeting; most discussions with reviewers, ACs, and PCs will take place via CMT or email, with some video conferences toward the end of the reviewing process. Reviewer identities are visible to other reviewers, ACs and SACs. After decisions have been made, reviews and meta-reviews will be made public (but reviewer and SAC/AC identities will remain anonymous).

  • Area Chair (AC). Each area chair (AC) oversees around 20 submissions, making sure that the reviewing process goes smoothly. ACs are the principal contact for reviewers during the whole reviewing process. ACs are responsible for helping the program chairs (PCs) recruit reviewers, recommending reviewers for submissions, chasing late reviewers, facilitating discussions among reviewers, writing meta-reviews, evaluating the quality of reviews, and helping make decisions. Finally, ACs are responsible for flagging papers to desk reject, providing a light motivation which is going to be shared with authors. The reviewing process is double blind at the level of ACs; each AC will work with a senior area chair (SAC), who is responsible for cross checking desk reject recommendations, identifying conflicts of interest (and other unusual activity, such as suspicious bidding patterns) and re-assigning submissions to ACs or reviewers accordingly. SACs serve as the first port of call for ACs if they need assistance or guidance throughout the reviewing process. SACs also calibrate decisions across ACs. There are no physical meetings; most discussions with reviewers, SACs, and PCs will take place via CMT or email, with some video conferences toward the end of the reviewing process. Reviewer identities are visible to other reviewers, ACs and SACs. AC identities are also visible to reviewers. After decisions have been made, reviews and meta-reviews will be made public (but reviewer and SAC/AC identities will remain anonymous).

  • Reviewer. Each reviewer will be assigned around 4--6 submissions to review. Reviewers are responsible for reviewing submissions, reading author responses, discussing submissions and author responses with other reviewers and area chairs (ACs), and helping make decisions. The reviewing process is double blind at the level of reviewers. There are no physical meetings; discussions with other reviewers and ACs will take place via CMT or email. After decisions have been made, reviews and meta-reviews will be made public (but reviewer and AC identities will remain anonymous). As in the previous years, ACs will be asked to evaluate the quality of each review using three scores: “exceeded expectations”, “met expectations,’ and “failed to meet expectations.” The 8% of reviewers with the highest scores will be awarded free NeurIPS registrations. The remaining 30% or so highest-scoring reviewers will have registrations reserved for them (for a limited time frame). The lowest-scoring reviewers may not be invited to review for future conferences.

SAC best practices

  • It is okay to be unavailable for part of the review process (e.g., on vacation for a few days), but if you will be unavailable for more than that -- especially during important windows (e.g., decision-making) -- you must let the program chairs know ASAP.

  • With great power comes great responsibility! Take your job seriously and be fair, particularly this year since you will play a key role in the desk rejection process.

  • If you notice a conflict of interest with a submission that is assigned to one of your ACs, please contact your point of contact program chair immediately. 

  • DO NOT talk to other SACs about submissions assigned to your ACs without prior approval from the program chairs; other SACs may have conflicts with these submissions.

  • DO NOT talk to other SACs or ACs about your own submissions (i.e., submissions you are an author on) or submissions with which you have a conflict of interest.

  • Be professional and listen to the reviewers and ACs, but do not give in to undue influence.

  • If an AC wants to make a decision that is not clearly supported by the reviews, please check that they justify their decision appropriately, including, but not limited to, reading the submission in depth and writing a detailed meta-review that explains their decision.

  • Help calibrate decisions by working closely with your ACs. We will provide you with templates you can use to organize the discussion. Make sure that each AC has the chance to meet with a couple of other ACs to better calibrate. Pay particularly close attention to borderline papers.

AC best practices

  • It is okay to be unavailable for part of the review process (e.g., on vacation for a few days), but if you will be unavailable for more than that -- especially during important windows (e.g., discussion, decision-making) -- you must let your SAC know ASAP.

  • With great power comes great responsibility! Take your job seriously and be fair, particularly this year since you will play a key role in the desk rejection process.

  • If you notice a conflict of interest with a submission that is assigned to you, please contact your SAC immediately so that the paper can be reassigned.

  • Please make sure that every submission in your batch is matched with suitable reviewers whose opinion you can trust on this submission.  It really pays off to invest some time before the reviewing process starts to ensure that your batch has expert reviewers.

  • If two submissions are detected to be sufficiently close by our tools, reviewers will receive both and you will be informed; please ensure that the arguments to discard or accept each submission are thoroughly discussed.

  • DO NOT talk to other ACs about submissions that are assigned to you without prior approval from your SAC; other ACs may have conflicts with these submissions. In general, your primary point of contact for any discussions should be your SAC. Your SAC does not have any conflicts with any of the submissions that are assigned to you.

  • DO NOT talk to other SACs or ACs about your own submissions (i.e., submissions you are an author on) or submissions with which you have a conflict of interest.

  • Be professional and listen to the reviewers, but do not give in to undue influence.

  • Make sure your reviewers read and (if appropriate) respond to all author responses.

  • As soon as the author response is entered in the system, you must initiate a discussion via CMT for each submission and make sure the reviewers engage in the discussion phase. Monitor and moderate the discussion to ensure that it is respectful of everyone’s opinion. Read the papers in your stack to steer the discussion towards the most critical aspects that need discussion. It is your responsibility to moderate and lead the discussion.

  • Read all reviews carefully. After reading a review, please evaluate its quality by indicating (on CMT) whether it “exceeded expectations,” “met expectations,” or “failed to meet expectations.” You should use the information in the reviewer instructions (see the section “Review Content” below), as well as how helpful the review and subsequent discussion by the reviewer were in making your decision about the submission. The top 8% or so highest-scoring reviewers will be awarded free NeurIPS registrations. The remaining 30% or so highest-scoring reviewers will have registrations reserved for them (for a limited time frame). The lowest-scoring reviewers may not be invited to review for future conferences.

  • Your meta-review should explain your decision to the authors. Your comments should augment the reviews, and explain how the reviews, author response, and discussion were used to arrive at your decision. Dismissing or ignoring a review is not acceptable unless you have a good reason for doing so. If you want to make a decision that is not clearly supported by the reviews, perhaps because the reviewers did not come to a consensus, please justify your decision appropriately, including, but not limited to, reading the submission in depth and writing a detailed meta-review that explains your decision. You are expected to discuss such difficult cases with your SAC and to have their approval.

Confidentiality

You must keep everything relating to the review process confidential. Do not use ideas, code and results from submissions in your own work until they become publicly available (e.g., via a technical report or a published paper for ideas/results, via open source for code, etc.). Do not talk about or distribute submissions (whether it is the code, or the ideas and results described in them) to anyone without prior approval from the program chairs. Code submitted for reviewing cannot be distributed.