Skip to yearly menu bar Skip to main content


Reviewer Guidelines

Frequently asked questions

Frequently asked questions can be found here.

Contacting the program chairs

If you encounter a situation that you are unable to resolve with your AC, please contact the program chairs at neurips2020pcs@gmail.com . Please refrain from writing to the program chairs at their own email addresses.

Introduction

Thank you for agreeing to serve for NeurIPS 2020! 

This page provides an overview of reviewer responsibilities and key dates.

Key dates

  • Abstract Submission deadline: May 27 (4pm EDT; 8pm UTC)
  • Paper Submission deadline: June 5 (4pm EDT; 8pm UTC)
  • ACs nominate desk rejects: June 12 -- June 25
  • SACs cross-check desk rejects: June 26 -- July 2
  • Reviewers write reviews: July 6 -- July 24
  • Authors response period: August 7-- August 13
  • ACs & reviewers discussion period: August 14 -- August 24
  • ACs make initial decisions with SACs: August 25 -- September 3
  • PCs & SACs finalize decisions: September 4 -- September 17
  • Notification date: September 25  

General information

  • Please respect deadlines and respond to emails as promptly as possible!

  • It is crucial that we are able to reach you in a timely manner. We will send most emails from CMT (i.e., email@msr-cmt.org). Such emails are sometimes accidentally marked as spam. Please check your spam folder regularly. If you find such an email in there, please whitelist the CMT email address so that you will receive future emails from CMT.

  • If you have changed or plan to change your email address, please update CMT accordingly. We have no way of knowing whether an email sent to you from CMT has bounced, so it is crucial that you make sure that CMT has the correct email address for you at all times. You should also make sure that your domain conflicts in CMT are up to date; these are important for preventing conflicts during the review process.

  • Read what constitutes conflict of interest for NeurIPS 2020.

  • NeurIPS uses the Toronto Paper Matching System (TPMS) and OpenReview to assign submissions to ACs and reviewers. Please log into TPMS and OpenReview to make sure that your profile is up to date. Using a stale profile causes poor assignments that may be outside your area of expertise.

  • All participants must agree to abide by the NeurIPS code of conduct.

Responsibilities

  • Unless you have specified a smaller quota, each reviewer will be assigned around 4-6 submissions. Reviewers are responsible for reviewing submissions, reading author responses, discussing submissions and author responses with other reviewers and area chairs (ACs), and helping make decisions. The reviewing process is double blind at the level of reviewers and ACs. There are no physical meetings; discussions with other reviewers and ACs will take place via CMT. Reviewer identities are visible to other reviewers, ACs and SACs. After decisions have been made, reviews and meta-reviews will be made public (but reviewer and SAC/AC identities will remain anonymous).

  • As soon as reviewers are notified of papers to review, reviewers are expected to log in to CMT to check for conflicts and to check that papers fall within the area of expertise. If reviewers are not comfortable reviewing a paper, they need to fill out a note in CMT and communicate to their AC immediately, because it takes time to find an alternate qualified reviewer.

  • ACs will be asked to evaluate the quality of each review using three scores: “exceeded expectations”, “met expectations,’ and “failed to meet expectations.” The 8% of reviewers with the highest scores will be awarded free NeurIPS registrations. The remaining top 30% highest-scoring reviewers will have registrations reserved for them (for a limited time frame). The lowest-scoring reviewers may not be invited to review for future conferences.

Reviewer best practices

  • It is okay to be unavailable for part of the review process (e.g., on vacation for a few days), but if you will be unavailable for more than that -- especially during important windows (e.g., discussion) -- you must let your ACs know.

  • With great power comes great responsibility! Take your job seriously and be fair.

  • Write thoughtful and constructive reviews. Your reviews must accord with the NeurIPS code of conduct. Although the double-blind review process reduces the risk of discrimination, reviews can inadvertently contain subtle discrimination, which should be actively avoided.

    • Example: avoid comments regarding English style or grammar that may be interpreted as implying the author is "foreign" or "non-native". So, instead of "Please have your submission proof-read by a native English speaker,” use a neutral formulation such as "Please have your submission proof-read for English style and grammar issues.”

  • DO NOT talk to other reviewers, ACs, or SACs about submissions that are assigned to you without prior approval from your AC; other reviewers, ACs, and SACs may have conflicts with these submissions. In general, your primary point of contact for any discussions should be the corresponding AC for that submission.

  • DO NOT talk to other reviewers, ACs, or SACs about your own submissions (i.e., submissions you are an author on) or submissions with which you have a conflict.

  • Keep papers assigned to you in absolute confidentiality.

  • Be professional, polite, and listen to the other reviewers, but do not give in to undue influence.

  • Engage actively in the discussion phase for each of the submissions that you are assigned, even if you are not specifically prompted to do so by the AC.

  • It is not fair to dismiss any submission without having thoroughly read it. Think about the times when you received an unfair, unjustified, short, or dismissive review. Try not to be that reviewer! Always be constructive and help the authors understand your viewpoint, without being dismissive or using inappropriate language. If you need to cite existing work to justify one of your comments, please be as precise as possible and give a complete citation.

  • If you would like the authors to clarify something during the author response phase, please articulate this clearly in your review (e.g., “I would like to see results of experiment X” or “Can you please include details about the parameter settings used for experiment Y”). 

 

Reviewer Instructions

Online submission system (CMT)

All reviews must be submitted via the NeurIPS 2020 CMT site. You may visit the site multiple times and revise your reviews as often as necessary. If you are both an author and a reviewer, please use the same email address for both roles in CMT. During the reviewing process, you will receive many emails from CMT (i.e., email@msr-cmt.org). Such emails are sometimes accidentally marked as spam. Please check your spam folder regularly and if you find such an email in there, please whitelist the CMT email address so that you will receive future emails from CMT.

Never reply directly to emails sent by CMT. Instead, log in and use the  “email meta-reviewer” and “email senior meta-reviewer” link to send emails.

Confidentiality

You must keep everything relating to the review process confidential. Do not use ideas, code and results from submissions in your own work until they become publicly available (e.g., via a technical report or a published paper for ideas/results, via open source for code, etc.). Do not talk about or distribute submissions (whether it is the code or the ideas and results described in them) to anyone without prior approval from the program chairs. Code submitted for reviewing cannot be distributed. 

Double-blind reviewing

The reviewing process will be double blind at the level of reviewers and ACs (i.e., reviewers and ACs cannot see author identities) but not at the level of SACs and program chairs. Authors are responsible for anonymizing their submissions. In particular, they should not include author names, author affiliations, or acknowledgements in their submissions and they should avoid providing any other identifying information (even in the supplementary material). If you are assigned a submission that is not adequately anonymized (e.g., includes author names, author affiliations, acknowledgements, or other identifying information) then please contact the corresponding AC. Under no circumstances should you attempt to find out the identities of the authors for any of your assigned submissions (e.g., by searching on Google or arXiv). If you accidentally find out, please do not divulge the identities to anyone. You should not let the authors’ identities influence your decision in any way.

Supplementary material

Authors may submit up to 100MB of supplementary material, such as proofs, derivations, data, or source code; all supplementary material must be in PDF or ZIP format. Your responsibility as a reviewer is to read and review the submission itself; looking at supplementary material is at your discretion. That said, NeurIPS submissions are short, so you may wish to look at supplementary material before criticizing a submission for insufficient details, proofs, or experimental results.

Executing Code & Clicking on Links

Please remember that just like any other untrusted code, any submitted code may contain security vulnerabilities. If you are planning to run the supplementary source code, please make sure you are doing this in a secure environment because this code is not vetted by our submission system. We recommend running source code (1) inside a Docker container, or (2) a Virtual Machine image (using VirtualBox or VMWare), or (3) on a network-isolated cloud instance.

You may wish to also be cautious about accessing other web links provided from the paper, as these may contain vulnerabilities or may log visitor IP addresses.

Formatting instructions

Submissions are limited to eight content pages, including all figures and tables, in the NeurIPS “submission” style; additional pages containing only a) discussion of broader impact and b) references are allowed. Authors must use the NeurIPS 2020 LaTeX style file. If you are assigned any submissions that violate the NeurIPS style (e.g., by decreasing margins or font size) or page limits, please notify the AC immediately.

Dual submissions

Submissions that are identical or substantially similar to papers that are in submission to, have been accepted to, or have been published in other archival conferences, journals, workshops, etc. should be deemed dual submissions. Submissions that are identical or substantially similar to other NeurIPS submissions should also be deemed dual submissions; submissions should be distinct and sufficiently substantial. Slicing contributions too thinly may be sufficient for submissions to be deemed dual submissions. If you suspect that a submission that has been assigned to you is a dual submission or if you require further clarification, please contact the corresponding AC. For more information about dual submissions, please see the Call for Papers and FAQ.

Review content

We know that serving as a reviewer for NeurIPS is time consuming, but the community needs outstanding people like yourself to uphold the scientific quality of NeurIPS. Review content is the primary means by which ACs, SACs, and program chairs make decisions about submissions. Please make your review as detailed and informative as possible; short, superficial reviews that venture uninformed opinions or guesses are worse than no review since they may result in the rejection of a high-quality submission.

Review content is also the primary means by which authors understand their submissions’ decisions. Reviews for rejected submissions help authors understand how to improve their work for other conferences or journals. Reviews for accepted submissions help authors understand how to improve their work for the camera-ready versions.

The review form will ask you for the following:

1. Summary and contributions: Briefly summarize the paper and its contributions

Summarize the paper motivation, key contributions and achievements in a paragraph.

Although this part of the review may not provide much new information to authors, it is invaluable to ACs, SACs, and program chairs, and it can help the authors determine whether there are misunderstandings that need to be addressed in their author response.

There are many examples of contributions that warrant publication at NeurIPS.  These contributions may be theoretical, methodological, algorithmic, empirical, connecting ideas in disparate fields (“bridge papers”), or providing a critical analysis (e.g., principled justifications of why the community is going after the wrong outcome or using the wrong types of approaches.).  

For more examples of what is intended for this question, see section Examples of Review Content of this guide.

2. Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the NeurIPS community.

List the strengths of the submission. For instance, it could be about the soundness of the theoretical claim or the soundness of empirical methodology used to validate an empirical approach. Another important axis is the significance and the novelty of the contributions relative to what has been done already in the literature, and here you may want to cite these relevant prior works. One measure of the significance of a contribution is (your belief about) the level to which researchers or practitioners will make use of or be influenced by the proposed ideas. Solid, technical papers that explore new territory or point out new directions for research are preferable to papers that advance the state of the art, but only incrementally. Finally, a possible strength is the relevance of the line of work for the NeurIPS community.

Clarification of these axes of evaluation, as well as example quotes from past NeurIPS reviews, can be found in the section Examples of Review Content of this guide.

3. Weaknesses: Explain the limitations of this work along the same axes as above.

This is like above, but now focussing on the limitations of this work.

Your comments should be detailed, specific, and polite. Please avoid vague, subjective complaints. Think about the times when you received an unfair, unjustified, short, or dismissive review. Try not to be that reviewer! Always be constructive and help the authors understand your viewpoint, without being dismissive or using inappropriate language. Remember that you are not reviewing your level of interest in the submission, but its scientific contribution to the field!

4. Correctness: Are the claims and method correct? Is the empirical methodology correct?

Explain if there is anything incorrect with the paper. Incorrect claims or methodology are the primary reason for rejection. Be as detailed, specific and polite as possible. Thoroughly motivate your criticism so that authors will understand your point of view and potentially respond to you.

5. Clarity: Is the paper well written?

Rate the clarity of exposition of the paper. Give examples of what parts of the paper need revision to improve clarity.

6. Relation to prior work: Is it clearly discussed how this work differs from previous contributions?

Explain whether the submission is written with the due scholarship, relating the proposed work with the prior work in the literature. The related work section should not just list prior work, but explain how the proposed work differs from prior work appeared in the literature.

Note that authors are excused for not knowing about all non-refereed work (e.g, those appearing on ArXiv). Papers (whether refereed or not) appearing less than two months before the submission deadline are considered contemporaneous to NeurIPS submissions; authors are not obligated to make detailed comparisons to such papers (though, especially for the camera ready versions of accepted papers, authors are encouraged to).

7. Reproducibility: Are there enough details to reproduce the major results of this work?

Mark whether the work is reasonably reproducible. If it is not, lack of reproducibility should be listed among the weaknesses of the submission.

8. Additional feedback, comments, suggestions for improvement and questions for the authors

Add here any additional comment you might have about the submission, including questions and suggestions for improvement.

9.  Overall score:

You should NOT assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. The “Overall Score” for each submission should reflect your assessment of the submission’s contributions.

10: Top 5% of accepted NeurIPS papers. Truly groundbreaking work.

 9: Top 15% of accepted NeurIPS papers. An excellent submission; a strong accept.

 8: Top 50% of accepted NeurIPS papers. A very good submission; a clear accept.

 7: A good submission; accept.

     I vote for accepting this submission, although I would not be upset if it were rejected.

6: Marginally above the acceptance threshold.

     I tend to vote for accepting this submission, but rejecting it would not be that bad.

5: Marginally below the acceptance threshold.

     I tend to vote for rejecting this submission, but accepting it would not be that bad.

4: An okay submission, but not good enough; a reject.

     I vote for rejecting this submission, although I would not be upset if it were accepted.

3: A clear reject.

     I vote and argue for rejecting this submission.

2: I'm surprised this work was submitted to NeurIPS; a strong reject.

1: Trivial or wrong or already known.

10.  Confidence score:

5: You are absolutely certain about your assessment. You are very familiar with the related work.

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

3: You are fairly confident in your assessment.  It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

2: You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

11. Have the authors adequately addressed the broader impact of their work, including potential negative ethical and societal implications of their work?

Yes, no or only partially. In order to provide a balanced perspective, authors are required to include a statement of the potential broader impact of their work, including its ethical aspects and future societal consequences. Authors should take care to discuss both positive and negative outcomes. Indicate whether you believe the broader impact section was adequate.

12. Does the submission raise potential ethical concerns? This includes methods, applications, or data that create or reinforce unfair bias or that have a primary purpose of harm or injury. If so, please explain briefly.

Yes or No. Explain if the submission might raise any potential ethical concern. Note that your rating should be independent of this. If the AC also shares this concern, dedicated reviewers with expertise at the intersection of ethics and ML will further review the submission. Your duty here is to flag only papers that might need this additional revision step.

14. Have you previously reviewed or area chaired (a version of) this work for another archival venue?

Yes or No.  This information will be useful to ACs and SACs in putting your review details in context of having already seen an earlier version. ACs and SACs have access to a resubmission statement by the authors which declares whether the submission was previously rejected and what changes were made to the current version. The AC may decide to share this information with reviewers if needed.

15. Agree to abide by the NeurIPS code of conduct

The NeurIPS code of conduct can be found here: https://neurips.cc/public/CodeOfConduct

16. Confidential comments for the area chair

If you have comments that you wish to be kept confidential from the authors, you can use the “Confidential Comments to Area Chair” text field. Such comments might include explicit comparisons of the submission to other submissions and criticisms that are more bluntly stated. If you accidentally find out the identities of the authors, please do not divulge the identities to anyone.

Author response

Authors will be given the opportunity to respond to their reviews before decisions are made. This is to enable them to address misunderstandings, point out parts of their submissions that were overlooked, or disagree with the reviewers’ assessments. In previous years, some authors felt that their responses were ignored. As a reviewer, it is your responsibility to read and (if appropriate) respond to each author response. It is not fair to ignore any author response, even for submissions that you think should be rejected. Although it is possible that an author response will not change your assessment of a submission, you must convey to the authors that you have carefully considered their comments. As you read each author response, keep an open mind. Have you overlooked something? Please update each review to indicate that you have read the author response and whether you agree or disagree with it. You should be more specific than “I have read the author response and my opinion remains the same.” If that is the case, you should explain why it remains it remains the same, what the author response failed to address, etc.

Discussion

After the author response phase, the AC for each submission will initiate a discussion via CMT to encourage the reviewers to come to a consensus. If the reviewers do come to a consensus, the program chairs will take it seriously; only rarely are unanimous assessments overruled. The discussion phase is especially important for borderline submissions and submissions where the reviewers’ assessments differ; most submissions fall into one or another of these categories, so please take this phase seriously. When discussing a submission, try to remember that different people have different backgrounds and different points of view. Ask yourself, “Do the other reviewers' comments make sense?" and do consider changing your mind in light of their comments, if appropriate. That said, if you think the other reviewers are not correct, you are not required to change your mind. Reviewer consensus is valuable, but it is not mandatory.

Examples of Review Content

Contributions:

The following are examples of contributions a paper might make.  This list is not exhaustive.

“The paper provides a thorough experimental validation of the proposed algorithm, demonstrating much faster runtimes without loss in performance compared to strong baselines.”

“The paper proposes an algorithm for [insert] with computational complexity scaling linearly in the observed dimensions; in contrast, existing algorithms scale cubicly.”

“The paper presents a method for robustly handling covariate shift in cases where [insert assumptions], and demonstrated the impact on [insert application].”

“The authors provide a framework that unifies [insert field A] and [insert field B], two previously disparate research areas.”

“This paper demonstrates how the previously popular approach of [insert] has serious limitations when applied to [insert].”

“The authors propose a new framework for quantifying fairness of ML algorithms.”

“The authors show how the definition of fairness in [insert citation] fails to capture [insert], which is a critical example of its failure mode.”

Quality: Is the submission technically sound? Are claims well supported by theoretical analysis or experimental results? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

Example from nips30/reviews/1548.html

“The technical content of the paper appears to be correct albeit some small careless mistakes that I believe are typos instead of technical flaw (see #4 below).

…

4. The equation in line 125 appears to be wrong. Shouldn't there be a line break before the last equal sign, and shouldn't the last expression be equal to E_q[(\frac{p(z,x)}{q(z)})^2]?”

“The idea of having a sandwich bound for the log-marginal likelihood is certainly good. While the authors did demonstrate that the bound does indeed contain the log-marginal likelihood as expected, it is not entirely clear that the sandwich bound will be useful for model selection. This is not demonstrated in the experiment despite being one of the selling point of the paper. It's important to back up this claim using simulated data in experiment.”

Example from OpenReview

“Technical issues: The move from (1) to (2) is problematic. Yes it is a lower bound, but by igoring H(Z), equation (2) ignores the fact that H(Z) will potentially vary more significantly that H(Z|Y). As a result of removing H(Z), the objective (2) encourages Z that are low entropy as the H(Z) term is ignored, doubly so as low entropy Z results in low entropy Z|Y. Yes the -H(X|Z) mitigates against a complete entropy collapse for H(Z), but it still neglects critical terms. In fact one might wonder if this is the reason that semantic noise addition needs to be done anyway, just to push up the entropy of Z to stop it reducing too much. In (3) arbitrary balancing parameters lamda_1 and lambda_2 are introduced ex-nihilo - they were not there in (2). This is not ever justified. Then in (5), a further choice is made by simply adding L_{NLL} to the objective. But in the supervised case, the targets are known and so turn up in H(Z|Y). Hence now H(Z|Y) should be conditioned on the targets. However instead another objective is added again without justification, and the conditional entropy of Z is left disconnected from the data it is to be conditioned on. One might argue the C(X,Y,Z) simply acts as a prior on the networks (and hence implicitly on the weights) that we consider, which is then combined with a likelihood term, but this case is not made. In fact there is no explicit probabilistic or information theoretic motivation for the chosen objective. Given these issues, it is then not too surprising that some further things need to be done, such as semantic noise addition to actually get things working properly. It may be the form of noise addition is a good idea, but given the troublesome objective being used in the first place, it is very hard to draw conclusions. In summary, substantially better theoretical justification of the chosen model is needed, before any reasonable conclusion on the semantic noise modelling can be made.”

Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note: a superbly written paper provides enough information for an expert reader to reproduce its results.)

Example from /nips30/reviews/1548.html

“While the paper is pretty readable, there is certainly room for improvements in the clarity of the paper. I find paragraphs in section 1 and 2 to be repetitive. It is clear enough from the Introduction that the key advantages of CHIVI are the zero avoiding approximations and the sandwich bound. I don't find it necessary to be stressing that much more in section 2. Other than that, many equations in the paper do not have numbers. The references to the appendices are also wrong (There is no Appendix D or F). There is an extra period in line 188.

The Related Work section is well-written. Good job!”

Example from /nips30/reviews/1173.html

“The paper is generally well-written and structured clearly. The notation could be improved in a couple of places. In the inference model (equations between ll. 82-83), I would suggest adding a frame superscript to clarify that inference is occurring within each frame, e.g. q_{\phi}(z_2^{(n)} | x^{(n)}) and q_{\phi}(z_1^{(n)} | x^{(n)}, z_2^{(n)}). In addition, in Section 3 it was not immediately clear that a frame is defined to itself be a sub-sequence.”

Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? Is it clear how this work differs from previous contributions? Is related work adequately cited? (Abstracts and links to many previous NeurIPS papers are available here.)

Example from /nips30/reviews/60.html

“The main contribution of this paper is to offer a convergence proof for minimizing sum fi(x) + g(x) where fi(x) is smooth, and g is nonsmooth, in an asynchronous setting. The problem is well-motivated; there is indeed no known proof for this, in my knowledge.

…

There are two main theoretical results. Theorem 1 gives a convergence rate for proxSAGA, which is incrementally better than a previous result. Theorem 2 gives the rate for an asynchronous setting, which is more groundbreaking.”

Example from /nips30/reviews/1173.html

“The paper is missing a related work section and also does not cite several related works, particularly regarding RNN variants with latent variables (Fraccaro et al. 2016; Chung et al. 2017), hierarchical probabilistic generative models (Johnson et al. 2016; Edwards & Storkey 2017) and disentanglement in generative models (Higgins et al. 2017). The proposed graphical model is similar to that of Edwards & Storkey (2017), though the frame-level Seq2Seq makes the proposed method sufficiently original. The study of disentanglement for sequential data is also fairly novel.”

Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

Example from /nips30/reviews/688.html

“I liked this article very much. It answers a very natural question: gradient descent is an extremely classical, and very simple algorithm. Although it is known not to be the fastest one in many situations, it is widely used in practice; we need to understand its convergence rate. The proof is also conceptually simple and elegant, and I found its presentation very clear.”

Example from /nips30/reviews/3278.html

”This paper seems to be a useful contribution to the literature on protein docking, showing a modest improvement over the state of the art. As such, I think the paper would be well-suited for publication in a molecular biology venue, or perhaps as an application paper at NIPS. The main weakness of the paper in my view is that it is a fairly straightforward application of an existing technique (GCNs) to a new domain (plus some feature engineering). As such I am leaning towards a rejection for NIPS.”

Please comment on and take into account the strengths of the submission. It can be tempting to only comment on the weaknesses; however, ACs, SACs, and program chairs need to understand both the strengths and the weaknesses in order to make an informed decision. It is useful for the ACs, SACs, and program chairs if you include a list of arguments for and against acceptance. If you believe that a submission is out of scope for NeurIPS, then please justify this judgement appropriately, including, but not limited to, looking at subject areas and previous NeurIPS papers.  If you need to cite existing work, please be as precise as possible and give a complete citation.

Example from /nips30/reviews/587.html

“There are several things to like about this paper:

- The problem of safe RL is very important, of great interest to the community and without too much in the way of high quality solutions.

- The authors make good use of the developed tools in model-based control and provide some bridge between developments across sub-fields.

- The simulations support the insight from the main theoretical analysis, and the algorithm seems to outperform its baseline.

However, I found that there were several shortcomings:

- I found the paper as a whole a little hard to follow and even poorly written as a whole. For a specific example of this see the paragraph beginning 197.

- The treatment of prior work and especially the "exploration/exploitation" problem is inadequate and seems to be treated as an afterthought: but of course it is totally central to the problem! Prior work such as [34] deserve a much more detailed discussion and comparison so that the reader can understand how/why this method is different.

- Something is confusing (or perhaps even wrong) about the way that Figure 1 is presented. In an RL problem you cannot just "sample" state-actions, but instead you may need to plan ahead over multiple timesteps for efficient exploration.

- The main theorems are hard to really internalize in any practical way, would something like a "regret bound" be possible instead? I'm not sure that these types of guarantees are that useful.

- The experiments are really on quite a simple toy domain that didn't really enthuse me.”

Example from https://openreview.net/forum?id=SkkTMpjex¬eId=rkgMSRKrx

“The main contributions of the paper are:

1) Distributed variant of K-FAC that is efficient for optimizing deep neural networks. The authors mitigate the computational bottlenecks of the method (second order statistic computation and Fisher Block inverses) by asynchronous updating.

2) The authors propose a “doubly-factored” Kronecker approximation for layers whose inputs are too large to be handled by the standard Kronecker-factored approximation. They also present (Appendix A) a cheaper Kronecker factored approximation for convolutional layers.

3) Empirically illustrate the performance of the method, and show:

- Asynchronous Fisher Block inversions do not adversely affect the performance of the method (CIFAR-10)

- K-FAC is faster than Synchronous SGD (with and without BN, and with momentum) (ImageNet)

- Doubly-factored K-FAC method does not deteriorate the performance of the method (ImageNet and ResNet)

- Favorable scaling properties of K-FAC with mini-batch size

Pros:

- Paper presents interesting ideas on how to make computationally demanding aspects of K-FAC tractable.

- Experiments are well thought out and highlight the key advantages of the method over Synchronous SGD (with and without BN).

Cons:

- “…it should be possible to scale our implementation to a larger distributed system with hundreds of workers.” The authors mention that this should be possible, but fail to mention the potential issues with respect to communication, load balancing and node (worker) failure. That being said, as a proof-of-concept, the method seems to perform well and this is a good starting point.

- Mini-batch size scaling experiments: the authors do not provide validation curves, which may be interesting for such an experiment. Keskar et. al. 2016 (On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima) provide empirical evidence that large-batch methods do not generalize as well as small batch methods. As a result, even if the method has favorable scaling properties (in terms of mini-batch sizes), this may not be effective.”