NeurIPS 2022 Guidance for Workshop Proposals

(Document updated: )

** Workshop Application Deadline EXTENDED to June 6 11:59pm AoE **

 

With the rapid growth and interest in NeurIPS and its associated workshops, the competition for workshops has increased alongside logistical constraints. In an attempt to mitigate confusion and anxiety regarding what is expected, the workshop chairs have agreed on the following guidance for proposals to hold a NeurIPS workshop in 2022. Organizers of workshop proposals should take care to respect all guidance provided here, and to present explicit answers to the questions implied throughout, as well as explicitly addressing the selection criteria listed below.

Important Dates

1. Workshop Application Open: OPEN NOW

2. Workshop Application Deadline:  Jun 06, 2022 AOE, AoE

3. Workshop Acceptance Notification:  Jul 05, 2022 AOE, AoE

4. Suggested Submission Date for Workshop Contributions: Sep 22, 2022 AOE, AoE

5. Mandatory Accept/Reject Notification Date:  Oct 20, 2022 AOE, AoE

6. Mandatory SlidesLive upload for workshop videos:  Nov 10, 2022 AOE, AoE

Note that the submission date for workshop contributions is suggested, and there is a trade-off between how much time workshops give authors to submit versus reviewers to review in the period between Jul 05, 2022 AOE and Oct 20, 2022 AOE.

The mandatory author notification deadline is Oct 20, 2022 AOE, and the new dates reflect the expected requirements of both in-person and virtual workshops. Workshops that do not meet this deadline will have their speaker tickets withheld, or the workshop may even be canceled if workshop organizers are not responsive after this deadline.

Workshop Format

NeurIPS 2022 plans to hold both in-person and virtual workshops. In order to support the hybrid format of NeurIPS, many aspects of these two formats will be similar.

1. In-person workshops will follow the in-person main conference program on December 2 and December 3. We require that talks appearing at in-person workshops be pre-recorded before the speaker video upload deadline so that virtual attendees have access to them as well. Most in-person workshops will be one-day events that spread 7-9 hours. In-person workshops may request to be two-day events, however, few two-day in-person workshops will be considered, and other applications may be asked to cut down their programs to one day.

2. Virtual workshops will follow the virtual main conference program on December 9. Virtual workshops are required to be within 7 hours of content. We also require that talks in virtual workshops’ programs be pre-recorded before the speaker video upload deadline. Virtual workshops can optionally request a hybrid plan to interact with some local attendees in non-NeurIPS locations.

Workshop Goals

1. Workshops provide an informal, cutting edge venue for discussion of work in progress and future directions. Good workshops have helped to crystallize common problems, explicitly contrast competing frameworks, and clarify essential questions for a subfield or application area.

2. Workshops are a structured means of bringing together people with common interests to form communities. We expect the workshops, in their designated formats, to include some form of community building, and stand apart from other parts of the NeurIPS program, such as Tutorials or Competitions.

 

Selection Criteria

• The degree to which the proposal is focused on an important and topical problem, and the degree to which it is expected that the community will find the workshop interesting, exciting, and useful.

• Diversity and inclusion, in all forms. (See expectations below.)

• The degree to which the proposed program offers an opportunity for discussion.

• Quality of proposed invited speakers. Workshop organizers are encouraged to confirm tentative interest from proposed invited speakers and mention this in their proposal.

• Organizational experience and ability of the team.

• Other dimensions in the expectations below not explicitly listed in these criteria.

• Points of difference. What makes this workshop enticingly different from the hundreds of NeurIPS workshops held previously?

• Details of logistics for the workshop in its preferred format

  • the face-to-face components of an in-person workshop
  • the interactive components of a virtual workshop
  • the sessions within an in-person workshop that can be shared online
  • the proper use of our software stack Zoom + SlidesLive + Topia to facilitate the workshop

Workshop proposal format

No more than three pages of proposal, no more than two pages of organizer information, (optional) discussion about how the workshop could be held in hybrid format for a virtual workshop, and unlimited references.

The three pages (or fewer) for the main proposal must include:

• A title and a brief description of the workshop topic and content.

• A list of invited speakers, if applicable, with an indication of which ones have already agreed and which are indicative.

• An account of the efforts made to ensure diversity of the organizers and speakers (WiML, Black in AI, and LXAI directories, among others, may be a useful resource). Also an account of any efforts to include diverse participants (e.g., via mentoring, subsidies, or the wording and topics in the CFP).

• An estimate of the number of attendees.

• A description of special requirements and technical needs.

• [Optional] If the workshop is requesting a two-day event, a discussion of why a two-day event is preferred.

• If the workshop has been held before, a note specifying how many submissions the workshop received, how many papers were accepted (extended abstract/long format), and how many attendees the workshop attracted.

• A very brief advertisement or tagline for the workshop, up to 140 characters, that highlights any key information you wish prospective attendees to know, and which would be suitable to be put onto a web-based survey (see below).

• A URL for the workshop website.

The two pages (or fewer) for information about organizers must include:

• The names, affiliations, and email addresses of the organizers, with one-paragraph statements of their research interests, areas of expertise, and experience in organizing workshops and related events. Please indicate what other workshops (if any) are concurrently being proposed by an organizer.

• A list of Programme Committee members, with an indication of which members have already agreed. Organizers should do their best to estimate the number of submissions (especially for recurring workshops) in order to (a) ensure a sufficient number of reviewers so that each paper receives 3 reviews, and (b) anticipate that no one is committed to reviewing more than 3 papers. This practice is likely to ensure on-time and more comprehensive and thoughtful reviews.

Hybrid Plan (Optional; one-page max): Workshop organizers who are interested in having an in-person meeting or some hybrid of in-person and online format may propose what aspects would be held in person, the location(s), discuss feasibility from the public health perspective, and logistics (including what support you’d need from the NeurIPS foundation). The NeurIPS foundation and Workshop Chairs may support some of these initiatives if it is safe to do so.

 

Assessment Process and Criteria

The workshop chairs will appoint a number of reviewers who will provide written assessments of the proposals against the criteria listed above. Reviewers reports will be considered by the workshop chairs who will jointly decide upon the selected workshops (subject to the notes on COIs listed below). The final decisions will be made by the workshop chairs via consensus and judgment; we will not simply add up scores assigned to all the criteria.

Hard Constraints/Workshop Requirements

Managing Organizer Conflicts of Interest

Other Guidance and Expectations for Workshop Proposals

Frequently Asked Questions From Past Workshops

Common Pitfalls From Past Workshops (see 2019 workshop summary for more discussion)

2022 Workshop Chairs

Hanie Sedghi, Hsuan-Tien Lin, Sungjin Ahn, and Tristan Naumann
workshop-chairs@neurips.cc

Technical or website questions about workshops go to support@neurips.cc

 

• Mandatory Accept/Reject Notification Deadline before Oct 20, 2022 AOE: By submitting a workshop proposal, workshop organizers commit to notifying those who submit contributions (including talks and posters) to the workshop of their acceptance status before Oct 20, 2022 AOE. A timeline should be included in the proposal that will allow for this. This deadline of Oct 20, 2022 AOE will be published on the NeurIPS main web page and cannot be extended under any circumstances.

• Use CMT or OpenReview for Contributed Work: Workshops that accept contributions must use CMT or OpenReview to manage their submission process. This ensures that accepted submissions can be efficiently uploaded to the NeurIPS.cc site.

• Managing Chair and Reviewer Conflicts of Interest

  1. Workshop chairs cannot be organizers or give invited talks at any workshop. However, they can submit papers and give contributed talks.
  2. Workshop reviewers cannot review any proposal on which they are listed as an organizer or invited speaker, or which they have a conflict of interest with as defined by the NeurIPS guidelines. Moreover, they can not accept invitations to speak at any workshop they have reviewed after the workshop is accepted.
  3. Workshop chairs and reviewers cannot review or shape acceptance decisions about workshops with organizers from within their organization. (For large corporations, this means anyone in the corporation worldwide).

  1. Workshop organizers cannot give talks at the workshops they organize. They can give a brief introduction to the workshop and/or act as a panel moderator.

  2. Workshop organizers should state in their proposal how they will manage conflicts of interest in assessing submitted contributions. At a minimum, an organizer should not be involved in the assessment of a submission from someone within the same organization.

  1. We encourage, and expect, diversity in the organizing team and speakers. This includes the diversity of viewpoints and thinking regarding the topics discussed at the workshop, gender, race, affiliations, seniority, etc. If a workshop is part of a series, the organizer list should include people who have not organized in the past. Organizers should articulate how they have addressed diversity in their proposal in each of these senses.

  2. Since the goal of the workshop is to generate discussion, sufficient time and structure need to be included in the program for this. Proposals should explicitly articulate how they will encourage broad discussion.

  3. Workshop proposals should list explicitly what the problems are they would like to see solved or at least advances made, as part of their workshop. They should explain why these are important problems and how the holding of their proposed workshop will contribute to their solution.

  4. Workshops are not a venue for work that has been previously published in other conferences on machine learning or related fields. Work that is presented at the main NeurIPS conference should not appear in a workshop, including as part of an invited talk. Organizers should make this clear in their calls and explain in their proposal how they will discourage the presentation of already finalized machine learning work.

  5. We encourage workshop submissions of varying lengths and scopes. Organizers should state whether their workshops are meant to be large-attendance talk format or small group presentations. Organizers should articulate what they hope to achieve from the format of the proposal beyond the talks listed.

  6. Workshops should allow for the choice of attendance based on content. Good workshops will put talk titles up publicly before site publication and note the archival status of their submissions. Organizers should articulate how they will do this.

  7. Organizing a workshop is a complex task, and proposals should outline the organizational experience and skills of the proposed organizers (as a team). We encourage junior researchers to be involved in workshop organization but prefer some collective experience in organizing a complex event.

  1. Workshop Series
     We neither encourage nor discourage workshops on topics that have appeared before. Membership of an existing sequence of workshops is irrelevant in the assessment of a workshop proposal (it neither helps nor hinders). Workshop proposals will be evaluated solely on their merits for this year’s conference.

  2. Overlapping Proposals
     We will not forcibly merge proposals. If multiple strong proposals are submitted on similar topics, we will choose a single proposal to accept. We will then reach out to the organizers of the rejected proposals to ask whether they would like us to share their proposals with the organizers of the accepted workshop. The organizers of the accepted workshop may then optionally initiate a merge.

  1. Leaning too heavily on past success
     Proposals for workshops that are part of a series sometimes leaned too heavily on the declared popularity of previous workshops. In some cases, this led to proposals that were less creative and innovative than what we had hoped to see.
  2. Unconfirmed or irrelevant speakers
     The vast majority of proposals included lists of confirmed invited speakers. This made it hard to champion any workshop that didn’t have at least a few speakers confirmed, especially when many unconfirmed big-name speakers were listed (it’s unlikely all would say yes), or when the diversity statement centered on the assumed presence of unconfirmed speakers. There were also several proposals featuring long lists of “celebrity” speakers without clear relevance to the topic of the workshop.
  3. Insufficient time for discussion
     Too many invited speakers—some proposals listed a dozen or more—does not make for a great audience experience, and a workshop with nothing but long-form talks is unlikely to lead to new breakthroughs. We encourage organizers to allocate a larger amount of time to contributed content and open discussion.
  4. Going too big
     We saw only a few proposals that we felt were too narrow, but many we found too broad. There seems to be a tendency to overreach for the sake of going big, while we’d prefer to see more focused workshops.
  5. Too many organizers or too many workshops
     Several proposals had remarkably large organizing committees. It’s not clear why more than five or six organizers would be necessary for a workshop, and it raises concerns about name dropping or organizers added just for an appearance of diversity. Similarly, organizing too many workshops can be a negative factor towards high-quality workshops.
  6. Diversity lip service
     While we were pleased overall by the effort that organizers put into diversity, a lack of diversity in a proposal could be fatal. We were particularly wary of proposals that claimed to be big on diversity while having a full lineup of North American white male speakers or a list of organizers who all recently graduated from the same institution.