IUI 2013 Reviewing Process

Over the past year, I served as co-program chair for Intelligent User Interfaces 2013 with Pedro Szekely. We modified the review process somewhat from previous versions of the conference, and feedback from those attending IUI this year was that the program was strong and interesting.

Over the course of at least two blog posts, I want to write a little bit about the process that we used and to discuss the results of that process.  In this first post, I describe our review process.

Our process consisted of the following steps, closely mimicking the process used by the UIST and CHI conferences.

1. Chairs Recruit SPCs - We invited about 50 people to be on the SPC based on prior program committees, recent accepted papers in the IUI program, and suggestions from various members of the community.  Our goal was to assemble an SPC of a size such that each member would handle approximately 8-10 papers. Choosing the number of papers for each SPC member to handle is a trade-off between workload and decreasing variance in decisions (more submissions handled should lead to less variance), and we chose this range to balance these issues.
2. SPCs Bid for Papers - After the deadline for abstracts, each SPC member was able to specify their conflicts and bid on the papers they most wanted to handle.
3. Chairs Assign Papers to SPCs - We assigned papers using a greedy algorithm where all papers with non-competing SPC bids were assigned first, then papers with multiple bids were given assignments that best balanced workload, and then finally papers with no bids were assigned based on our estimate of the match between paper and SPC member.
4. SPCs Find Reviewers - We did not pre-recruit a program committee to review papers, so SPC members were allowed and encouraged to find the most qualified reviewers to handle each submission. 3 reviewers were required for each paper. There are at least two reasons that we chose this approach: 1) allowing the reviewers to be selected from anyone in the world allows SPC members to bring in expertise from outside the community when needed, and also should ensure that each reviewer is highly qualified to review the paper.  This is especially true when compared to other processes where the set of reviewers is fixed a priori and SPC members must choose the most qualified from among the fixed set. 2) Because the SPC member recruited each reviewer, they should have knowledge of their perspective on the paper and their strengths and weaknesses as a reviewer. Thus the SPC member should be better able to interpret and trust the reviews they receive.
5. Reviewers Submit Reviews - Reviewers had approximately 4 weeks to write reviews, depending on when their SPC member recruited them. We also allowed SPC members to grant additional latitude to their reviewers with the understanding that extensions to reviewers would cut into the time that SPC members would have for writing meta-reviews.
6. SPCs Write Meta-Reviews - Meta-reviews synthesize the reviews written by the external reviewers and describe the points that the SPC member finds compelling both in favor and against the paper's acceptance. SPC members were given approximately 2 weeks to write their meta-reviews, though this period overlapped the US Thanksgiving holiday.
7. Chairs Define Cutoff for Rebuttal - To reduce workload on the SPC members, we decided to reject some papers without rebuttal. These papers were those that received only below neutral or neutral scores. Papers that were allowed to continue to rebuttal received one or more scores of 4 or higher, two or more scores of 3, or an SPC score of 3. This eliminated approximately 50% of papers pre-rebuttal.  
8. Authors Submit Rebuttal - For papers still alive, authors were invited to submit a rebuttal of 5000 characters that focused on correcting any mistakes or misconceptions made by the reviewers. Most, but not all, papers who were eligible submitted a rebuttal.
9. Chairs, SPCs, Reviewers Discuss - Following the submission of rebuttals, reviewers were encouraged to return to the reviewing web site to read the rebuttal, update their reviews, and engage in discussion with the other reviewers, SPC member, and in a few cases ourselves.
10. Pre-SPC Meeting Decisions - Following a week of discussion, we asked each SPC member to make a preliminary decision for each of their papers: accept, discuss, or reject. These decisions were used to determine thresholds for discussion at the PC meeting. A few papers with high scores and accept decisions by their SPC members were automatically accepted and not chosen to be discussed. Similarly, papers with low scores and reject decisions from their SPC member were automatically rejected and not discussed. Remaining papers, those with discuss decisions and/or within a particular score range, were assigned a secondary SPC member to provide an additional opinion and in many cases an additional review. 
11. Two Chair/SPC Telecon Meetings - We conducted two remote meetings to discuss papers and reach final decisions for most, if not all, papers. Two meetings were conducted to reflect the timezone needs of our international SPC; one meeting was scheduled to accomodate the US west coast and asian countries, and the second meeting was scheduled to accomodate the US east coast and Europe (secondary SPC members were assigned with the constraint of ensuring that both SPCs for each paper would be in the same meeting). In most cases, the discussion led to an accept or reject decision. In a few cases, an additional SPC member or a Chair was assigned to provide an additional review and the decision was tabled for later in the week.
12. Final Accepts - After additional reviews came in during the 4 days after the SPC meetings, final decisions were made for the remaining papers.

Overall, I thought the process worked very well. Hurricane Sandy and the US Thanksgiving holiday impacted us a little bit more than I would have liked, slowing down some reviewing and the writing of meta-reviews, which ultimately delayed the release of reviews for rebuttals by a day. 

Some things that worked well:

• The remote SPC meetings seemed to work particularly well, and I think many of the decisions were improved by the discussions over the phone. I was very worried that the technology would get in the way of a productive meeting, but we were able to muddle through fairly well.
• Pre-rebuttal rejections removed a large number of papers from consideration with little noticeable impact on the process. At least one author was happy to be able to revise and resubmit their paper elsewhere earlier than they might have been able to under the normal process.
• SPC bidding provided a lot of information that was highly useful for making the SPC assignments. Thanks to the bidding information, SPC assignments took much less time than I initially anticipated.
• The week gap between the abstract submission deadline and the full paper submission deadline allowed us to have all paper assignments ready to go by the time full papers were submitted. There was little lag between full paper submission and the beginning of the reviewer assignment process.

Were I to do this again, there are a couple things I would change:

• Allowing the review deadline to be flexible and managed between the SPCs and their reviewers made the job of tracking and providing reminders more difficult than I would have liked. I would definitely go with a stricter deadline in the future.
• We initially did not allow for conditional acceptances, although we ultimately allowed a few in the end for papers that clearly described good work but had writing flaws that were too substantial to rely on the authors alone to fix. In the future, the concept of conditional acceptance should be in place from the beginning, so that reviewers and SPC members can take this into account.

Coming up next, some statistics on the results of the reviewing process.