Example review instructions from AAAI-10

(This message has been copied to all AAAI-10 AIW SPC members for information only.)

Dear AAAI-10 AIW Program Committee Members,

The AAAI-10 AIW paper assignments have been completed, and the review process is about to begin. Your assignments will be mailed to you today, and your assigned papers will be available to download at that time. It is important that we begin to receive your reviews as soon as possible. We assigned PC members no more than 5 papers to review.

Reviewing is double blind. If a paper's authorship is apparent to you AND the authorship is any of the following: you, your current or recent students or advisor, other members of your department, or recent close collaborators, friends, or enemies, please request the paper be reassigned ASAP by sending a message to us at l.m.aroyo@gmail.com, rao@asu.edu, with a copy to aaai10@aaai.org. Otherwise, please try to follow the spirit of the double blind review process and disregard your guess about authorship when reviewing the paper.

The rest of this email has three parts: a summary of the conference review process, specific instructions for obtaining your papers and entering reviews on the Confmaster AIW website, and a copy of the AAAI-10 AIW review instructions and criteria.

 
Overview of AAAI-10 AIW Review Process
=======================================================================
We need your timely participation during these times:
 
   February 5 - March 5: Reviews written
   March 8 - 10: Author feedback period
   March 10 - 18: Reviewer discussion period and responses to authors comments
   March 19: Deadline for ALL final reviews to be entered onto site

=======================================================================
 

The review process begins now! (See below for instructions on downloading papers and entering reviews on the website.)

From 5:00 PM PDT March 8 to 5:00 PM PDT March 10, there will be an author feedback phase where the contact authors of the papers get to see the anonymous reviews and optionally submit a length-limited response.

After the close of the author feedback phase on March 10, discussion and review should take place until Thursday, March 18. Please look at the authors' feedback and respond to any legitimate issues it raises during this time. You are encouraged to discuss any disagreements in review recommendations and/or details. You may be asked to revise your initial review to make it as clear and helpful as possible for the author.
 
Any revisions or additions to reviews must be submitted no later than the end of the discussion period on Friday, March 19 at 5:00PM PDT.

Website Instructions for Submitting Reviews

Please go to

   http://aiweb.confmaster.net

and log on to the system, using the login and password sent to you (or the updated information you have entered into the system).  Click on "View Assigned Papers."
 
After you click on "View Assigned Papers," the list of papers assigned to you will appear, with the following symbols on the far right of the screen:
 "R" stands for: Enter your review
 "Lens" stands for: View paper details, which opens a separate page
 "Disk" stands for: Download this paper
 

Please download and read the papers assigned to you. When you are ready to write your reviews, we suggest that you do the following:

a) In addition to a written review, you are being asked to assign the paper a series of numerical ratings via ConfMaster, the criteria for which are described below. Keeping in mind these criteria, please write your review offline, breaking it into two sections, using the form at the end of this message: comments for the author and confidential comments (for SPC and PC members only). Be sure to save the review offline. This step is extremely important because it will prevent you from losing all of your review due to a technical error in the ConfMaster system, such as not pressing the submit button or being timed out during the time that you are writing your review.

 
b) When you are ready to submit your review, log on to the system and click on "View Assigned Papers." Click on the "R" button. This will take you to the review form for each paper. Select the appropriate radio buttons corresponding to your ratings for the 8 criteria listed below. Then cut and paste the lengthy text portions of your document into the review form.

 
c) Be absolutely certain to press the SUBMIT button at the bottom of the page. IF YOU DO NOT PRESS THIS BUTTON, YOUR REVIEW WILL NOT BE RECORDED.
 
Initial review must be submitted to the ConfMaster site no later than Friday, March 5.
 
If there are any problems with the ConfMaster system, please contact AAAI at aaai10@aaai.org. They will be happy to help in any way they can. We highly encourage submission of a couple of reviews right away to help work through any questions/comments/bugs early.
 
Thank you for agreeing to serve on the AIW Program Committee for AAAI-10. Your help is invaluable in selecting the best papers for the conference. If you have any questions please feel free to contact us.
 

Lora Aroyo and Subbarao Kambhampati

AAAI-10 AIW Program Cochairs
 
**********************************************************************
**********************************************************************

AAAI-10 GENERAL REVIEW CRITERIA GUIDELINE

All AAAI papers should be solid scientific papers, regardless of their specific area. Your review should be constructive, thorough, and polite.

We judge the merit of a paper based on six criteria: Relevance; Significance; Technical soundness; Novelty; Quality of Evaluation; and Clarity. For each criterion you have to
assign a score between 1 and 10. You should also provide detailed comments justifying your evaluation along with suggestions for improving the paper. Furthermore, please provide specific information on which issues you would like the authors to address in their rebuttal.

Below we provide a detailed explanation of the different criteria and scores. While all criteria are important, we want to encourage highly original and novel papers, and therefore would like you to pay special attention to the "Novelty" criterion.

Here is some guidance on the meaning of the criteria and their numerical scores.

Relevance
What is the relevance of this paper to an AI audience?
1: Not relevant
4: Moderately relevant
  7: Relevant to researchers in subarea only
      10: Relevant to general AI

Significance
Are the results important? Are other people (practitioners or researchers) likely to use these ideas or build on them? Does the paper address a difficult problem in a better way than previous research? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions on existing data, or a unique theoretical or pragmatic approach?
    1: Not significant
      4: Moderately significant
       7: Significant
  10: Highly significant

Technical soundness
Is the paper technically sound? Are the concepts correct and accurate?
1: Has major errors
     4: Has minor errors
     7: Technically sound
    10: Major technical contribution

Novelty
Are the problems or approaches novel? Is this a novel combination of familiar techniques? Is it clear how this work differs from previous contributions? Is related work adequately referenced?
  1: Not novel
    4: Moderately novel
     7: Novel

        10: Novel and innovative; will open up new areas of research

Quality of Evaluation
Are claims well-supported by theoretical analysis or experimental results? How convincing is the evidence in support of the conclusions? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work? Is the evaluation appropriate for the contribution?

        1: Not convincing
       4: Moderately convincing
        7: Convincing
   10: Very convincing

Clarity

Is the paper clearly written? Is it well-organized? (If not, feel free to make suggestions to improve the manuscript.) Does it adequately inform the reader? (A well written paper should provide enough information for the expert reader to reproduce its results.)
   1: Poor
4: Satisfactory

        7: Good
10: Excellent

OVERALL SCORE
1. Trivial or wrong or known.
      Clearly below AAAI threshold, I assume no further discussion is needed.

2. A strong rejection.
  I will strongly argue for rejection.

3. A clear rejection.

        I argue for rejection.

4. An OK paper, but not good enough. A rejection.

        This should be rejected, although I would not be upset if it were accepted.

5. Marginally below the acceptance threshold.

        I tend to think it should be rejected it, but having it in the program would not be that bad.

               
6. Marginally above the acceptance threshold.

        I tend to think it should be accepted, but leaving it out of the program would be no great loss.

7. Good paper, accept.

        It should probably be accepted, although I would not be upset if it were rejected.

8. Top 50% of accepted AAAI papers, a very good paper, a clear accept.

        I advocate acceptance of this paper.

9. Top 15% of accepted AAAI papers, an excellent paper, a strong accept.

        I advocate and will fight for acceptance.

10. Top 5% of accepted AAAI papers, a seminal paper for the ages.

        Clearly an outstanding paper. I assume no further discussion is needed.

 
CONFIDENCE SCORE
1: The reviewer's evaluation is an educated guess and it is quite likely that the reviewer did not understand central parts of the paper. Either the paper is not in the reviewer's area, or it was extremely difficult to understand

4: The reviewer is fairly confident that the evaluation is correct. It is possible that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature. Mathematics and other details were not carefully checked.

7: The reviewer is confident but not absolutely certain that the evaluation is correct. It is unlikely but conceivable that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature.

10: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature.

*************************************************************
*************************************************************

AAAI-10 Review Form
--------------------

Comments for the Author:

RELEVANCE:

SIGNIFICANCE:

TECHNICAL SOUNDNESS:

NOVELTY:

QUALITY OF EVALUATION:

CLARITY:
REBUTTAL QUESTIONS:

*************************

Confidential Comments (for SPC/PC):

Main reasons for your recommendation:

      
Other comments (include strong opinions about acceptance or rejection):

     

SHOULD THIS PAPER BE NOMINATED AS AN OUTSTANDING PAPER?____ WHY?