How to Review Papers 

Loosely adapted from the Interaction Lab page (obsolete page was here) at USC

Here are some (transcribed) tips for paper reviewing from Allan Newell:

When we get a paper to review, at the beginning we should always have as the default that we accept the paper. While reading the paper, we may start raising specific objections along the issues in the review form, namely the goals are not stated, the system is not well described, the approach is not novel or not validated, etc. etc.

Each objection weighs a little against our initial default acceptance. Rejecting a paper is to see if these objections weigh more than our threshold, based on our experience with other other conferences, papers, and advice from the specific conference. (Of course, the review can also raise the initial default acceptance, and then it's even a clearer accept.)

One word of care that I recall: It may happen that we raise either unjustified objections or support to a paper. In particular in AI [dm: actually in any field], it may be rather common that we completely "disagree" or "agree" with the paper's approach/results. We have to be very careful, as much as possible, not to include objections or support that corresponds to subjective, or sometimes dogmatic, opinions about the work.