CS 2710 / ISSP 2160: Homework 5
Draft and Revise a Review Paper; Provide Feedback on Review Papers written by your peers
11/15/10: Submit your First Draft
11/29/10: Review the First Drafts of 4 peers
12/08/10: Submit your Final Version
12/08/10: Submit your Back Reviews
12/13/10: Evaluate the Final Version of 4 peers
Via the writing and reading of commentaries used to facilitate
class discussion, you have now both written a number of short reviews of
research papers, and read many reviews written by your
The goal of this assignment is to more formally improve your skills in 1) writing short reviews of research papers, and 2) providing useful
writing feedback to your peers. Both of these skills are very important for a
successful graduate and research career.
For this assignment, everyone will
write a review paper discussing Unifying
SAT-based and Graph-based Planning by Henry Kautz and Bart Selman,
Proceedings International Joint Conferences on Artificial
Intelligence (IJCAI), 1999. First, you will submit a First Draft
of your paper. Second, you and your peers will engage in Peer Review, where everyone will provide feedback about how to
improve the First Draft of others. Third, you will use this feedback to revise
your First Draft, to produce a Final Version.
your peers as well as the instructor will evaluate your Final Version.
Paper submissions and peer reviewing will be done online using the
SWORD system, and will be completely anonymous.
Login and register now as a new SWORD user.
I. First Draft (30 points)
Submit a first draft of your Review Paper, using SWORD.
Your paper should
be approximately 1250-1750 words, and submitted as a pdf file. Make sure you do NOT include your
name anywhere in your paper, so that the peer reviewing process is
As with the commentaries you have done to date, your Review Paper should not
primarily summarize/reiterate the Kautz and Selman paper.
Instead, your review should present your deeper understanding
and include critical analysis. For this homework your
review should in particular focus on the following three points:
Contributions and Importance: A one or two paragraph summary of the
scientific contributions of the Kautz and Selman paper, along with a
critical analysis of the contributions with respect to significance. What problem
does this paper solve? What are the major contributions to the field,
i.e. which important issues in the field does this paper illuminate
Was this paper worth doing,
i.e. is the paper solving an important problem? What are the
strengths as well as the limitations of the solution, particularly
those not already identified by the authors? Would other evaluations
have been more compelling?
Recall from How
to read a research paper that scientific contributions can take on
many forms, depending on the kind of a paper it is.
Comparison of Approaches: In the paper the authors discuss
pure satisfiability planning, pure graph planning, and their unified
approach. Identify one or more planning problems that highlight the
strengths and weaknesses of each approach and use them to illustrate
the strengths as well as the limitations of each solution,
particularly those not already identified by the authors.
Future work: What extensions or future directions do you
see for the ideas in the paper, that are not already mentioned by the
authors? For example, does the method described seem mature enough to
use in real applications? Why or why not? If so, what applications
seem particularly amenable to this approach? What good ideas does the
problem formulation, the solution, the approach or the research method
contain that could be applied to other areas of Artificial
Intelligence? What would be good follow-on projects and why?
Things to consider with respect to your writing:
Make it clear:
Make sure each paragraph has a clear point, and that the ordering of points from paragraph to paragraph makes sense. Consider using section headers. Give concrete examples.
Make logical arguments: You are trying to explain something to your peers. Help them understand why they should believe you. Provide evidence for claims.
Make it interesting: After reading your paper, your peers should feel like they learned something.
II. 4 Peer Reviews (30 points)
Access the 4 First Draft papers assigned to you for peer reviewing, using SWORD.
After reading and evaluating your peers' papers, use SWORD to submit
your 4 peer reviews. Your peer reviews will consist of textual
comments (e.g., provide feedback comments that praise the Contributions
discussion in the First Draft) as well as numerical ratings.
Each textual comment should focus on a single idea, with the most important
comments given first.
Peer reviewing involves providing constructive feedback on your peers'
First Drafts, which your peers will then use to revise (and hopefully
improve) their papers. Generating feedback for others should also
help you improve your general writing skills.
When peer reviewing, ask yourself: did you understand the review, did you believe the arguments, did you learn something?
Other things to consider when peer reviewing:
Be nice: Mention the strengths of your peers' papers, so
people know where they succeeded. Flaming is not helpful.
Be constructive and concrete: Give particular ideas for how to improve
the paper. Don't just complain about a problem; give some ideas for
how to fix it.
Be precise about where
particular problems occur, and give exact examples of problems
Be open-minded about style:
There is no one fixed way these papers have to be written. For example, the paper doesn't have to be formal or informal. Also, do not focus on
low-level writing problems, like typos and simple grammar problems. Please limit these types of comments to issues that are so bad that they make it hard to understand the text.
III. Final Version (30 points)
Access the peer reviews of your First Draft, using SWORD.
Revise your First Draft based on the peer feedback, then use
SWORD to submit your Final Version.
Also use SWORD to submit Back Reviews (evaluations of the peer reviews you received).
To revise your First Draft, it will likely be helpful to first make a list of the things that you will need to change. Try to address all of the issues brought up by your peers. Even if you do not agree with the feedback, your peer was confused by something, so you should try to make your paper clearer.
IV. 4 Peer Evaluations (10 points)
Finally, access the Final Versions of the 4 papers originally assigned to you, using SWORD.
Use SWORD to evaluate whether you think the Final Version
improves over the First Draft.
This assignment borrows liberally from sample peer reviewing assignments kindly provided by Melissa Patchan. For psychological research on peer reviewing, see this
paper. For computational work on enhancing web-based peer review systems, see