Advice for Graders#

Graders should very carefully follow the rubric guidelines for scoring. It is important to use the full scale. Typically this grading scheme is devised to create the biggest signal between students doing high-quality work and students doing low-quality work. It is important that the numbers significantly reflect the quality of work.

If manual review grading is enabled, then review grading is especially important. Review grades are especially impactful on the first few assignments where students are learning how to to peer review. It must be the case that insufficient or low quality reviews get low grades. For the first assignments, it is also important to give very clear comments on what is wrong to the peer reviewers. Effort spent in on the first few assignments of giving the peers good feedback will translate to everything working more smoothly later in the course, when the peers are providing each other high quality reviews and grading tasks will be much easier. For a smooth grading experience, the following guidelines should be adhered to:

  • Use a 10 point scale for all rubric elements.

  • Design a rubric which uses most of the grading range. For example, on a 10 point scale, it is recommended that an 8 is a “solid submission” while 10 is reserved for the few submissions that are “truly excellent”.

  • Give your graders clear instructions on the grading scale.

  • Clearly communicate the grading range to the students.

  • Give open ended assignments with a peer reviewing rubric that require critical thinking of the reviewer, rather than checking off items of a list of correct answers.

  • Enable manual grading of peer reviews by creating the review assignment with a rubric.

  • (For manual grading of peer reviews) Give the graders clear instructions on how to grade peer reviews. Specifically, scores should reward the appropriate amount of effort, and penalize insufficient effort. The grader should give written comments, especially on the first few assignments, to help the peer reviewers to provide more helpful and accurate reviews.

  • Grading can be done very quickly and thus students can get feedback very quickly. It is best to have graders complete their grading the day after the peer reviews are due.

  • The instructor (or a grader) will need to manually run the matching, check-augment-and-archive, and grading algorithms. This should be done very shortly after the appropriate deadlines.

  • It is difficult to accept late submissions and also keep the fast turn-around of grades that is possible with peer grading, choose a late policy accordingly.

Submission and Review Reports#

The feedback that the peers will see include the following (shown below):

  • Their original peer review of the submission.

  • The grader’s review of the submission.

  • The grader’s review of the peer’s review (if manual review grading is enabled).

_images/advice_for_graders_1.png

The review report with manual review grading. Mouse-over “[?]” opens written feedback on each rubric element. This particular review rubric has three components “Advice”, “Evaluation”, and “Score”.#

As noted above, the peer reviewers will be able to see your scores and comments on the submission. Thus, there is no need to duplicate, in your comments to a peer, the comments you made to the submitter. The feedback to the submitter of the submission includes the following:

  • The reviews of all peers.

  • The review of the grader (if there is one).

The submitter does not see the grader’s review of the peers’ reviews.

Manual Scoring of Peer Reviews#

One reason to manually score peer reviews is that sometimes the written feedback is on the right track, but there are multiple scores that could be reasonable. In this case, manual scoring allows the grader of the peer reviews to assign the right grade. Specifically, some submissions are very difficult to review and these reviews should be scored leniently. On the other hand, typical good quality submissions are pretty easy to review, and these grading tasks do not measure the graders ability very well. The following guidelines are recommended in the context of scoring on a 10 point scale.

The scoring philosophy is that a good peer review should get 8 (of 10). An excellent review should get 10. A review that ok, but subpar, should get a 6. Per the above discussion of difficulty of review tasks varying from submission to submission, how to evaluate excellence or inferiority of a review depends on the submission.

Scoring Guidelines#

  • Excellent submissions (e.g., 10s): Reviews that note excellence should get an 8. Go up if the review adds great feedback or discussion that matches the excellence of the submission.

  • Pretty good submissions (e.g., 8s): Reviews that accurately score at 8ish without much discussion should get an 8. Go up if the review adds great feedback or discussion, especially with recommendations that would have made the submission excellent.

  • Minor-issue submissions (e.g., 6s): Reviews that accurately point out the issues and score accordingly should get a 9. Go up if the review adds great feedback or discussion, especially with recommendations that would have made the submission better. Reviews that consider the submission good and do note observe that there are issues should be scored at 7 or below. Reviews that consider the submission to be excellent shoud be scored at 5 or below.

  • Serious-issue submissions (e.g., 2-4s): Score at 8 for reviews that rightly observe the submission has issues. Score at 9 or 10 for concretely idenfied issues and suggestions. Reviews that score about 7 should be scored below 6, the higher the submission score, the lower the review score. These are peers that are not paying attention.

  • Trivial submissions (e.g., 0s): Score at 8.