Mechanical TA: Partially Automated High-Stakes Peer Grading.

We describe Mechanical TA, an automated peer review system, and report on our experience using it over three years. Mechanical TA differs from many other peer review systems by involving human teaching assistants (TAs) as a way to assure review quality. Human TAs both evaluate the peer reviews of students who have not yet demonstrated reviewing proficiency and spot check the reviews of students who have. Mechanical TA also features "calibration" reviews, allowing students to quickly gain experience with the peer-review process. We used Mechanical TA for weekly essay assignments in a class of about 70 students, a course design that would have been impossible if every assignment had had to be graded by a TA. We show evidence that it helped to support student learning, leading us to believe that the system may also be useful to others.