COMPARING THREE RATING FORMATS FOR ERRORS, ACCURACY, AND USER REACTIONS
Sumer, H.C. and Basak Ok, A., Middle East Technical University, Turkey
Three instructor evaluation formats were compared in terms of rating errors (i.e., halo and leniency), accuracy (i.e., consensual validity), and user reactions. The three formats were a traditional Graphic Rating Scale (GRS), a Behaviorally Anchored Rating Scale (BARS), and a Modified Graphic Rating Scale (MGRS). Same performance dimensions were used in all three formats. Different from the traditional GRS format, in the MGRS format, after providing a rating on a given dimension, the raters were required to justify their rating by answering a "Why" questions. The three formats were administered to 395 university students during regular class meetings in a State University, in Ankara, Turkey. The students were asked to evaluate the instructor of the course by using one of the three formats. The MGRS format was expected to be less prone to rating errors and the BARS format was expected to receive more favorable user reactions. None of the formats was found consistently superior to the others. Also, contrary to our expectations, psychometrically, the GRS format was found slightly better than especially the BARS format, and both the GRS and the MGRS formats received more favorable user reactions than did the BARS format. Results seemed to lend support for the use of a GRS format with clear dimensional definitions and scale anchors.