Effects Of Item Presentation And User Flexibility On Rater Perceptions

Abstract

The purpose of this study was to examine the effects of item presentation and user flexibility on performance ratings and rater perceptions of the performance rating process. The participants were 252 undergraduate students enrolled in one or more psychology courses at the University of Texas at Arlington. Participants rated the performance of their psychology instructor using four different rating formats: (a) Multiple-item presentation, high user-flexibility, (b) Multiple-item presentation, low user-flexibility, (c) Single-item presentation, high user-flexibility, and (d) Single-item presentation, low user-flexibility. Dependent measures were administered within each performance rating format to obtain ratings of (a) instructor performance, (b) rater confidence in the accuracy of performance ratings, and (c) rater satisfaction with the performance rating process. At the end of the study, participants were asked to choose their most preferred rating format. Item presentation and user flexibility did not significantly affect performance ratings or rater perceptions; however rater preference were found to significantly differ across the four rating formats. The proportion of raters selecting the multiple-item presentation, high user-flexibility rating format was significantly higher than the other three rating formats, and the proportion of raters selecting the single-item presentation, low user-flexibility rating format was significantly lower than the other three rating formats.