The accuracy of graded paired comparison analysis procedures

One area of interest that has not been widely addressed in the graded paired comparison literature is the extent to which the various analysis procedures accurately reflect the underlying perceptions of the rater. The present paper reviewed the literature on the method of paired comparison as applied to performance ratings, critiqued the psychometric advances that have been made in quantifying these ratings and offered two unique linear goal programming (GP) procedures that alleviated the conceptual inconsistencies. A computer simulation study was conducted that tested the accuracy of these analytic procedures under varying ratee sample sizes, rater error rates and dimensionality. Results indicated that the non-GP procedures were superior to the GP procedures and that the author's GP procedures were superior to the other GP procedures. Finally a graded paired comparison pilot study was summarized where these analytic procedures were applied to actual rater data.