Comparison of methods for combining incomplete sets of rank order data by computer simulation

The present study was concerned with the problem of combining incomplete sets of rank order data. The problem is frequently encountered in applied psychology and requires the choice of one of several alternative methods to obtain a composite ranking. In the past, this choice was typically based on intuition or on the results of rather sketchy research reported in the literature. The importance of the composite ranking was recognized as it typically served as a criterion for research or institutional decisions. However, a definitive comparison of the error introduced by the computational artifacts of each method with various types of data had not been reported prior to this study. This investigation utilized the Monte Carlo system of computer simulation to generate data in a relatively exhaustive exploration of method performance under various assumptions and parameter settings. The research could be termed semi-empirical in that the validity of computer simulations was judged by comparing selected runs with analogous results utilizing real data. The investigation consisted of five phases summarized below: Phase I: The best method for combining complete sets of rank order data was determined. The composite of complete rankings served as one of the criteria used later in the study. Phase II: The performance of methods in combining incomplete sets of rank order data for various assumptions and parameter settings with 5 judges was compared. The criteria utilized for this phase was the underlying 'true' rank order values and the criterion of 'observed' values described above. Phase III: The variables and assumptions shown to be significant in Phase II were further explored utilizing an increasing number of judges. A rectangular underlying distribution was assumed. The criterion of 'true' underlying rank order values was used throughout this phase. Phase IV: The effects of additional assumptions concerning error variance and number of subjects were explored in this phase. Phase V; Real data were utilized to evaluate the validity of results and provide an example of avoiding faulty application. Contrary to the findings of previous studies, very significant results were consistently obtained under widely varying conditions. The relationships observed between performance of methods and the critical variables of expected mean tau and observation ratio were far more complex than previous studies had indicated. Finally, the performance of the methods recommended by Garrett (1924) and Denenberg and Besco (1961) was shown to be markedly inferior under all conditions with incomplete rankings. The methods of Percentile Ranks and Normalized Ranks performed robustly under all experimental conditions and the performance of Thorndike1s method was shown to be superior under conditions of high expected mean tau between rankings. A decision strategy for selecting the best or an acceptable method based on the results of this research is presented in detail in Chapter V under the subtitle, 'Application of Findings.' This strategy is based on the principle of using the method which best fits the data.