Methods of analysis with response-shift bias

Date

1980

Journal Title

Journal ISSN

Volume Title

Publisher

Abstract

Howard and his colleagues have discovered an instrumentation related contaminaton which confounds the results of studies which employ self-report measures in a pre/post or posttest only design. This confounding influence is referred to as response-shift bias. Research has demonstrated that the traditional methods of analysis (i.e., analysis of posttests only, analysis of pre/post difference scores, and analysis of covariance using prescores, ANCOVA) do not consider responseshift bias and produce biased estimates of the treatment effect. A retrospective pre/post design is recommended by Howard and his colleagues to control for response-shift bias. The only method of analysis which yields an unbiased estimate of the treatment effect is posttest minus retrospective pretest difference scores. Other methods of analysis (posttest only, pre/post difference scores, ANCOVA using prescores, and ANCOVA using retrospective prescores) all yield biased estimates of the treatment effect. ANCOVA using prescores traditionally yields an unbiased estimate of the treatment effect and also is the most powerful method of analysis. However, when response-shift bias is present this is not the case. The purpose of the present study is to determine the relative loss in statistical power of the traditional methods of analysis when response-shift bias is present. In addition, the power of ANCOVA using retrospective prescores and the power of the retrospective pre/post difference score method is compared to determine if ANCOVA using retrospective prescores has superior power under certain conditions. Analytic and Monte Carlo techniques were employed to compare the powers of the five methods of analysis under various conditions of sample size, treatment effects, responseshift effects, variances, and correlations between pretests, posttests, and retrospective pretests. Mathematical proofs were used to provide power comparisons between the retrospective pre/post method and the other four methods to establish boundary conditions in which the retrospective pre/post method is more powerful than the other four methods. The results of the analytic and simulation analyses indicate that (1) when there is a response-shift the most powerful method of analysis, overall, is the retrospective pre/post method, and (2) when there is not a response-shift the ANCOVA methods are generally more powerful than the other methods. However, the response-shift studies indicate that the correlation between retrospective presocres and postscores is usually higher than the correlation between prescores and postscores, ANCOVA using retrospective prescores and the retrospective pre/post method would have superior power over the traditional methods. Recommendations and applications to applied research are discussed.

Description

Keywords

Psychoanalysis

Citation