The effects of distributions of unretrieved studies on validity generalization results



Journal Title

Journal ISSN

Volume Title



Validity generalization procedures have been accepted by most Industrial/Organizational psychologists as legitimate methods for removing artifactual variance from distributions of employment test validities. One criticism however, is that validity generalization studies are biased toward published, significant results. To examine this criticism some researchers have applied Rosenthal's (1979) "file-drawer" analysis, often finding that as many as 65,000 unretrieved null studies would have to be located in order to nullify a validity generalization study. The present research notes the differing assumptions of the "file-drawer" analysis and validity generalization and develops a procedure for examining the effects of unretrieved studies on validity generalization results. The analysis was applied to both empirical published data sets as well as simulated data sets. The findings indicate that the new analysis is more conservative than file-drawer analysis and is effective in addressing the basic problem in validity generalization. Additionally, the results show some validity generalization results may be more resistant to unretrieved studies than others.



Employment tests, Personnel management--Research--Statistical methods