A study of the effects of coaching on correlational validities of tests utilizing verbal analogy items

Date

1968

Journal Title

Journal ISSN

Volume Title

Publisher

Abstract

The study concerned itself with submitting some aptitude tests to coaching on test-taking techniques. Its purpose was to determine whether the tests would be affected by coaching that yielded extrinsic abilities, those which aid performance on the tests but not on criteria. A review of the literature on test theory suggested that this coaching process could reveal superficial or extrinsic correlational validities of aptitude tests. Tests were chosen which utilized verbal analogy items. They were the Differential Aptitude Test--Verbal Reasoning section and a test assembled by the writer, consisting of items sampled randomly from the California Analogies and Reasoning Test, the Terman-McNemar. Test of Mental Ability-Analogies Section, the Wesman Personnel Classification Test, and the 1954 American Council on Education Test for College Freshmen-Verbal Analogies section. The writer hypothesized that scores on these tests would change significantly from pretest to posttest for a coached group compared to no change in scores made by an uncoached group. Ninety-one students were given two forms of the above tests as pretest and posttest. Forty-four students were coached for approximately one hour. The coaching session consisted of two phases. In phase one, the subjects were presented a list of categories and sub-categories of relationships typically found in verbal analogy test items and a list of sample verbal analogies, each of which was matched with a sub-category, illustrating the particular relationship described by the sub-category. The list of categories was studied and the sample analogies were solved and studied according to the type of relationship each represented. Phase one was group activity, directed by the writer. In phase two, the subjects were presented a list of sixteen analogies (one for each subcategory discussed in phase one) and were requested to solve the analogies and to categorize the relationships exemplified by each analogy. The other group of forty-seven students (matched with the coached group on pretest scores) received no instruction between the pretest and posttest; however, in an attempt to hold practice effects constant, they did solve the same list of sixteen verbal analogies the coached group solved in their phase two. When pretest and posttest scores for the coached and uncoached groups were compared using covariance analysis, it was apparent that coaching had no significant effect on Differential Aptitude Test (DAT) or assembled test scores. Both groups scored higher on the posttest, presumably due to practice. The scores of twenty-two coached students who demonstrated considerable facility in performing the categorizing skills in phase two were compared against those of a group of twenty-two uncoached students, matched on pretest scores. Analysis of covariance was used again; this time the DAT posttest scores of the coached subgroup were significantly higher than EAT posttest scores of the uncoached group. This was not the case for the assembled posttest scores. Although the attempt was made to determine the relative degree of difficulty among sets of items typifying the various categories of relationships discussed in the coaching session, this was made Impossible by the inability to control for individual item difficulty across all tests used in the study. It was concluded that the strength of the correlation validity of the EAT is questionable, since scores were affected by coaching yielding extrinsic abilities. Future studies are needed to determine ways to protect the DAT against this type of coaching. Since the assembled tests showed no significant effects of coaching (primarily because the level of difficulty of the tests was too low), it remains for other studies to determine if aptitude tests other than the DAT are influenced by coaching on test-taking techniques.

Description

Keywords

Ability testing

Citation