An investigation of criterion-referenced pre-assessment instruments developed from the TABS table of specifications using generalizability and classical test theory

Date

1985

Journal Title

Journal ISSN

Volume Title

Publisher

Abstract

The Texas State Board of Education has given high priority to the improvement of student achievement in mathematics, reading and written composition. The Texas Assessment of Basic Skills (TABS) tests were rigorously developed from a table of specifications and administered in 1980 to third, fifth and ninth graders. A regional Educational Service Center (ESC) developed Pre-Assessment Tests (PAT) from the TABS table of spedfications. These are used by school districts to identify students who need further remediation in order to pass the TABS tests. The purpose of this study was to evaluate the psychometric properties of the ninth grade PAT mathematics and reading test which had been administered to a random sample of 313 ninth grade students. This was accomplished in two phases. The first phase used descriptive statistics and classical true score analyses. The KR-20 for both tests were in the .80's indicating good reliability. The PAT's parallelism to the TABS tests could not be evaluated directly since the item by subject matrix for the TABS tests were not available to this researcher. However, the TABS item statistics were available identified only by objective and could be compared by objective to the PAT's. The following statistical procedures were performed on the respective TABS and PAT objectives: (1) Pearson product moment correlations, (2) Crosstabulation of the pass-fail groups by objective and total test scores and 3) percent of mastery by objective and total test. The California Achievement Test (CAT) and Test of Cognitive Skills (TCS) were also included in regression and discriminant analyses. It was found that the three tests together did a slightly better job of predicting the pass-fail groups on TABS, but the PAT tests alone correctly identified over 90 percent. However in predicting the fail TABS group, which was particulary small in this sample, the PAT tests did not do as well (0 to 30 percent correct). The second phase of this study used generalizability theory to examine the PAT mathematics and reading objectives. Using SPSS to write computer programs, an ANOVA p X 1 design for each objective was used to calculate the variance components for persons, items and interaction plus error. The estimated variance-covariance matrices were also computed for the objectives. The variances were small for the objectives which is consistent with a criterion referenced test. The interaction plus error variance term for each of the objectives ranged from 59 to 93 percent of the total objective's variance. Most were usually over 70 percent of the total variance. The measurement errors for each objective were large indicating that more than four items per objective were needed to give precise measurement of that content domain. [...]

Description

Keywords

Educational tests and measurements--Texas

Citation