Test Theory Then and Now

It was in the early 1950s that the founder of modern test theory, Fredrick Lord,1 laid out the basic principles of multivariate test theory that is the foundation of today’s practices. Although he recognized that the responses to multiple-choice items formed a multinomial distribution, (Page 9), he reduced this distribution to the more familiar binomial expression (page 10).

He did this for two reasons. First he assumed that the learning was a linear process. Second, he knew that the inclusion of more than one answer for each item created serious mathematical problems for analyzing these data with the then-available procedures (Page 4); hence the restrictions on the scoring methods. The Problem of even partial scaling of alternative answers was not solved until about a decade later by Georg Rasch.2

From here he recognized that using differing item parameters caused the shapes of the item response curves to change (Page 12). This means that the choice of items became critical in the establishment of the underlying assumption that learning was linear. In fact, other than expediency, there is no good reason to assume that all learning is linear. Piaget had already reported, from clinical observation (See: Flavell 3) that he had observed two types of learning, one linear (assimilation) and one non-linear (accommodation) or transformational.

Linear models do not capture the leap-like transformations that occur during non-linear learning. This short-coming of linear models arises because the thought processes that are characteristic of each stage are discretely different, not just in quantity but in kind.

Our present approach goes back to the realization that for multiple option tests, the distribution of each alternative is at a different level on the total-score scale, as is demonstrated using Rasch Scaling. What isn’t evident from the mathematics, but becomes clear using interviews, is that the answer selection reflects the thinking behind the answer s the answer choice (Powell; 1968, 4 1977 5) as would seem reasonable, were we to apply Heisenberg’s 6 Uncertainty Principle to question interpretation.

The evidence thus far suggests that the way a student interprets the question predetermines the answer selected. This relationship is reciprocal in that the answer selected discloses the reasoning behind it. The answer choice therefore becomes a probabilistic approximation of the student’s reasoning outcome meaning that more interpretation can lead to the selection of the same answer. The ways answers pair up across the entire test is more meaningful than particular answers.

Thus, our procedure returns to Lord’s observation that answering is multinomial and uses this mathematical relationship to bypass the linear dependency problem to extract the change dynamics among all answers across time (Powell and Shklov; 7 1992). Both the linear and non-linear components detected clinically by Piaget become evident using this procedure. The results we are achieving suggest that Lord’s simplification to the binomial was ill advised, though understandable in the state of technology at that time, and in error so far as a general theory of test-taking behavior was concerned.

1 Lord, Fredrick, (1952) A Theory of Test Scores: Psychometric Monographs * Number 7 The Psychometric Society, Chapel Hill, NC

2 Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. (Copenhagen, Danish Institute for Educational Research)

3 Flavell, John H. (1963) The Developmental Psychology of Jean Piaget Van Nostran, Ney York

4 Powell, J. C. (1968) The interpretation of wrong answers from a multiple-choice test Educational and Psychological Measurement, 28, 403-412

5 Powell, J. C. (1977) The developmental sequence of cognition as revealed by wrong answers Alberta Journal of Educational Research XXIII (1) 43 – 51.

6 Heisenberg, Werner (1927) The Uncertainty Principle.

7 Powell, J. C. and Shklov, N. (1992) Obtaining information about learners’ thinking strategies from wrong answers on multiple-choice tests. The Journal of Educational and Psychological Measurement, 52, 847-865.