Development of a Scientific Test: A Practical Guide

Michael C. Ramsay; Cecil R. Reynolds

INTRODUCTION

Most authors portray test construction as a matter of carefully composing groups of items, administering them to a representative sample of people, and analyzing the responses using established statistical techniques. Many writers (e.g., Allen & Yen, 1979; Anstey, 1966; Kline, 1986; Robertson, 1990) lay out steps for the prospective test developer to follow. They often look something like this (adapted from Allen & Yen, 1979):

1. Develop a plan to cover the desired content.

2. Design items that fit the plan.

3. Conduct a trial administration.

4. Analyze the results, and modify the test if needed.

5. Administer the test again.

6. Repeat as necessary, beginning with Step 2 or 4.

The ensuing pages will, to a large extent, cajole and bludgeon the reader to follow this same time-tested trail to reliability and validity. Recent trends in test development, however, suggest that constructing a test requires more than close attention to the test itself. As growing numbers of psychologists and psychometricians tiptoe across the lines that once divided them, test developers are increasingly turning to mainstream research to test, and even to provide a basis for, the measures they develop.

The viewpoint that a good test should have a basis in empirical research, not theory alone (e.g., Reynolds & Bigler, 1995a, 1995b; Reynolds & Kamphaus, 1992a, 1992b), has gained in popularity. At a minimum, most test constructors would agree that a well-designed experiment can help explain the characteristics measured by a test (Embretson, 1985). Inevitably, such constructs as aptitude and schizophrenia do not respond readily to laboratory controls. Furthermore, obvious ethical constraints prevent scientists from manipulating certain variables, such as suicidal tendencies, and from holding others constant, such as learning rate. These considerations should restrain the influence of experimentalism on testing. Still, the foundations of testing have subtly shifted, and to some degree, the content of this chapter reflects this shift.

Theory, too, has played an important role in psychological test development. Before devising the ground-breaking Metrical Scale of Intelligence with Theodore Simon, Alfred Binet spent many years building and refining a concept of intelligence (Anastasi, 1986; Binet & Simon, 1916/1980). Soon after Binet and Simon released their scale, Charles Spearman (1927; 1923/1973; Spearman & Jones, 1950) presented a model of intelligence derived from correlational studies. Spearman (1904) posited four kinds of intelligence: present efficiency, native capacity, common sense, and the impression made on other people. Modem psychologists might recast these constructs loosely as achievement, ability, practical intelligence, and social intelligence. Thus, Spearman’s model begins to resemble recent theories that include practical, tacit, and social intelligence (Gardner, 1983; Sternberg, 1985, 1990; Sternberg & Wagner, 1986). Notably, however, Spearman’s model appears to omit creativity. Other contributors to the theory and modeling of mental ability include Guilford (1967, 1977), Cattell (1971), Thurstone (1938; Thurstone & Thurstone, 1941), Luria (1962/1980, 1966, 1972), Das, Kirby, and Jarman, (1979), and Kaufman and Kaufman (1983a, 1983b, 1983c). Personality testing, too, has a diverse theoretical base. Personality theories linked with testing include the big five personality factors (Costa & McCrae, 1992a, 1992b; John, 1990; Norman, 1963), trait or disposition theory (Mischel, 1990; Zeidner, 1995), Guilford’s 14 personality dimensions (Guilford & Zimmerman, 1956) and Murray’s manifest need system (Anastasi, 1988; Murray, 1938). The role of theory in test development becomes important in construct definition, Step 2 below.

FIRST GLIMPSES: TERMS, DEFINITIONS, AND CONCEPTS

Reynolds (1986) defines measurement as a set of rules for assigning numbers to objects, events, or actions. Reynolds goes on to define a psychological test as an instrument for applying these rules to behavior, whether overt or covert. The rules of measurement are the standardized procedures by which a measurement is taken so that it is reproducible. To determine the length of a rod, by which land was once measured, the king of England decreed that in each village, 10 men emerging from church—some tall, some short, some portly, some lean—would be taken and stood side by side. The distance from the beginning to the end of the line of men was the measure of a rod. Using the rod, a villager could measure a tract of land repeatedly and obtain the same result. Hence, the measurement was reproducible.

Assigning numbers according to rules is a part of everyone’s life. We measure such varied dimensions as the height of a child, the force of an earthquake, and the earned-run average of a baseball player. Some characteristics we measure informally. For example, a shopper might measure a bagful of grapes by dropping it onto a grocery store scale. Other characteristics call for moderate formality, as when a nurse weighs a patient as part of an intake procedure. Finally, some measured properties demand elaborate techniques to ensure the best possible estimates. Psychological characteristics fall into this category. In every case, however, a person must follow certain rules to obtain a reproducible measurement.

In standardized testing, the manual provides the examiner with a set of rules for measuring performance: Begin with item 5 for children ages 6 through 9; Stop after 4 consecutive failures; If an examinee gives an incomplete response, say, “Tell me more about it;” Allow 20 seconds for each item; and so on. This reproducible set of procedures represents the rules of measurement that the examiner uses for a particular test.

Many instruments appear in popular publications accompanied by claims that readers can use them to assess their personality, their intelligence, their compatibility with their mates, and so forth. These informal inventories present qualitative descriptors such as, If you scored between 20 and 25, you are highly self-aware. These inventories offer no means, no standard scores, and no evidence of reliability, validity, or norms. (The ensuing pages explicate such concepts as reliability and validity). Accordingly, these scales do not constitute meaningful tests. Fortunately, such dime-store exercises are becoming relatively rare. However, Eyde and Primoff (1992) call attention to a new complication. Software billed as tests, but lacking scientific documentation, is becoming available to consumers.

Along different lines, courts sometimes treat so-called anatomically correct dolls as if they yielded authoritative results. Yet efforts to develop sound instruments based on these dolls are largely in the exploratory stages, and empirical support for the dolls remains equivocal, at best (Bauer, 1994; DeLoache, 1995; Skinner, Giles, & Berry, 1994). Additionally, anatomic dolls vary so widely in size, design, detail, proportion, and dress that if psychologists established the validity of one model, they would have little basis for applying the results to the others.

For the present, other introductory considerations merit attention. Most important, any test constructor should adhere to high ethical and technical standards. Rudner (1996) provides a brief outline for anyone in a position to evaluate a test. Additionally, educators and psychologists have worked together to produce Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1985). The Standards, under revision at the time of this writing, include a section on test construction and evaluation. Guidelines for Computer-Based Tests and Interpretations (American Psychological Association, 1986) have become available as well (See also Committee on Professional Standards & Committee on Psychological Tests and Assessment, 1986; Most & Zeidner, 1995). In a different vein, the steps involved in developing a test for commercial publication may also interest readers. Robertson (1990, 1992) describes this process.

STEP 1. REVIEWING THE LITERATURE

Every scientific study starts with an idea. So, too, does a scientific test. Many works on test construction create the impression that as soon as an idea springs to mind, the test developer should go to work busily charting the areas that the inchoate test should cover. Not at all! An aspiring test designer should turn first to the research literature to see how researchers are handling the construct in question, and how scientists or other professionals have measured this construct in the past. Both researchers and clinicians can guard against duplicating earlier efforts, and sidestep the rabbit trails...