20051228 028 - DTIC
AFRL-HE-WP-TP-2005-0028AIR FORCE RESEARCH LABORATORYTests of Cognitive AbilityMalcolm James ReeOur Lady of the Lake UniversitySan Antonio TXThomas R. CarrettaHuman Effectiveness DirectorateWarfighter Interface DivisionWright-Patterson AFB OH 45433-7022December 200520051228 028Approved for public release;Distribution is unlimited.Human Effectiveness DirectorateWarfighter Interface DivisionWright-Patterson AFB OH 45433
Form ApprovedREPORT DOCUMENTATION PAGEOMB No. 0704-0188Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining thedata needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducingthis burden to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 222024302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currentlyvalid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.3. DATES COVERED (From - To)2. REPORT TYPE1. REPORT DATE (DD-MM-YYYY)Technical PaperDec-20055a. CONTRACT NUMBER4. TITLE AND SUBTITLETests of Cognitive Ability5b. GRANT NUMBER5c. PROGRAM ELEMENT NUMBER6. AUTHOR(S)5d. PROJECT NUMBER**Thomas R. Carretta5e. TASK NUMBER*Malcolm James Ree7184095f. WORK UNIT NUMBER728. PERFORMING ORGANIZATION REPORTNUMBER7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)AND ADDRESS(ES)*Our Lady of the Lake UniversitySan Antonio TXAFRL-HE-WP-TP-2005-00289. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES)**Air Force Materiel10. SPONSOR/MONITOR'S ACRONYM(S)AFRL/HECVCommandAir Force Research LaboratoryHuman EffectivenessWarfighter11. SPONSOR/MONITOR'S REPORTDirectorateNUMBER(S)Interface DivisionWright-Patterson AFB OH 45433-702212. DISTRIBUTION / AVAILABILITY STATEMENTApproved for public release;distribution isunlimited.13. SUPPLEMENTARY NOTESThis isa book chapter.Clearance No.AFRL/WS-05-2719,1 Dec 0514. ABSTRACTThis chapter consists of six parts. Part one briefly reviews the historical foundation of the concept of cognitive abilityand early attempts to measure it. Part two reviews modern theories of the structure of cognitive ability and theemergence of the concept of general cognitive ability. Next, part three introduces the concepts of specific abilities,knowledge, and noncognitive traits. Part four discusses psychometric characteristics of tests including reliabilityand validity. Part five reviews the issues to be considered when deciding whether to choose from amongcommercially-available tests or develop a test. Example questions to help in test construction are provided. Thesixth and final part is a general summary.15. SUBJECT TERMSCognitive ability, knowledge16. SECURITY CLASSIFICATION OF:17. LIMITATION18. NUMBER19a. NAME OF RESPONSIBLE PERSONUnclassifiedOF ABSTRACTOF PAGESThomasa. REPORTUNCb. ABSTRACTUNCc. THIS PAGEUNCSAR63R.Carretts19b. TELEPHONE NUMBER (include areacode)(937)255-656-7014Standard Form 298 (Rev. 8-98)Prescribed by ANSI Std. 239.18
TESTS OF COGNITIVE ABILITYMalcolm James ReeOur Lady of the Lake University, San Antonio, TexasThomas R. Carretta1Air Force Research Laboratory, Wright-PattersonAir Force Base, OhioOVERVIEWThis chapter consists of six parts. Part one briefly reviews the historicalfoundation of the concept of cognitive ability and early attempts to measure it.Part two reviews modem theories of the structure of cognitive ability and theemergence of the concept of general cognitive ability. Next, part three introducesthe concepts of specific abilities, knowledge, and noncognitive traits. Part fourdiscusses psychometric characteristics of tests including reliability and validity.Part five reviews the issues to be considered when deciding whether to choosefrom among commercially-available tests or develop a test. Example questions tohelp in test construction are provided. The sixth and final part is a generalsummary.
HISTORICAL FOUNDATIONSThe concept of cognitive ability can be traced back over 2,500 years.Zhang (1988) reported that in the sixth century BC, the great Chinese philosopherConfucius, divided people into three groups based on intelligence: people of greatwisdom, people of average intelligence, and people of little intelligence. AnotherChinese philosopher, Mencius (fourth century BC), likened intellectualmeasurement to measurement of physical properties. Within a century, the Handynasty (202 BC -200 AD) had heeded Confucius and Mencius and implementeda system of civil service tests in China.In the fourth century BC, Aristotle made a distinction between ability(dianoia)and emotional and moral capacity (orexis). Zhang (1988) reported onthe custom of testing children at one year of age beginning in the sixth century ADin China, particularly in southern China. This was described in the writings ofYen (531-590 AD). Zhang (1988) also noted that the use of puzzles to testcognitive ability was popularized during the Song dynasty (960-1127 AD). Oneexample consisted of several geometric shapes that could be manipulated and fitinto a variety of designs. The test was designed to measure creativity, divergentthinking, and visual-spatial perception. Another popular Chinese puzzle testdesigned to measure reasoning ability consisted of interconnected copper rings
TESTS OF GENERAL COGNITIVE ABILITY3mounted on a bar with a rod running through their center. The goal of the test wasto remove the bar from the center of the rings.In the west, the examination of human cognitive abilities was taken up byreligious philosophers. In the16 thcentury AD, Descartes, the French secularphilosopher regarded ability as res cogitans, the thing that thinks.In 1575, Juan Huarte published in Spanish (Peir6 & Munduate, 1994) atreatise on work and human ability called Examen de Ingenios. It was laterpublished in English as The examination of men Is wits: Discovering the greatdifferences of wits among men and what sort of learning suits best with eachgenius.The modem scientific study of human cognitive abilities, however, is oftenattributed to Binet in France and to the World War I Army Alpha and Beta tests inAmerica.GENERAL COGNITIVE ABILITYThe English polymath, Sir Francis Galton (1869), invented the construct ofgeneral cognitive ability, calling it g as shorthand. Charles Spearman (1927, 1930)made the concept of g more accessible to psychology through his two-factortheory of human abilities which proposed that every measure of ability had twocomponents, a general component (g) and a specific component (s).
4While the general component was measured by every test, the specificcomponent was unique to each test. Though each test might have a differentspecific component, Spearman also observed that s could be found in commonacross a limited number of tests. Therefore, the two-factor theory allowed for aspatial factor or other factor that was distinct from g but could be found in severaltests. These factors shared by tests were called "group factors." Spearman (1927)identified several group factors and noted (Spearman, 1937) that group factorscould be either narrow or broad. He further observed that s could not be measuredwithout measuring g. As we have written elsewhere (Ree & Carretta, 1996, 1998),To be accurate, we should call mathematics not M butg Mwith g written large to indicate its contribution to the variance ofthe factor. (Ree & Carretta, 1996, p. 113).In fact, tests that do not even appear to measure g do so as illustrated byRabbitt, Banerji, and Szymanski (1989) who demonstrated a strong
TESTS OF GENERAL COGNITIVE ABILITY5correlation (.69) between "Space Fortress" a psychomotor task that lookslike a video game, and an IQ test.Controversy about g has not abated despite Spearman's early assertion(1930) that g was beyond dispute. In contrast to Spearman's model, Thurstone(1938) proposed a multiple ability theory. Thurstone allowed no general factor,only seven "unrelated abilities" that he called "primary." Spearman (1938)reanalyzed Thurstone's data noting that g had been submerged through rotation.He then demonstrated the existence of g in Thurstone's tests. This finding wasindependently confirmed by Holzinger and Harmon (1938) and finally byThurstone and Thurstone (1941). Despite empirical evidence, theories of multipleabilities held sway (Fleishman & Quaintance, 1984; Gardner 1983; Guilford,1956, 1959; Steinberg, 1985). This was particularly true in psychometrics, wherethese theories lead to the construction of numerous multiple ability tests such asthe Differential Aptitude Test, General Aptitude Test Battery, Armed ServicesVocational Aptitude Battery, Air Force Officer Qualifying Test, FlanaganAptitude Tests, Flanagan Industrial Tests, and others. Cleaving to the empiricaldata, other researchers continued to study g (Arvey, 1986; Gottfredson, 1986,1997; Gustafsson, 1980, 1984, 1988; Jensen, 1980, 1993, 1998; Schmidt &Hunter, 1998, 2004; Thomdike, 1986;.Vernon, 1950, 1969).
6Fairness and Similarity: Near Identity of Cognitive StructureThere are several issues that must be addressed when measuring ability insex and ethnic groups. One of these is that the same factors should be measuredfor all groups. McArdle (1996) among others has advocated that factorialinvariance (i.e., equality of factor loadings) should be demonstrated before othergroup comparisons (e.g., mean differences) are considered. McArdle stated that iffactorial invariance is not observed, the psychometric constructs being measuredmay be qualitatively different for the groups being compared, obscuring theinterpretation of other group comparisons.Several studies of cognitive factor similarity have been conducted.Comparing the factor structure of World War II U. S. Army pilot selection testsfor Blacks and Whites, Michael (1949) found virtually no differences. Humphreysand Taber (1973) also found no differences when they compared factor structuresfor high and low socio-economic status boys from Project Talent. Although theethnicity of the participants in Project Talent was not specifically identified, theyexpected that the ethnic composition of the two groups would differ significantly.Using 15 cognitive tests, DeFries, Vandenberg, McClearn, Kuse, Wilson,Ashton, and Johnson (1974) compared the structure of ability for Hawaiians ofeither European or Japanese ancestry. They found the same four factors and nearlyidentical factor loadings for the two groups.
TESTS OF GENERAL COGNITIVE ABILITY7These studies all examined common factors. Using a hierarchical model,Ree and Carretta (1995) examined the comparative structure of ability across sexand ethnic groups. They observed only small differences on the verbal/math andspeed factors. No significant differences were found for g on ability measures.Carretta and Ree (1995) made comparisons of aptitude factor structures inlarge samples of young Americans. The factor model was hierarchical including gand five lower-order factors representing verbal, math, spatial, aircrewknowledge, and perceptual speed. The model showed good fit and little differencefor both sexes and all five ethnic groups (White, Black, Hispanic, AsianAmerican, and Native-American). Correlations between factor loadings for thesex groups and for all pairs of ethnic groups were very high, approaching, r 1.0.Comparisons of regression equations between pairs of groups indicated that therewas no mean difference in loadings between males and females or among theethnic groups. These and previous findings present a consistent picture of nearidentity of cognitive structure for sex and ethnic groups.Predictive FairnessSeveral researchers have conducted studies of predictive fairness ofcognitive ability tests. Jensen (1980) noted that numerous large scale studiesprovided no evidence for predictive unfairness. He concluded that predictive bias
did not exist, although intercept differences could be observed and were likely dueto sampling error or differences in reliability for the two groups (p. 514).Putting a finer point on it, Carretta (1997) demonstrated that even whenintercept differences were observed in statistical tests of differences of regressionequations for two groups, the differences were due solely to differing reliabilityfound in the two groupsHunter and Schmidt (1979) investigated 39 studies of Black-White validityand found no evidence of differential prediction for the groups. Schmidt andHunter (1982) illuminated pitfalls in assessing the fairness of regressions usingtests of differences in regression (linear) models. In these two studies, Hunter andSchmidt concluded that artifacts accounted for the apparent differential predictionand that no predictive bias was present. Carretta (1997) and Jensen (1980)provide clear statistical explanations of the issues.In sum, no evidence exists that cognitive ability tests are unfair.SPECIFIC ABILITY, KNOWLEDGE, AND NONCOGNITIVE TRAITSThe measurement of specific abilities, knowledge, and noncognitive traitsoften has been proposed as crucial for understanding human characteristics andoccupational performance. Ree and Earles (1991) have demonstrated the lack ofpredictiveness for specific abilities while Ree and others (Olea & Ree, 1994; Ree,
TESTS OF GENERAL COGNITIVE ABILITY9Carretta, & Doub, 1998/1999; Ree, Carretta, & Teachout; 1995; Ree, Earles, &Teachout, 1994) demonstrated the predictiveness ofjob knowledge.McClelland (1993), for example, suggested that under some circumstancesnoncognitive traits such as motivation may be better predictors ofjob performancethan cognitive abilities. Sternberg and Wagner (1993) proposed the use ofmeasures of tacit knowledge and practical intelligence in lieu of measures of"academic intelligence." They define tacit knowledge as "the practical know howone needs for success on the job" (p. 2). Practical intelligence is defined as a moregeneral form of tacit knowledge. Schmidt and Hunter (1993), in a review ofSternberg and Wagner, note that their concepts of tacit knowledge and practicalintelligence are redundant with the well established construct ofjob knowledge.Additionally, Ree and Earles (1993) pointed out the lack of rigorous empiricalevidence to uphold the assertions of McClelland, Sternberg, and Wagner as wellas other critics.The construct of Emotional Intelligence (Goleman, 1995) has beenproposed as another facet that is more important than ordinary cognitive ability.Although its proponents (e.g., Mayer, Salovey, & Caruso, 2002) consider it to be adistinct construct, Schulte, Ree, and Carretta (2004) have demonstrated that it isnot much more than a combination of the existing constructs of cognitive abilityand personality.
10PSYCHOLOMETRIC CHARACTERISTICS OF MEASURES OFCOGNITIVE ABILITYCourses in statistics and research methods are common for humanresources/personnel specialists and there are established guidelines for conductingstudies of personnel measurement and selection (American PsychologicalAssociation, American Educational Research Association, & National Council onMeasurement in Education, 1999; Society for Industrial-OrganizationalPsychology, 2003). Reliability and validity are two core concepts that must beconsidered whether choosing a commercial test or developing a test.ReliabilityReliability is best defined as precision of measurement; that is how muchof the measurement is true and how much is error. In this statistical context"error" does not mean wrong, but random fluctuation. An error has not beencommitted, rather random fluctuation happens perforce and cannot be avoidedalthough it can be minimized. From this basic definition flow the other populardefinitions of reliability such as stability over time and consistency across testforms, as well as internal consistency. Stability over time typically is measured byretesting people after a period of time to ensure that their scores are consistent(i.e., test-retest reliability). Stability across test forms measuring the same
TESTS OF GENERAL COGNITIVE ABILITY11construct(s) is referred to as alternate form reliability. Internal consistency ismeasured by assessing the extent to which items are correlated with each other(e.g., correlating odd items with even items or split-half reliability or coefficientalpha). All three of these indices of reliability are typically measured usingcorrelations or approximations to correlations. Although correlations usuallyrange from 1.0 to -1.0, a reliability coefficient is a ratio of true variance to totalvariance.Two widely used cognitive ability tests are the Wonderlic Personnel Testand the Watson-Glaser Critical Thinking Appraisal. According to research cited inthe Wonderlic PersonnelTest & ScholasticLevel Exam User'sManual,the testretest reliability ranges from .82 to .94; alternate form reliability ranges from .73to .95; and split-half reliabilities range from .88 to .94. Similarly high levels ofreliability are noted in the Watson-Glaser CriticalThinking AppraisalManual(Form S). Test-retest reliability was .81 for a sample of 42 employees and internalconsistency reliabilities ranged from .66 to .87 in a wide variety ofjobs. The datafrom these two well-known and frequently used tests shows that cognitive abilityis a reliably measured construct.For a test to be reliable there must also be consistent administration,consistent collection of answers, and objective scoring. Test administrationprocedures must not vary from examinee to examinee and the data collection
12methods must be consistent. For example, Ree and Wegner (1990) showed thatapparently minor changes in machine-scored answer sheets could produce majorchanges in tests scores, particularly in speeded tests. This issue looms larger as weconsider placing our test for the selection of applicants on a computer where thepresentation could vary by screen size, contrast, and font type. Additionally, whendifferent administration modes or response collection are necessary, it is essentialto develop statistical corrections for the scores (Carretta & Ree, 1993). The use oftests of poor reliability to make decisions about excluding applicants, especiallyapplicants near the minimum cutting point from a training program, is badpractice and may lead to indefensible consequences in cour,t should a legalchallenge arise.Scoring must be objective. A correct answer must be counted correct by allscorers. To deviate from this will cause scores to vary by who did the scoring andwill reduce reliability of the test leading to reduced validity and possibly anindefensible position in court. This is less of a problem for a multiple choice testwhere the answer is presented and must be identified from among answerspresented. It is more of a problem for an essay type exam where the answer mustbe produced and evaluated.General cognitive ability can be reliably measured through severalmethods. Because it is the greatest source of variance in cognitive tests it is
TESTS OF GENERAL COGNITIVE ABILITY13relatively easy to get acceptable reliability by careful item construction and byadding items. However, as Thompson (2003) has pointed out, the reliability to beconsidered is the reliability in the sample currently being investigated, not thatfrom previous test administrations or the normative sample."It is important to evaluate score reliability in all (emphasis inoriginal) studies, because it is the reliability of the data in hand thatwill drive study results, and not the reliability of the scoresdescribed in the test manual." (Thompson, 2003, p. 5).ValidityThe important question
TESTS OF GENERAL COGNITIVE ABILITY 7 These studies all examined common factors. Using a hierarchical model, Ree and Carretta (1995) examined the comparative structure of ability across sex and ethnic groups. They observed only small differences on the verbal/math and speed factors. No significant differences were found for g on ability measures.
Título original: Ranger’s apprentice: the ruins of Gorlan 1. Literatura infanto-juvenil I. Título. 07-7502 CDD-028.5 Índices para catálogo sistemático: 1. Literatura infanto-juvenil 028.5 2. Literatura juvenil 028.5 . M orgarath, senhor das Montanhas da Chuva e da Noite, ex-barão de Gorlan no Reino de Araluen, observou, talvez .
canty, david j. 2000-064, 2001-038, 2002-028 capers, kaynerra 2016-111 capitol works communication 2008-079 cardwell, joanna 2013-123 care-tex, inc. 2010-028 carolina film alliance 2006-058 carolink 2008-044 carter, clinton m. 2014-028
wall tubing will help the ferrules to overcome minor surface defects that could contribute to gas leakage. HOKE recommends the following minimum wall thickness for tubing when system media contains gases. TUBE OD (inches) NOMINAL MINIMUM WALL THICKNESS (inches) 1/ 8 0.028 3/ 16 0.028 1/ 4 0.028 5/ 16 0.035 3/ 8 0.035 1/ 2 0.049 TUBE OD NOMINAL .
a. Teleworker Self-Assessment SPO-2008-028-002; b. Request for Teleworking Approval SPO-2008-028-003; and, c. Work Space Self-Certificate SPO-2008-028-004 to verify that an alternative work place is available, as well as safe and suitable for the purposes of the employee’s w
MAINTENANCE MANUAL CENTRAL OFFICE, TELEPHONE, AUTOMATIC AN/TTC-41 (V) 1 (NSN 5805-01-028-8393), AN/TTC-41(V)2 (NSN 5805-01-028-8394), AN/TTC-41 (V)3 (NSN 5805-01-028-8392), AND AN/TTC-41(V)4 (NSN 5805-01-044-8869). REPORTING OF ERRORS You can improve this manual b
Crowley and Debra Hawhee. 52527_CH01_p001-028 11/5/10 1:51 PM Page 1. 2 CHAPTER 1 Ancient Rhetorics: Their Differences and the Differences They Make Ancient teachers of rhetoric thought that disagreement among human beings
Dr Colm Owens Dr Andrew Sands Dr Mark Spence Dr Brian McCrossan Clinical Nurse Specialist Jan Gordon Tel: 028 9063 5211 / 07775 598 258 Email: jan.gordon@ belfasttrust.hscni.net Secretary to Dr Lockhart Louise Dragonetti Tel: 028 90 63 3173 Email: louise.dragonetti@belfasttrust.hscni.net Patient Council Clare Caulfield , Bravehearts NI
IEE Colloquia: Electromagnetic Compatibility for Automotive Electronics 28 September 1999 6 Conclusions This paper has briefly described the automotive EMC test methods normally used for electronic modules. It has also compared the immunity & emissions test methods and shown that the techniques used may give different results when testing an identical module. It should also be noted that all .
