Tailored Adaptive Personality Assessment System (TAPAS)

Tailored AdaptivePersonality AssessmentSystem (TAPAS)Fritz DrasgowUniversity of Illinois at UrbanaChampaignIPAC July 22, 2013

Thanks to my Colleagues:Sasha Chernyshenko Steve Stark Chris Nye Len White and Tonia Heffner, ARI Chris Kubisiak and Kristen Horgen,PDRI Deidre Knapp and our friends atHumRRO

TAPAS Vision We wanted to build a fully customizable assessment ofpersonality to fit an array of users’ needsUsers should be able to select: any dimension from a comprehensive superset of 22facets of the Big Five;a scale length to suit their needsa fake resistant response format (if faking is a problem)adaptive or staticResulting scores can be used to predict multiple criteriaor as source of feedback

Tailored Adaptive PersonalityAssessment System (TAPAS) To this end, TAPAS incorporates recentadvancements in: Item response theory (IRT);Models of personality; andComputerized adaptive testing (CAT)and a fake resistant format to provide a means foroperational use of personality assessment for preemployment testing

Today, I’ll talk about the 15 year journey thathas led to today’s TAPAS

The Beginning, Sasha Chernyshenko and Steve Stark weredoctoral students interested in fitting itemresponse theory models to personality dataThey fit the two- and three-parameter logisticmodels to 16 Personality Factor (16PF) dataThe fit was not good, which was surprisingbecause Steve Reise had already publishedpapers about fitting IRT models to personalitydata

The 2PL and 3PL are Dominance ModelsProb of Positive ResponseA person endorses an item if his/herstanding on the latent trait, theta, is moreextreme than that of the -2-10Theta123

Examples of Dominance ModelsFactor analysis Structural equations models Item response theory Classical test theory

An alternative Conceptualization:Thurstone ScalingThurstone assumed people endorseitems reflecting attitudes close to theirown feelings Coombs (1964) called this an idealpoint process Sometimes called an unfolding model

Example of an Ideal Point Process Person endorses item if his/her standing on thelatent trait is near that of the item. “I enjoy chatting quietly with a friend at a cafe.” Disagree either because:Too introverted (uncomfortable in public places)Too extraverted (chatting over coffee is boring)TooIntrovertedItemTooExtraverted

GGUM IRFs for twoPersonality Statements"I enjoy chatting quietly with a friend at a café."(Sociability)"I am about as organized as most .03.0

Important Point:The item-total correlation ofintermediate ideal point items will beclose to zero! This led Likert (1932) to assert suchitems were double-barreled andshould be avoided

Which Process is Appropriatefor Temperament Assessment? In a series of studies, we’veExamined the appropriateness of dominanceprocess by fitting models of increasing complexityto data from several personality inventories Compared the fits of dominance and ideal pointmodels of similar complexity to several existingmeasures of personality Compared the fits of dominance and ideal pointmodels to sets of items not preselected to fitdominance models

Key Findings: Dominance models only fit personality data ifthe items are carefully pre-selected to screenout those assessing intermediate trait valuesIdeal point models fit items assessing low,intermediate, and high trait valuesFor CAT to work well, we need to use a modelthat fits the data well and assesses trait valuesthroughout the trait continuum Ideal pointIRT

The Generalized GradedUnfolding Model (GGUM)Roberts, Donoghue, Laughlin (2000) Implemented in the GGUM2004computer program For dichotomously scored items, exp i j i i1 exp i 2 j i i1 P[U i 1 j ] 1 exp i 3 j i exp i j i i1 exp i 2 j i i1 i

TAPAS Model of Personality Based on factor analysis of each of the BigFive dimensions E.g., Roberts, B., Chernyshenko, O.S., Stark, S., & Goldberg, L.(2005). The structure of conscientiousness. Personnel PsychologyCurrently 22 facetsResulted from analyses of Lewis Goldberg’sdata set – 7 major personality inventoriesadministered to a sample of over 700

Goldberg Data SetA sample of 737 respondents, rangingin age from 22 to 90, all levels ofeducation, average of 2 years of postsecondary schooling Over a period of 5 years, participantscompleted 7 personality measures

Goldberg Data Set Included the following scales: The revised NEO Personality Inventory(NEO-PI-R), 240 items, 30 facetsCalifornia Psychological Inventory (CPI), 462true-false items, 20 facetsHogan Personality Inventory (HPI), 206items, 41 “homogeneous item composites”(HICs)Jackson Personality Inventory-Revised (JPIR), 300 items, 15 scales

Goldberg Data Set Multidimensional Personality Questionnaire(MPQ), 272 items, 11 primary scalesAbridged Big 5 Circumplex scales from theInternational Personality Item Pool (AB5CIPIP), over 400 items, 45 facetsSixteen Personality Factor Questionnaire(16PF), 185 items, 16 primary scales

So, What Is a Comprehensive Set ofFacets Underlying the Big 5? E.g., for Conscientiousness, Roberts et al.(2005) identified all of the facets, HICs, primaryscales, etc. of the seven instruments that wererelated to conscientiousness, ran factoranalysisThis is the method of “Standing on theshoulders of giants” i.e., “extending scienceby understanding and using the research andworks of great thinkers of the past”

Example of TAPAS FacetsConscientiousness Six facet hierarchical structure: Industriousness: task- and goal-directed Order: planful and organized Self-control: delays gratification Traditionalism: follows norms and rules Social Responsibility: dependable andreliable Virtue: ethical, honest, and moral

Conscientiousness(1/1).98.63Proactive Aspects ofConscientiousness(2/1)Inhibitive Aspects /3)Self Control(4/3).91From Roberts et al. 2005.99Self rtue(5/3).99Responsibility(6/3).92.96Self /6)

TAPAS Facets Conscientiousness: 6Emotional Stability: Adjustment, EvenTempered, Well BeingAgreeableness: Warmth, Selflessness,CooperationExtraversion: Dominance, Sociability,Excitement Seeking, EnergyOpenness: Intellectual Efficiency,Curiosity, Ingenuity, Aesthetic,Tolerance, Depth

Computerized AdaptiveTesting (CAT) Has been used by DoD for ASVAB preenlistment testing for 20 yearsBy selecting the next item based, in part, on thetest taker’s previous responses, we can adaptthe difficulty level to the ability of a test takerWe can use the same logic for personalityassessment: adapt the extremity of the itemsadministered to the trait level of the respondent

Average Correlations of True vs. EstimatedTrait Values for Static vs. CAT SimulatedPersonality Assessments For 7 facets:70 item static: .84 35 item CAT: .85 For 10 facets:100 item static: .84 50 item CAT: .84 Notes: items administered in MDPP formatfrom Stark et al., 2012, ORM

Overcoming theFaking Problem

Examples of “Traditional” Itemsthat Appear to Be Easily FakedWhat is the the positively keyed response to theseitems? Do you “Agree” or “Disagree”?I get along well with others. (A ) I try to be the best at everything I do. (C ) I insult people. (A-) My peers call me “absent minded.” (C-) Because these items consist of individual statements, theyare commonly referred to as “single stimulus” items.

Forced Choice Formats There has been a long interest inmultidimensional forced choice formats: Edwards (1954) Personal PreferenceScheduleWhite & Young’s Assessment of IndividualMotivation (AIM)Christiansen et al. (1998)Jackson et al. (2000)SHL’s OPQ

Why Forced-Choice Items? Correctingor detecting faking doesn’tseem to work well: Validity doesn’t increase after corrections (Schmitt &Oswald, 2006) Scales to detect faking are nowhere close to 100% effectiveand it is not clear what to do with “disqualified” applicants Warnings may not be very effective in settings with coachedapplicants Solution– Discourage faking through the useof forced-choice response formats

Jackson et al. (2000) Compared traditional “single stimulus” personalityitems to “quartets” formed by: First placing pairs of statements from differentdimensions into dyads statements in dyads hadsimilar endorsement rates (as single stimulusitems) and social desirability ratings Then combining high-desirability dyads with lowdesirability dyads to form a quartet Respondents chose the statement “Mostcharacteristic of me” and “Least characteristic ofme” from each quartet

Jackson et al. (2000) Respondents were given the quartets undertwo conditions: Answer honestlyImagine you’re a job applicant who reallywants to get hiredMean scores were higher in the jobapplicant condition for the quartet format by.30 SD but were .95 SD higher in theapplicant condition for the single stimulusitems Fake Resistant (but not fake-proof)

Heggestad et al. (2005) Also examined the multidimensional forcedchoice (MFC) format as a way to combatfakingCompared an MFC format to two Likert-typemeasures (NEO, IPIP) under Honest andFake Good conditionsAlso used “Most like me” and “Least likeme” ratingsCreated quartets by matching on statementextremity on the dimension it assesses, butnot social desirability

Heggestad et al. (2005)Effect sizes for Fake Good vs. Honestconditions were generally larger forthe single stimulus format But, for Conscientiousness, the effectsize was 1.23 for the single stimulusformat vs. 1.20 for the MFC format Not too much Fake Resistance

TAPAS Multidimensional PairwisePreference (MDPP) Format Create items by pairing stimuli that are similarin social desirability, but represent differentdimensions“Which is more like you?” I get along well with others. (A ) I always get my work done on time. (C )

Our Experience with Faking First study at recruit training centers: Matched statements on social desirability Found score inflation for 2AFC just as largeas single statementsSecond study: Matched statements on social desirabilityand their IRT extremity parameters Found greatly improved resistance to fakingfor 2AFC

Example MDPP Items “For each of the following pairs, select the statement that ismore like you.” 1a) People come to me when they want fresh ideas. ( Ingenuity)1b) Most people would say that I’m a “good listener”. ( Warmth) 2a) I almost always complete assignments on time. ( Industrious)2b) I generally perform well under pressure. ( Adjustment) 3a) I set high goals and work to meet them. ( Industrious)3b) I get along well with other people. ( Cooperation)36

IRT Model for Scoring MDPP Tests(Stark, 2002; Stark, Chernyshenko, & Drasgow, 2005)s 1st statementt 2nd statement1 Agree0 DisagreePst {1,0}Ps {1}Pt {0}P( s t ) i ( d s , d t ) Pst {1,0} Pst {0,1} Ps {1}Pt {0} Ps {0}Pt {1}Respondent evaluates each statement in pair separately andmakes independent decisions about endorsement. Statement endorsement probabilities P{0} and P{1} computedusing the GGUM model Traitscores are obtained via Bayes modal estimationinvolving k-dimensional minimization37

So, Does it Work?

TAPAS Research US Army and Air Force began implementationof TAPAS for enlistment screening at sixMilitary Enlistment Processing Stations (MEPS)on June 8, 2009 and at all MEPS in September200915 facets, 120 items, median response time ofabout 20 minutesArmy applicants were told that their scoresmight affect their enlistment eligibilityAir Force given “for research only” instructionsWill TAPAS predict attrition and “will-do”behaviors?

Is there score inflation for Armyapplicants?

Descriptive Statistics for TAPAS CATScores in Regular Army and AirForce SamplesTAPAS FacetAchievementAdjustmentCooperationDominanceEven TemperedAttention quencyOrderPhysicalConditioningSelf ControlSociabilityToleranceOptimismArmy - 50.19-0.19-0.22Air 5-0.04Note. Sample Sizes: Regular Army 86,962; AirForce 30,6580.070.08-0.070.13-0.08-0.010.02

Is there adverse impact?