Jeffrey M. Cucina, Ph.D. Henry Busciglio, Ph.D. Kathlea .
Category Ratings and Assessments:Impact on validity, utility, and managerial choiceJeffrey M. Cucina, Ph.D.U.S. Customs and Border ProtectionHenry Busciglio, Ph.D.U.S. Customs and Border ProtectionKathlea Vaughn, M.A.U.S. Customs and Border ProtectionPresented at the Annual Conference of the International Personnel Assessment CouncilTuesday, July 19, 2011, Washington, DC.Opinions expressed are those of the authors anddo not represent the position of U.S. Customs andBorder ProtectionToday’s Presentation Description of top-down selection and categoryratings Description of data analyses and methodology– Used real and simulated data Presentation of five research questions withresults Conclusions, recommendations, and topics forpractitioners to considerCategory Ratings – IPAC 20112
Federal Government Selection Applicants compete for positions based on theirknowledge, skills, and abilities Traditionally, applicants are rank-ordered usingassessment scores (from 70-100) and hiring istop-down Recent Presidential Memorandum (November2010) included switch to category ratings– Can loosely be described as a form of bandingCategory Ratings – IPAC 20113Purpose of Study Category ratings have become a hot topicamong HR professionals, hiring managers,and media outlets covering Federal issues We could find no past published/presentedresearch addressing category ratings Testing professionals in the FederalGovernment need to convert raw testscores into category ratingsCategory Ratings – IPAC 20114
Focus of Study Large-scale mission critical occupations.– Hundreds of openings, thousands ofincumbents, tens of thousands of applicants– Often use a professionally developed andvalidated test battery– Federal agencies that hire assessmentprofessionals usually have them to focus onthese large occupations– Focus of this study Small occupations not examined in our study– One opening, 5-10 applicantsCategory Ratings – IPAC 20115Top-Down 70-100 Explained Raw test scores are “transmuted” to70-100 scale– Linear transformation– 70 is required passing/cutoff score. Failingapplicants do not receive transmuted score.– Veterans can receive 5 or 10 bonus points– Hiring is top-downCategory Ratings – IPAC 20116
Rule of Three Explained Hiring manager choose among top 3– Hiring manager can make an offer to any ofthe top 3 applicants (based on 70-100 scores)– If multiple offers are made, then new groupsof 3 are created Occurs if an applicant declines an offer or 1position to be filled– If an applicant is passed over three times (i.e.,appears in the top 3 but never made an offer)he or she is automatically eliminated Unless he or she has veterans’ preferenceCategory Ratings – IPAC 20117Category Ratings Explained Raw test scores are placed into categories– Three categories are the most common Highly-Qualified (top) Well-Qualified (middle) Qualified (bottom)– Hiring manager can choose any applicant within acategory (ignoring veterans’ preference)– Proposed as an alternative to the rule of three as partof Federal hiring reform– Test scores used to place applicants into categories– Can merge categories when 2 applicants in onecategoryCategory Ratings – IPAC 20118
Method: Predictor Tests Used Composite Predictor (validity of .43)– Archival applicant data was used– For purposes of this study, we created a composite variable of acognitive measure and non-cognitive measureRaw Test Scoren 59,000Transmuted 70-100 Scoren 850 Composite Criterion– Training academy scores (also used as separate criterion)– Task-Based Job Simulation Scores– Supervisory RatingsCategory Ratings – IPAC 20119Method: Creating Category Ratings Officially, must use a job analysis (more on this later) We used six different approaches Best Case Scenario Categories– Used an empirical method, which maximizes criterion-relatedvalidity– Two cut scores used were those with the highest rpbi with jobperformance– These cut scores resulted in the following predictor score rangesfor each Category RatingRating Score Range391 - 100284 - 90170 - 83Category Ratings – IPAC 201110
Method: Creating Category Ratings Decades Categories – Based on transmutedscores– Category 1 70s– Category 2 80s– Category 3 90s-100 Tertiles – Top, Middle, and Bottom Thirds– Similar to quartiles or quintiles, but with three groupsCategory Ratings – IPAC 201111Method: Creating Category Ratings Worst Case – Negative Skew––––Based on transmuted scoresCategory 1 70Category 2 71Category 3 72-100 Worst Case – Middle––––Based on transmuted scoresCategory 1 70Category 2 71 through 99Category 3 100 Worst Case – Positive Skew––––Based on transmuted scoresCategory 1 70-98Category 2 99Category 3 100Category Ratings – IPAC 201112
Method: DatasetsLarge Applicant Dataset– n 59,000– Represented all applicants taking oneparticular form/series– Raw test scores normally distributedCategory Ratings – IPAC 201113Method: DatasetsLarge Applicant DatasetTraining Validity Dataset– Subset of Large Applicant Dataset– n 6,000– Applicants who were hired andwent to training academy– Training performance criterionCategory Ratings – IPAC 201114
Method: DatasetsLarge Applicant DatasetTraining Validity DatasetComplete Validity Dataset– n 850 incumbents– Criterion-related validitystudyCategory Ratings – IPAC 2011151. What is the impact of category ratings (vs. topdown 70-100 rankings) on criterion-relatedvalidity?– MacLane (2010) hypothesized decrease in validity– We concur and hypothesize that validity will decrease– Used complete validity dataset– Correlated composite criterion with transmuted 70100 scores, and category ratingsCategory Ratings – IPAC 201116
1. Category Ratings Æ Lower ValidityrUncorrectedpRaw Test score.430 .001Transmuted 70-100.429 .001- Best Case.414 .001- Decades.374 .001- Tertiles.335 .001- Worst Case Positive Skew.164 .001- Worst Case Middle.158 .001- Worst Case Negative Skew.053.125Predictor/MethodCategoriesCategory Ratings – IPAC 2011171. Category Ratings ignore valid information Within each category, the transmuted score was statisticallysignificant. (note: †: p .056)Predictor/MethodValidity of Transmuted Score within:Category 1(Bottom)Category 2(Middle)Category 3(Top)- Best Case.162**.117*.166*- Decades.185†.230**.232**- Tertiles.081.089.332**.407**(constant)(constant)- Worst Case Middle(constant).409**(constant)- Worst Case Negative Skew(constant)(constant).427**Categories- Worst Case Positive SkewNote: †: p .056.Category Ratings – IPAC 201118
1. Category Ratings Æ Decremental Validity Conducted hierarchical linear regression; Step 1: Category Rating Score;Step 2: Transmuted ScoreTransmuted score always added incremental validityUsing category ratings instead of transmuted has decremental validity R2 vs. Transmutedp- Best Case- .018 .001- Decades- .045 .001- Tertiles- .072 .001- Worst Case Positive Skew- .157 .001- Worst Case Middle- .159 .001- Worst Case Negative Skew- .181 .001Predictor/MethodCategoriesCategory Ratings – IPAC 2011191. Conclusion– Category ratings do decrease validity– Amount of decrease in validity dependson how categories are formed– Consistent with MacLane’s (2010)hypothesisCategory Ratings – IPAC 201120
2. What is the impact of category ratings on merit;in other words, are the top applicants (in terms ofcriterion scores) always selected?– Two hypotheses (drawn from banding literature) Pro-Banding Hypothesis – Differences in transmuted scoreswithin a category are largely due to chance and notmeaningful Anti-Banding Hypothesis – Differences in transmuted scoresare meaningful, especially with large pools of applicants– See OPM white paper by Frank Schmidt (no date)– Used training validity datasetCategory Ratings – IPAC 2011212. Average criterion score for applicants at eachtransmuted scorer .438, p .001Category Ratings – IPAC 201122
2. Average criterion score for applicants at eachcategory: Best Case Categoriesr .401, p .001Category Ratings – IPAC 2011232. Average criterion score for applicants at eachcategory: Decades Categoriesr .400, p .001Category Ratings – IPAC 201124
2. Average criterion score for applicants at eachcategory: Tertiles Categoriesr .345, p .001Category Ratings – IPAC 2011252. Average criterion score for applicants at eachcategory: Worst Case Positive Skew Categoriesr .183, p .001(Transmuted 99)(Transmuted 100)Category Ratings – IPAC 201126
2. Average criterion score for applicants at eachcategory: Worst Case Middle Categoriesr .156, p .001(Transmuted 100)(Transmuted 70)Category Ratings – IPAC 2011272. Average criterion score for applicants at eachcategory: Worst Case Negative Skew Categoriesr .065, p .001(Transmuted 70)(Transmuted 71)Category Ratings – IPAC 201128
2. Conclusion– Using transmuted score allows for finerdistinctions among applicants on the criterion– Using category ratings erases the finerdistinctions Æ Applicants with (slightly) lowercriterion scores may be selected ahead ofthose with (slightly) higher criterion scoresCategory Ratings – IPAC 2011293. What is the impact of category ratings on utility,compared to using transmuted scores? Key benefit of testing is return on investment via better quality hiresCompared change in utility when moving from transmuted scores tocategory ratingsUsed below utility formula and assumptionsΔU T Ns (r1 - r2) SDY z -Ns (C1 - C2)pT Tenure in years of average selectee 20 years (agent hired by age 37 retires at age 57 20 years)Ns Number selected per year 1,000 (same as congressionally mandated FY11 hiring goal for Border Patrol)r1 Validity of new selection system (e.g., category ratings)r2 Validity of old selection system (e.g., transmuted)SDY z - Dollar value of performance .32 (medium complexity job) 60,274 (GS-12-Step-1)z mean score of those who were selected 0.78 (used for both transmuted and category ratings)C1 Cost of old selection system C2 cost of new selection system N/A (cancels out)p selection ratio N/A (cancels out)Category Ratings – IPAC 201130
3. Category Ratings Æ Lower UtilityPredictor/MethodTransmuted (vs. raw score)Change in Dollars- 301,034Categories (vs. transmuted) - Best Case- 4,515,522- Decades- 16,556,915- Tertiles- 28,297,273- Worst Case Positive Skew- 79,774,228- Worst Case Middle- 81,580,437- Worst Case Negative Skew- 113,189,094Conclusion: Category ratings reduces return on investmentCategory Ratings – IPAC 2011314. What is the impact of category ratings onveterans’ preference?– Refresher on veterans’ preferenceTP Veterans - Preference eligibles with no disability rating- Receive 5 points under rule of threeXP Veterans - Disability rating less than 10%- Receive 10 points under rule of threeCP Veterans - Disability rating of at least 10% but less than 30%- Receive 10 points and move to very top of listCPS Veterans - Disability rating of 30% or more- Receive 10 points and move to very top of listCategory Ratings – IPAC 201132
4. What is the impact of category ratings onveterans’ preference?– Rule of Three Veterans receive an extra 5 (TP) or 10 (XP) pointsthat is added to their raw 70-100 transmuted score– Yields scores ranging from 70 to 110 (for all applicants)– If there are ties, then veterans listed first– Category Ratings Within a category, TP (5-point) and XP (10-point)veterans now move to the top of their originalcategory and must be hired first (if hiring madefrom that category) CP and CPS move out of their category (ifnecessary) to the top of the top categoryCategory Ratings – IPAC 2011334. Two TP (5-point) veterans under Decades model– Rule of ThreeVeteran A: 90Add Vets.Pref.Veteran A: 95Moves ahead of non-veterans with scores of 90-95Veteran B: 89Add Vets.Pref.Veteran B: 94Moves ahead of non-veterans with scores of 89-94Category Ratings – IPAC 201134
4. Two TP (5-point) veterans under Decades model– Category RatingsHighest Add Vets.Assign toVet. A: 90 CategoryPref.CategoryTop ofHighestCategoryMoves ahead of non-veterans with scores of 90-100Vet. B: 89Assign toCategoryMiddleAdd Vets.Pref.CategoryTop ofMiddleCategoryNow only moves ahead of non-veterans with scoresof 89. Unlike rule of three, now behind 90-94.Category Ratings – IPAC 2011354. Practical Significance: How many applicantswould really be impacted by this?– Used large applicant dataset– Rank-ordered applicants under decadescategory ratings model vs. 70-100– Added veterans’ preference points and movedfloaters to top– Used a random number to rank-orderapplicants with ties (same random numberused for both scenarios)Category Ratings – IPAC 201136
Change In Rank4. Average change in rank (category ratings vs. rule ofthree) for TP (5-point) veterans by transmuted scoreTransmuted Score(Without Veterans’ Preference) Applicants above red linewere ranked higher undercategory ratings Applicants below red linewere ranked lower undercategory ratingsNote: To create this chart, we split thedatafile by transmuted score andcomputed the average change inranking (i.e., rule of three rank –category ratings rank) for veterans witheach raw score under the decadesmodel.Category Ratings – IPAC 201137Change In Rank4. Average change in rank (category ratings vs. rule ofthree) for XP (10-point) veterans by transmuted scoreTransmuted Score(Without Veterans’ Preference) Applicants above red linewere ranked higher undercategory ratings Applicants below red linewere ranked lower undercategory ratingsCategory Ratings – IPAC 201138
4. Results: TP (5-point) veteransUnder category ratings (vs. rule of three):# Veterans ranked higher: 3,483 (48%)# Veterans ranked lower: 3,756 (52%)# Veterans ranked same: 0 (0%)Average change in rank: -637 placesRange of change in rankLargest drop: -13,011 placesLargest gain: 11,589 placesWilcoxon signed-rank test: Z -7.706; p .001Category Ratings – IPAC 2011394. Results: XP (10-point) veteransUnder category ratings (vs. rule of three):# Veterans ranked higher: 42 (16%)# Veterans ranked lower: 216 (84%)# Veterans ranked same: 0 (0%)Average change in rank: -6,499 placesRange of change in rankLargest drop: -19,135 placesLargest gain: 250 placesWilcoxon signed-rank test: Z -12.642; p .001Category Ratings – IPAC 201140
4. Number of TP (5-point) and XP (10-point) veterans bytransmuted score500TP (5‐point)450Number of Applicants400XP (10‐point)350N 59,000nVets. 848586878889909192939495969798991000Transmuted Score(Without Veterans’ Preference)Category Ratings – IPAC 2011414. Why this could matter.(Veterans’ preference is popular topic in the courts)– Consider recent court cases over veterans’ preference The Federal Career Intern Program (FCIP) was recentlystruck-down as written by an Administrative Law Judge (ALJ)at the Merit Systems Protection Board (MSPB) (Dean v. OPMand Evans v. Department of Veterans Affairs, 2010, MSPB213)– FCIP didn’t require a public job posting– An agency used FCIP to circumvent hiring a veteran– ALJ ruled that this could prevent veterans from being hiredand was not legal OPM’s ALJ exam had a legal challenge involving scorecompression (0-100 vs. 70-100) and 5 vs. 10-point preference.(Azdell and Fishman v. OPM, 2003, SCOTUS 03-624)Category Ratings – IPAC 201142
4. Conclusion– Category ratings changes the nature ofveterans’ preference Some veterans do better, but others do worse– Some veterans who would be hired under ruleof three but not under category ratings– Which veterans get ranked higher and whichdo not is somewhat arbitrary Is this in the spirit of the law?Is this fair?(These are points to ponder)(Note, none of us have a J.D.)Category Ratings – IPAC 2011435. What is the impact of category ratings onmanagerial choice?(Moving away from veterans and validity to new a topic.)–Often cited benefit of category ratings is that hiringmanager can choose anyone within a category (ignoringveterans’ preference)–Categories can be combined when 2 or fewer applicantsremain in the higher category If higher category did not have 2 applicants at first, then all but2 must have been offered a position. Applicants that hiring manager didn’t choose are still counted–With large occupations, will need to fill more positionsthan candidates in highest category We propose that the rule of three may lead to bettermanagerial choice in these situationsCategory Ratings – IPAC 201144
5. Scenario to consider– Imagine applicants assessed using a measure thateither has lower validity or misses importantcompetency for the job– There are 300 applicants, in three categories of 100applicants each– Hiring Manager does not want to hire 30% of theapplicants (for whatever valid or invalid reason) (In each category, 30 of the 100 applicants are unchosen byhiring manager)– Hiring goal is to hire 150 applicantsCategory Ratings – IPAC 201145Category Ratings – IPAC 2011465. A graphical depictionTop 10 ChosenApplicantsMiddle 10 UnchosenApplicantsBottom
5. Make all top category job offers.HiredTopn 30Middlen 70BottomCategory Ratings – IPAC 2011475. We still need to hire 80 more applicants.Must hire 28 of the 30unchosens from top,before hiring frommiddleTopn 30MiddlePlan B: Start fromscratch with a newannouncement. (Hopethat 30 top unchosensdon’t reapply.)BottomCategory Ratings – IPAC 201148
Method: DatasetsFictitious datasets–––––Small scale Monte Carlo Simulationn 300Three categories, each with 100 applicantsVaried number of new hires neededVaried percent of applicants who werechosen or unchosen– No veteransCategory Ratings – IPAC 2011495. Category Ratings vs. Rule of Three: Percent ofUnchosen Applicants Discarded Similar situation with 3 categories of 100 applicants eachIn the table below we vary the percent of unchosen applicantsHiring Goal: Only from Top CategoryUnchosenCategoryRatingsRule OfThreeUnchosenCategoryRatingsRule umns 2 & 3 show percentage of unchosen applicants not selected(i.e., able to be passed over)Category Ratings – IPAC 201150
5. Category Ratings vs. Rule of Three: Percent ofUnchosen Applicants DiscardedHiring Goal: 150CategoryRatingsRule OfThree10%77%100%20%63%30%CategoryRatingsRule 4%90%52%11%50%64%56%UnchosenUnchosenCategory Ratings – IPAC 2011515. Category Ratings vs. Rule of Three: Percent ofUnchosen Applicants DiscardedHiring Goal: Everyone (but unchosens)CategoryRatingsRule OfThree10%63%100%20%68%30%CategoryRatingsRule 8%90%66%10%50%61%52%UnchosenUnchosenCategory Ratings – IPAC 201152
5. Conclusion– Category ratings approach maximizes managerialchoice when selections are limited to candidates inthe top category– Rule of three approach maximizes managerial choicewhen categories are collapsed Except when 50% or more of candidates are unchosen, thencategory ratings approach maximizes managerial choice– Rule of three approach may give more managerialchoice for large occupations with mass hiring Since categories must be collapsed to meet hiring goals– Category ratings could give more managerial choicefor small occupations with few hires Since hiring will take place only from top categoryCategory Ratings – IPAC 201153Things to Think About Cutoff scores for categories– Must be created before job is posted– Must be created using job analysis Per OPM regulations and Delegated Examining Operations Handbook– How to set legally defensible cutoff scores on anobjectively scored multiple-choice test? Traditionally created using criterion-related validation study,Angoff standard setting study, etc.– This is not a “job analysis” as described in the literature Some job analysis surveys include rating scales that parallelbenchmarks for competency-based rating scales used instructured interview, KS
Jeffrey M. Cucina, Ph.D. U.S. Customs and Border Protection Henry Busciglio, Ph.D. U.S. Customs and Border Protection Kathlea Vaughn, M.A. U.S. Customs and Border Protection Opinions expressed are those of the authors and do not represent the position of U.S. Customs and Border Protection
The Cucina Series oven stands can be finished in virtually any material, including stucco, stone or brick. The Cucina Series oven stand comes in seven sizes, fitting ovens up to a 44″ cooking surface (Casa2G 110). The size of the stand corresponds to the oven. Giardino 60 takes a Cucina 60, etc Casa 80 takes a Cucina 80 setc.
Monocomando cucina con canna snodata lilla con doccetta a 2 getti Single lever sink mixer Küchen-Einhebelmischbatterie AM719.5/11C AM719.5/11B AM719.5/11R AM719.5/11R CR AM719.5/11B CR AM719.5/11C CR ø 35 mm ø 35 mm ø 35 mm CUCINA SN 70 CAO CR Monocomando cucina canna alta One-hole sink mixer, high spout Küchen-Einhebelmischbatterie mit .
Today, Mia Cucina enjoys continued success with our service and product quality, earning widespread recognition within the premium kitchen market segment. Mia Cucina is a statement of personal style manifested through a magnificent display of shape and form. It represents a lifestyle of charm enjoyed through beautifully ergonomic design.
Ogni giorno, nelle case di tutto il mondo, i miscelatori da cucina lasciano scorrere l‘acqua necessaria per dissetarsi e cucinare. Basti pensare che i rubinetti si aprono circa 100 volte al giorno! Questi gesti sono diventati così naturali che ormai non associamo più l‘acqua alla parola „cucina“.
Place Market, Cucina Fresca Gourmet Foods today is a leading producer of natural prepared foods for retail and foodservice. Chef-owned and operated, Cucina Fresca creates “real food” from real food – fresh, premium, natural ingredients readily found in home kitchens. Many of the company’s staple ingredients
Cucina Dinner Menu KIGALI MARRIOTT HOTEL KN3 AVENUE, NYARUGENGE DISTRICT KIGALI - RWANDA TEL: 250 222 111 111, kigalimarriott.com. Specialita di Pasta Fresca Pappardelle fresche alla campidanese con ragu di salsiccia fresca, olive, zafferano e salvia (Homemade fresh pappardelle with Sardinian
talvolta inaspettati. Ho pensato ad alcune ricette che vi permettano di scoprire quanto è bello utilizzare WMF Perfect Pro in cucina ogni giorno, utilizzando al meglio le caratteristiche di WMF Perfect Pro, e i vantaggi che vi può offrire mentre preparate una cena per voi o per i vostri amici. Quindi buona cucina e buon divertimento!
2.15.20 Profit sharing transactions 28 2.15.21 Re-importation of goods after repair or processing abroad 29 2.15.22 Split shipments or split consignments 29 2.15.23 Sole distributors, concessionaires and agents 30 2.15.24 Tie-in sales 30 . Effective 24 January 2014 Valuation of Imports – External Directive SC-CR-A-03 Revision: 2 Page 3 of 52 2.15.25 Time element 30 2.15.26 Transfer pricing .
