New York State Testing Program 2015: English Language Arts .

New York State TestingProgram 2015:English Language Artsand MathematicsGrades 3–8\Technical ReportPearson2015

Developed and published under contract with the New York State Education Department byPearson, 2510 North Dodge Street, Iowa City, Iowa 52245. Copyright 2015 by the New YorkState Education Department.Permission is hereby granted for New York State school administrators and educators toreproduce these materials, located online at http://www.p12.nysed.gov/assessment/reports/, inthe quantities necessary for their school’s use, but not for sale, provided copyright notices areretained as they appear in these publications. This permission does not apply to distribution ofthese materials, electronically or by any other means, other than for school use.ii

Table of ContentsSection 1: Introduction and Overview . 1Introduction . 1Test Purpose. 1Target Population . 1Test Use and Decisions Based on Assessment . 1Scale Scores. 2Statewide Percentile Ranks . 2Proficiency Level Cut Scores and Classification . 2Subscores. 3Testing Accommodations . 4Test Transcriptions . 5Test Translations . 5Section 2: Test Design and Development . 6Test Descriptions. 6ELA Tests. 6Mathematics Tests . 7Test Configuration. 7Test Book Design and Testing Times . 7Embedded Field Test Items . 8New York State Educators’ Involvement in Test Development . 8Development and Review Process . 9Test Blueprints . 9Passage Selection and Item Criteria Documents. 9Passage Finding . 12Item Development . 12Educator Item Review . 13Field Testing . 14Rangefinding . 15Item Selection and Test Creation (Criteria and Process). 15Test Form Production and Reviews . 16Final Eyes Committee . 16Proficiency and Performance Standards . 16Section 3: Validity . 17Content Validity . 17Construct (Internal Structure) Validity . 18Internal Consistency . 18Unidimensionality . 18Detection of Bias . 22Section 4: Test Administration and Scoring . 24Test Administration . 24Scoring Procedures of Operational Tests . 24iii

Scoring Models . 25Scoring of Constructed-Response Items . 25Scorer Qualifications and Training . 26Quality Control Process. 26Section 5: Operational Test Data Collection and Classical Analysis . 27Data Collection . 27Data Processing . 27Sampling Down for Representativeness. 27Classical Analysis and Calibration Sample Characteristics . 33Classical Data Analysis . 44Item Difficulty and Point Biserial Correlation Coefficients. 45Speededness . 46Differential Item Functioning . 48Section 6: IRT Calibration and Equating . 51IRT Models and Rationale for Use . 51Calibration Sample . 52Calibration Process . 58Item-Model Fit . 60Local Independence . 61Equating and Scaling . 62Anchor Set Evaluation . 65Test Characteristic Curves . 67Scoring Procedure . 80Raw Score-to-Scale Score and SEM Conversion Tables . 81Section 7: Reliability and Standard Error of Measurement . 83Test Reliability . 83Test Statistics and Reliability for Total Test . 83Reliability of MC Items . 85Reliability of CR Items. 85Test Reliability for Reporting Categories. 86Standard Error of Measurement . 99Performance Level Classification Consistency and Accuracy . 99Consistency . 100Accuracy. 101Section 8: Summary of Operational Test Results . 103Scale Score Distribution Summary . 103ELA Scale Score and Subscore Distributions . 103ELA Grade 3 . 105ELA Grade 4 . 106ELA Grade 5 . 107ELA Grade 6 . 108ELA Grade 7 . 109ELA Grade 8 . 110iv

Mathematics Scale Score Distributions . 111Mathematics Grade 3. 112Mathematics Grade 4. 113Mathematics Grade 5. 115Mathematics Grade 6. 116Mathematics Grade 7. 117Mathematics Grade 8. 118Performance Level Distribution Summary . 120ELA Test Performance Level Distributions . 120ELA Grade 3 . 122ELA Grade 4 . 123ELA Grade 5 . 124ELA Grade 6 . 125ELA Grade 7 . 126ELA Grade 8 . 127Mathematics Test Performance Level Distributions . 128Mathematics Grade 3. 128Mathematics Grade 4. 129Mathematics Grade 5. 131Mathematics Grade 6. 132Mathematics Grade 7. 133Mathematics Grade 8. 134References . 136Appendix A: ELA and Mathematics Test Configuration and Testing Times . 139Appendix B: ELA and Mathematics Test Blueprints . 144Appendix C: Passage Selection Guidelines for Assessing ELA . 146Appendix D: Universal Design Item Checklist . 147Appendix E: Criteria for Item Acceptability . 150Appendix F: Psychometric Guidelines for Operational Item Selection . 152Appendix G: Operational Item Maps . 153Appendix H: ELA Short-Response Rubric . 170Appendix I: ELA Extended-Response Rubric . 171Appendix J: Mathematics Short-Response Rubric. 174Appendix K: Mathematics Extended-Response Rubric . 175Appendix L: Factor Analysis Results for Select Subgroups . 176Appendix M: Classical Test Theory Statistics . 193Appendix N: Items Flagged for DIF . 210Appendix O: Item Response Theory Statistics . 212v

Appendix P: Derivation and Estimation of Classification Consistency and Accuracy. 247Appendix Q: Raw Score-to-Scale Score and Scale Score Frequency Tables. 250vi

List of TablesTable 1. ELA Subscore Categories and Total Possible Score Points . 4Table 2. Mathematics Subscore Categories and Total Possible Score Points . 4Table 3. Summary of Unique 2014 Field Test Items . 14Table 4. ELA Tests Factor Analysis . 20Table 5. Mathematics Tests Factor Analysis . 21Table 6A. ELA Grade 3 Data Cleaning . 28Table 6B. ELA Grade 4 Data Cleaning . 29Table 6C. ELA Grade 5 Data Cleaning . 29Table 6D. ELA Grade 6 Data Cleaning . 29Table 6E. ELA Grade 7 Data Cleaning . 30Table 6F. ELA Grade 8 Data Cleaning . 30Table 7A. Mathematics Grade 3 Data Cleaning . 31Table 7B. Mathematics Grade 4 Data Cleaning . 31Table 7C. Mathematics Grade 5 Data Cleaning . 31Table 7D. Mathematics Grade 6 Data Cleaning . 32Table 7E. Mathematics Grade 7 Data Cleaning. 32Table 7F. Mathematics Grade 8 Data Cleaning . 32Table 8A. ELA Grade 3 Sample Characteristics . 33Table 8B. ELA Grade 4 Sample Characteristics . 34Table 8C. ELA Grade 5 Sample Characteristics . 35Table 8D. ELA Grade 6 Sample Characteristics . 36Table 8E. ELA Grade 7 Sample Characteristics . 37Table 8F. ELA Grade 8 Sample Characteristics . 38Table 9A. Mathematics Grade 3 Sample Characteristics . 39Table 9B. Mathematics Grade 4 Sample Characteristics. 40Table 9C. Mathematics Grade 5 Sample Characteristics. 41Table 9D. Mathematics Grade 6 Sample Characteristics . 42Table 9E. Mathematics Grade 7 Sample Characteristics . 43Table 9F. Mathematics Grade 8 Sample Characteristics . 44Table 10A. ELA Correlations (across Books) . 47Table 10B. Mathematics Correlations (across Books) . 47Table 11A. ELA Correlations (Book by Book) . 47Table 11B. Mathematics Correlations (Book by Book) . 48Table 12. ELA Classical DIF Sample N-Counts . 49Table 13. Mathematics Classical DIF Sample N-Counts . 49Table 14. ELA Items Flagged for DIF . 50Table 15. Mathematics Items Flagged for DIF . 50Table 16A. ELA Grades 3 and 4 Demographic Statistics . 53vii

Table 16B. ELA Grades 5 and 6 Demographic Statistics . 54Table 16C. ELA Grades 7 and 8 Demographic Statistics . 55Table 17A. Mathematics Grades 3 and 4 Demographic Statistics. 56Table 17B. Mathematics Grades 5 and 6 Demographic Statistics . 57Table 17C. Mathematics Grades 7 and 8 Demographic Statistics . 58Table 18. ELA Calibration Results . 59Table 19. Mathematics Calibration Results . 59Table 20. ELA Equating Coefficients . 62Table 21. Mathematics Equating Coefficients . 63Table 22. ELA Scaling Coefficients . 64Table 23. Mathematics Scaling Coefficients . 64Table 24. ELA Performance-Level Cut Scores . 81Table 25. Mathematics Performance-Level Cut Scores . 82Table 26. ELA Test Form Statistics. 84Table 27. ELA Test Reliability and Standard Error of Measurement . 84Table 28. Mathematics Test Form Statistics . 84Table 29. Mathematics Test Reliability and Standard Error of Measurement . 84Table 30. ELA MC Item Reliability and Standard Error of Measurement . 85Table 31. Mathematics MC Item Reliability and Standard Error of Measurement . 85Table 32. ELA CR Item Reliability and Standard Error of Measurement. 86Table 33. Mathematics CR Item Reliability and Standard Error of Measurement . 86Table 34A. ELA Grade 3 Test Reliability by Subgroup . 87Table 34B. ELA Grade 4 Test Reliability by Subgroup . 88Table 34C. ELA Grade 5 Test Reliability by Subgroup . 89Table 34D. ELA Grade 6 Test Reliability by Subgroup . 90Table 34E. ELA Grade 7 Test Reliability by Subgroup . 91Table 34F. ELA Grade 8 Test Reliability by Subgroup . 92Table 35A. Mathematics Grade 3 Test Reliability by Subgroup . 93Table 35B. Mathematics Grade 4 Test Reliability by Subgroup . 94Table 35C. Mathematics Grade 5 Test Reliability by Subgroup . 95Table 35D. Mathematics Grade 6 Test Reliability by Subgroup . 96Table 35E. Mathematics Grade 7 Test Reliability by Subgroup . 97Table 35F. Mathematics Grade 8 Test Reliability by Subgroup . 98Table 36. Decision Consistency (All Cuts). 100Table 37. Decision Consistency (Level III Cut) . 101Table 38. Decision Agreement (Accuracy) Estimates . 102Table 39. ELA Scale Score Distribution Summary . 103Table 40. ELA Subscore Summary . 104Table 41A. ELA Grade 3 Scale Score Distribution by Subgroup . 105Table 41B. ELA Grade 4 Scale Score Distribution by Subgroup. 106viii

Table 41C. ELA Grade 5 Scale Score Distribution by Subgroup. 107Table 41D. ELA Grade 6 Scale Score Distribution by Subgroup . 108Table 41E. ELA Grade 7 Scale Score Distribution by Subgroup . 109Table 41F. ELA Grade 8 Scale Score Distribution by Subgroup . 110Table 42. Mathematics Scale Score Distribution Summary . 111Table 43. Mathematics Subscore Summary. 112Table 44A. Mathematics Grade 3 Scale Score Distribution by Subgroup . 113Table 44B. Mathematics Grade 4 Scale Score Distribution by Subgroup . 114Table 44C. Mathematics Grade 5 Scale Score Distribution by Subgroup . 115Table 44D. Mathematics Grade 6 Scale Score Distribution by Subgroup . 116Table 44E. Mathematics Grade 7 Scale Score Distribution by Subgroup . 118Table 44F. Mathematics Grade 8 Scale Score Distribution by Subgroup . 119Table 45. ELA Test Performance Level Distributions . 121Table 46A. ELA Grade 3 Performance Level Distribution by Subgroup. 122Table 46B. ELA Grade 4 Performance Level Distribution by Subgroup . 123Table 46C. ELA Grade 5 Performance Level Distribution by Subgroup . 124Table 46D. ELA Grade 6 Performance Level Distribution by Subgroup. 125Table 46E. ELA Grade 7 Performance Level Distribution by Subgroup . 126Table 46F. ELA Grade 8 Performance Level Distribution by Subgroup . 127Table 47. Mathematics Test Performance Level Distributions . 128Table 48A. Mathematics Grade 3 Performance Level Distribution by Subgroup . 129Table 48B. Mathematics Grade 4 Performance Level Distribution by Subgroup .

Table L1. English Language Arts Grade 3 Test Factor Analysis by Subgroup . 176 Table L2. English Language Arts Grade 4 Test Factor Analysis by Subgroup . 177 Table L3. English Language Arts Grade 5 Test Factor Analysis by Subgroup . 179 Table L4.