A FRAMEWORK ANALYSIS TIMSS Advanced 2015 AND

RESEARCH IN REVIEW 2016-1A Framework AnalysisTIMSS Advanced 2015and Advanced PlacementCalculus & PhysicsChristopher Lazzaro, Lee Jones, David C. Webb, Ryan Grover,F. Tony Di Giacomo, and Katherine Adele Marino00078 002

Christopher Lazzaro, Ph.D. – Director, Science Education,College BoardLee Jones, Ph.D. – Independent ConsultantDavid C. Webb, Ph.D. – Associate Professor, Math Education,University of Colorado BoulderRyan Grover – Ph.D. Candidate, University of Colorado, BoulderF. Tony Di Giacomo, Ph.D. – Director of International Research,College BoardKatherine Adele Marino – Ph.D. Candidate, PennsylvaniaState UniversityAbout the College BoardThe College Board is a mission-driven not-for-profit organization thatconnects students to college success and opportunity. Founded in1900, the College Board was created to expand access to highereducation. Today, the membership association is made up of over6,000 of the world’s leading educational institutions and is dedicatedto promoting excellence and equity in education. Each year, theCollege Board helps more than seven million students prepare fora successful transition to college through programs and services incollege readiness and college success — including the SAT and theAdvanced Placement Program . The organization also serves theeducation community through research and advocacy on behalf ofstudents, educators and schools. For further information,visit www.collegeboard.org. 2016 The College Board. College Board, Advanced Placement Program, SAT,and the acorn logo are registered trademarks of the College Board. ACES andAdmitted Class Evaluation Service are trademarks owned by the College Board.Visit the College Board on the Web: www.collegeboard.org.For more information on CollegeBoard research and data, visitwww.collegeboard.org/research.

ContentsExecutive Summary . 5Introduction . 8Methodology . 8Population . 9Purpose. 9Overview of AP and TIMSS Advanced Frameworks .10AP Physics 1 and 2 Framework.10AP Calculus Framework .13TIMSS Advanced 2015 Frameworks .16Alignment Methodology.20Categorical Concurrence — Content Areas .20Depth of Knowledge — Cognitive Domains .22Balance of Representation — Item Distribution.27Alignment Analysis.29Physics .29Calculus.44

Findings and Conclusion.61Overall Summary .61Categorical Concurrence .61Depth of Knowledge.61Balance of Representation.61Conclusion .62References.63

TablesTable 1. Big Ideas for AP Physics 1 and AP Physics 2 . 11Table 2. Big Idea 1: Objects and Systems. . 11Table 3. AP Science Practices. 12Table 4. Target Percentages of the TIMSS Advanced 2015 Mathematics Assessment. 18Table 5. Target Percentages of the TIMSS Advanced 2015 Physics Assessment . 19Table 6. TIMSS Advanced Physics Cognitive Domains . 23Table 7. TIMSS Advanced Mathematics Cognitive Domain and Thinking Processes . 24Table 8. Examples of AP Calculus Tasks Within TIMSS AdvancedMathematics Framework . 25Table 9. Number and Description of Tasks in Calculus AB and BC Exams. 26Table 10. Emphasis on Cognitive Domains Across Frameworks . 39Table 11. Total Number of Tasks Corresponding to Each Domain inTIMSS Advanced Mathematics. 54Table 12. Total Number of Tasks Corresponding to Each Domain inTIMSS Advanced, Expanded . 54

FiguresFigure 1. Hierarchy of the AP Physics Framework . 13Figure 2. Evaluation of degree of alignment: mechanics and thermodynamics . 30Figure 3. Evaluation of degree of alignment: electricity and magnetism. 31Figure 4. Evaluation of degree of alignment: wave phenomena and atomic andnuclear physics. 32Figure 5. Topic area weighting in TIMSS Advanced 2015 Physics and AP Physics 1. 41Figure 6. Topic area weighting in TIMSS Advanced 2015 Physics and AP Physics 2. 42Figure 7. Topic area weighting in TIMSS Advanced 2015 Physics andAP Physics 1 and 2 combined. 42Figure 8. Evaluation of degree of alignment: algebra . 46Figure 9. Evaluation of degree of alignment: calculus. 47Figure 10. Evaluation of degree of alignment: geometry. 48Figure 11. Example of an application task from AP Calculus Exam. 55Figure 12. Example of a reasoning task from AP Calculus Exam . 56Figure 13. Example of a justification task from AP Calculus Exam. 57Figure 14. Example of a recognize task from AP Calculus Exam . 58Figure 15. Comparison of topic area weighting in TIMSS Advanced Mathematicsand AP Calculus AB. 59Figure 16. Comparison of topic area weighting in TIMSS Advanced Mathematicsand AP Calculus BC . 59

TIMSS Advanced 2015 and Advanced PlacementExecutive SummaryWhile the results of the TIMSS Advanced 2015 assessments are not yet known, in recentyears U.S. students have underperformed relative to their international counterparts oninternational assessments such as TIMSS and PISA. While international assessment data arefrequently used to compare performance cross-culturally, the analysis of these data oftenlacks adequate contextualization for country-by-country comparisons. To ensure maximumutility of the assessments, an examination of each participating country’s curriculum contentand standards in relation to those on the international assessment is critical. Only then canone make cross-cultural comparisons effectively.The current study investigates the alignment between AP Physics and AP Calculusframeworks as compared to the TIMSS Advanced Physics and Mathematics frameworks. Dueto the rigor of AP course work and the performance of AP students versus non-AP studentson college course work, we expect AP student performance on TIMSS Advanced may exceedthe performance of the general population. Moreover, meaningful comparisons of countryachievement rankings are not possible without a study that considers alignment, and thus,there may also be additional compelling findings from the future performance report that canmake a potentially significant contribution to the literature, and to policy discussions.The methodology applied in this alignment study is modeled after Norman Webb (Webb,1997). External alignment and curriculum specialists in mathematics and physics wereengaged to evaluate the alignment of AP Physics 1: Algebra-Based and AP Physics 2:Algebra-Based Curriculum framework and AP Calculus framework to the TIMSS AdvancedFrameworks according to three criteria: (1) Categorical concurrence, (2) Depth of knowledge,(3) Balance of representation. The mathematics and science groups approached the alignmentcriteria commensurate with their respective frameworks.For categorical concurrence in physics, degree of alignment between the AP learningobjectives and each of the TIMSS Advanced topics were rated as: Complete alignment — The learning objective aligns with the full description of the TIMSStopic. Partial alignment — The learning objective aligns to a part of the TIMSS topic description. Extended alignment — The learning objective aligns to all or part of the TIMSS topicdescription but also includes knowledge and/or uses of that knowledge that extendbeyond what is described in the TIMSS topic. Not covered — The learning objective does not align with any of the 23 TIMSS topics.Of the 250 AP Physics 1 and 2 learning objectives, 180 (72%) are covered within one or moreof the 23 TIMSS Advanced Physics topics, while 70 (28%) of the learning objectives are notcovered in the TIMSS Advanced Physics Framework. After aligning the AP learning objectivesto the TIMSS topic, a holistic judgment was made as to whether the collection of AP learningobjectives represents partial, complete, or extended alignment of the TIMSS topic as a whole.None of the 23 TIMSS Advanced Physics topics had “no alignment” with the AP Physics1 and 2 curriculum frameworks, as each physics topic was represented in at least one APlearning objective, two (9%) had partial alignment, 14 (61%) had complete alignment, andseven (30%) had extended alignment.College Board Research in Review5

TIMSS Advanced 2015 and Advanced PlacementThe depth of knowledge alignment for physics was measured by considering the threecognitive domains (knowing, applying, and reasoning) outlined in the TIMSS Advanced PhysicsFramework and the science practices outlined in the AP Physics learning objectives. Theleast aligned cognitive domain was knowing, containing only 11.6% of the total AP learningobjectives. Applying was the domain with the strongest alignment with the AP learningobjectives, with reasoning also being strongly aligned.For balance of representation in physics, the percentages of AP Physics 1 and 2 itemsaligned with each of the seven TIMSS Advanced Physics topic areas were determined.Important differences were found between the TIMSS Advanced Physics assessment andthe AP Physics Exams in the proportional emphasis on certain topic areas. Most notably, APstudents who have completed only AP Physics 1 and then take the TIMSS Advanced Physicsassessment will encounter numerous items in Magnetism and Electromagnetic Induction andAtomic and Nuclear Physics (comprising 34% of the TIMSS assessment) that are not includedin AP Physics 1.For categorical concurrence in calculus, degree of alignment methodology between the APcontent topics and each of the TIMSS Advanced topics followed the same strategy used bythe physics team, defined as follows: Complete alignment — The content topic aligns with the full description of the TIMSStopic. Partial alignment — The content topic aligns to a part of the TIMSS topic description. Extended alignment — The content topic aligns to all or part of the TIMSS topicdescription but also includes knowledge and/or uses of that knowledge that extendsbeyond what is described in the TIMSS topic. Not covered — The content topic does not align with any of the 22 TIMSS topics.Of the 46 AP Calculus AB content topics, all are at least partially covered or extend beyondone or more of the 22 TIMSS Advanced Mathematics topics, and 44 (96%) of the AP CalculusAB content topics are completely covered. Of the 67 AP Calculus BC content topics, all are atleast partially covered or extend beyond one or more of the 22 TIMSS Advanced Mathematicstopics, and 56 (84%) of the AP Calculus BC content topics are completely covered. When theAP Calculus AB framework was compared to the 22 TIMSS Advanced Mathematics topics,three TIMSS topics (14%) had no alignment, seven (32%) had partial alignment, 10 (45%)had complete alignment, and two (9%) had extended alignment. The comparison of the APCalculus BC framework to the TIMSS Advanced Mathematics topics resulted in two TIMSStopics (9%) had no alignment, five (23%) had partial alignment, nine (41%) had completealignment, and six (27%) had extended alignment.To assess depth of knowledge in calculus, each of the 80 AP Calculus AB and BC items wasassigned a TIMSS Advanced cognitive domain. AP Calculus content topics were categorizedaccording to the TIMSS topics separately for AP Calculus AB and BC to assess balance ofrepresentation across frameworks. Of the AP Calculus AB content topics, 4.5% did notmatch any of the TIMSS topics. When focusing on the balance of representation for the threecalculus topics, the derivatives representation (43.2%) is almost double compared to therepresentation of limits and integrals (20.5%). The AP Calculus BC balance of representationranges from 1.5% to 28.4% across seven of the eight topics, with no representation fortrigonometry. Compared to the AP Calculus AB content topics, a much greater proportion6College Board Research in Review