21st Century Community Learning Centers - Tea.texas.gov
21st Century Community Learning Centers: Texas Afterschool Centers on Education 2014–15 through 2016–17 Evaluation Report JULY 2020 Brenda D. Arellano Neil Naftzger Brenda Ramirez Amanda Sutter American Institutes for Research Joseph Shields Rex Long Gibson Consulting Group Dan Diehl Diehl Consulting Group Submitted to the Texas Education Agency.
Funding Statement This project was funded in its entirety from the federal Elementary and Secondary Education Act (ESEA), Title IV, Part B, 21st Century Community Learning Centers grant through a contract with the Texas Education Agency. The ESEA was reauthorized in 2015 by the Every Student Succeeds Act. Copyright Notice. The materials are copyrighted and trademarked as the property of the Texas Education Agency (TEA) and may not be reproduced without the express written permission of TEA, except under the following conditions: (1) Texas public school districts, charter schools, and Education Service Centers may reproduce and use copies of the Materials and Related Materials for the districts' and schools' educational use without obtaining permission from TEA; (2) residents of the state of Texas may reproduce and use copies of the Materials and Related Materials for individual personal use only without obtaining written permission of TEA; (3) any portion reproduced must be reproduced in its entirety and remain unedited, unaltered and unchanged in any way; and (4) no monetary charge can be made for the reproduced materials or any document containing them; however, a reasonable charge to cover only the cost of reproduction and distribution may be charged. Private entities or persons located in Texas that are not Texas public school districts, Texas Education Service Centers, or Texas charter schools or any entity, whether public or private, educational or non-educational, located outside the state of Texas MUST obtain written approval from TEA and will be required to enter into a license agreement that may involve the payment of a licensing fee or a royalty. For information contact: Copyrights Office, Texas Education Agency, 1701 N. Congress Ave., Austin, TX 78701-1494; phone 512-463-9041; email: copyrights@tea.texas.gov. The State of Texas Assessments of Academic Readiness (STAAR ) and Texas Afterschool Centers on Education (Texas ACE ) are registered trademarks of the Texas Education Agency. Other product and company names mentioned in this report may be the trademarks of their respective owners.
This page intentionally left blank.
21st Century Community Learning Centers: Texas ACEs 2014–15 Through 2016–17 Evaluation Report Contents Page List of Acronyms . xiii Executive Summary .xv Summary Evaluation findings . xvii Chapter 1. Introduction. 1 Evaluation Questions . 3 Connection to a 21st CCLC Conceptual Framework . 4 Organization of the Report . 5 Chapter 2. Grantee and Center Characteristics . 7 Introduction . 7 Overview of Grantees and Centers . 8 Target Student Populations . 9 Texas ACE Operations . 10 Conclusion . 17 Chapter 3. Texas ACE Program Implementation. 20 Introduction . 20 Program Activities and Staffing . 20 Qualitative Insights . 23 Staffing, Partnerships, and Advisory Boards . 27 Family Engagement . 32 Key Implementation and Sustainability Factors . 35 Summary and Recommendations . 37 Chapter 4. Program Quality and Youth Experiences in Texas ACE Programs . 40 Program Quality . 40 Youth Experiences. 45 Summary . 55 Chapter 5. The Impact of the Texas ACE Program on Youth Outcomes . 58 Annual Impact Analyses . 58 Grade-Level Promotion. 67 Two Years of Texas ACE Participation Impact Analyses . 69 Center Characteristics and Center-Level Effects on Youth Outcomes . 71 Summary . 75 Chapter 6. Local Evaluation Summary . 78 AMERICAN INSTITUTES FOR RESEARCH AIR.ORG iv
21st Century Community Learning Centers: Texas ACEs 2014–15 Through 2016–17 Evaluation Report Local Program Evaluation Concept . 78 Recommendations and Next Steps . 87 Chapter 7. Summary of Findings and Recommendations . 90 Chapter 2: Grantee and Center Characteristics . 90 Chapter 3: Program Implementation Characteristics . 91 Chapter 4: Program Quality and Youth Experiences in Programming . 93 Chapter 5: Impact on Texas ACE Program on Youth . 93 Chapter 6: Local Evaluation Summary . 94 Recommendations . 95 References . 97 Appendix A. Chapter 2: Additional Data Tables and Figures . 99 Appendix B. Chapter 3: Additional Data Tables and Figures . 106 Appendix C. Description of Propensity Score Matching and Rasch Analysis . 120 Appendix D. Chapter 5 Additional Figures, Part A: Annual Program Impact Estimates by Grade Level . 126 Appendix E. Chapter 5 Additional Tables, Part B: Effect of Center-Level Characteristics on Student Outcomes . 143 Appendix F. Data Sources . 165 Appendix G. Site Visit Methodology . 169 Appendix H. Site Interview Protocols and Surveys . 172 Appendix I. Additional Documents . 208 AMERICAN INSTITUTES FOR RESEARCH AIR.ORG v
21st Century Community Learning Centers: Texas ACEs 2014–15 Through 2016–17 Evaluation Report Tables Page Table ES.1: 21st Century Community Learning Centers Cycles 7–9 Grantees, by Grant Year and Reporting Years for the Evaluation .xv Table ES.2: Texas Afterschool Centers on Education (Texas ACE) Grantees by Programming Period . xviii Table 1.1: 21st Century Community Learning Centers Cycles 7–9 Grantees, by Grant Year and Reporting Years for the Evaluation . 2 Table 1.2: Texas 21st Century Community Learning Centers Evaluation Objectives Aligned With the 2014–15, 2015–16, and 2016–17 Evaluation Report Chapters and Best Practice Briefs . 3 Table 2.1: Student Target Population Reported by 20 Texas Afterschool Centers on Education in Spring 2017 . 10 Table 2.2: Texas Afterschool Centers on Education Grantees and Centers by Cycle and Programming Period . 11 Table 2.3: Centers by Grantee Maturitya , Cycles 7 and 9 . 12 Table 2.4: Texas Afterschool Centers on Education Grantee Organization Type by Cycle . 14 Table 4.1: Youth and School Age Observation Tools: Domains and Dimensions. 41 Table 4.2: Significant Findings Related to Domain Scores and Center Characteristics . 45 Table 4.3: List of Activities Collected on the Activity Leader Survey . 49 Table 4.4: Number of Centers Identified as Spending the Majority of Time on a Given Activity and the Number of Youth Survey Respondents From Those Centers . 51 Table 5.1: Effect of 2 Years of Participation in Texas Afterschool Centers on Education (Texas ACE) for 60 Days or More Compared With the Range of Annual Program Effects: Outcomes Related to Academic Performance . 70 Table 5.2: Effect of 2 Years of Participation in Texas Afterschool Centers on Education (Texas ACE) for 60 Days or More Compared with the Range of Annual Program Effects: Outcomes Related to Youth Behaviors . 71 Table 7.1: Evaluation Objectives . 90 Table A2.1: Texas Afterschool Centers on Education Goals Reported Across 20 Centers During Spring 2017 Site Visits . 99 Table A2.2: Management of Texas Afterschool Centers on Education Grants by Programming Period, 2014–15 Through 2016–17 . 100 Table A2.3: Number and Percentage of Texas Afterschool Centers on Education by Locale Type and Year . 100 AMERICAN INSTITUTES FOR RESEARCH AIR.ORG vi
21st Century Community Learning Centers: Texas ACEs 2014–15 Through 2016–17 Evaluation Report Table A2.4: Texas Afterschool Centers on Education (Texas ACE) Grantees by Organization Type Across Cycles . 101 Table A2.5: Texas Afterschool Centers on Education (Texas ACE) Student Characteristics, 2014–15 Through 2016–17 . 101 Table A2.6: School-Level Characteristics Across Texas Afterschool Centers on Education (Texas ACE), 2014–15 through 2016–17 . 102 Table A2.7: Texas Afterschool Centers on Education (Texas ACE) Average Number of School Day Absences and Disciplinary Incidents among Program Participants, 2014–15 and 2016–17 . 102 Table A2.8: School-Level Results Across Texas Afterschool Centers on Education (Texas ACE) Average Number of School Day Absences and Disciplinary Incidents among Program Participants, 2014–15 and 2016–17 . 103 Table A2.9: Texas Afterschool Centers on Education (Texas ACE) Percentage of Career and Technical Education Courses where Credits were Earned, 2014–15 and 2016–17 . 103 Table A2.10: School-Level Results Across Texas Afterschool Centers on Education (Texas ACE) Percentage of Career and Technical Education Courses where Credits were Earned, 2014–15 and 2016–17 . 104 Table A2.11: Texas Afterschool Centers on Education Student Outcomes on the Early Reading Indicator, 2014–15 through 2016–17 . 104 Table A2.12: Texas ACE Students and School Averages Achieving STAAR Passing Standard in Reading, Mathematics, and EOC Examinations in 2014–15 through 2016–17 . 105 Table A2.13: Average Number of Summer Program Days and Hours Attended in Texas Afterschool Centers on Education (Texas ACE) 2014–15 through 2016–17 . 105 Table B3.1: Texas ACE Alignment With Program Goals and Objectives . 110 Table B3.2: Establishing Links to the School Day . 110 Table B3.3: New Staff Orientation at Texas Afterschool Centers on Education. 111 Table B3.4: Focus of Professional Development (PD) Attended by Texas Afterschool Centers on Education (Texas ACE) Staff and Perceptions of Helpfulness of PD . 112 Table B3.4 (Continued): Focus of Professional Development (PD) Attended by Texas Afterschool Centers on Education (Texas ACE) Staff and Perceptions of Helpfulness of PD . 113 Table B3.5: Texas Afterschool Centers on Education Reliance on Partnerships for Programming. 114 Table B3.6: Services Provided to Texas Afterschool Centers on Education Programs by Partners. 115 Table B3.7: Texas Afterschool Centers on Education (Texas ACE) Advisory Board Involvement in the Design and Delivery of Activities, Supporting Sustainability Efforts . 115 Table B3.8: Texas Afterschool Centers on Education Program Planning Efforts. 116 AMERICAN INSTITUTES FOR RESEARCH AIR.ORG vii
21st Century Community Learning Centers: Texas ACEs 2014–15 Through 2016–17 Evaluation Report Table B3.9: Methods to Engage Family Members in Texas Afterschool Centers on Education (Texas ACE) Programming . 116 Table B3.10: Family Engagement Activities at Texas Afterschool Centers on Education Programs . 117 Table B3.11: Features of High-Quality Texas Afterschool Centers on Education (Texas ACE) Programs . 118 Table B3.12: Student Data Used to Monitor Performance at Texas Afterschool Centers on Education (Texas ACE) Programs . 119 Table B3.13: Program Quality Data Used to Monitor Performance at Texas Afterschool Centers on Education Programs. 119 Table C1: Outcomes and Operationalizations . 123 Table E5.1: Phase 1 Center-Level Relationships: State of Texas Assessments of Academic Readiness (STAAR) Mathematics . 145 Table E5.1 (Continued): Phase 1 Center-Level Relationships: State of Texas Assessments of Academic Readiness (STAAR) Mathematics . 146 Table E5.2: Phase 1 Center-Level Relationships: State of Texas Assessment of Academic Readiness (STAAR) Reading . 147 Table E5.2 (Continued): Phase 1 Center-Level Relationships: State of Texas Assessment of Academic Readiness (STAAR) Reading . 148 Table E5.3: Phase 1 Center-Level Relationships: Attendance . 149 Table E5.3 (Continued): Phase 1 Center-Level Relationships: Attendance . 150 Table E5.4: Phase 1 Center-Level Relationships: Discipline. 151 Table E5.5: Phase 1 Center-Level Relationships: Career and Technical Education Credits . 153 Table E5.5 (Continued): Phase 1 Center-Level Relationships: Career and Technical Education Credits . 154 Table E5.6: Pooled Effect of High Average Attendance on Student Outcomes . 155 Table E5.7: Pooled Effect of College Student(s) or Paraprofessional(s) Staffing on Student Outcomes . 155 Table E5.8: Pooled Effect of High Summertime Programming Hours on Student Outcomes . 156 Table E5.9: Pooled Effect of School-Day Teacher Staffing on Student Outcomes . 156 Table E5.10: Pooled Effect of Serving a Higher Need Population on Student Outcomes . 157 Table E5.11: Annual Effect of High Average Attendance on Student Outcomes . 158 Table E5.12: Annual Effect of College Student(s) or Paraprofessional(s) Staffing on Student Outcomes . 159 AMERICAN INSTITUTES FOR RESEARCH AIR.ORG viii
21st Century Community Learning Centers: Texas ACEs 2014–15 Through 2016–17 Evaluation Report Table E5.13: Annual Effect of High Summertime Programming Hours on Student Outcomes . 160 Table E5.14: Annual Effect of School-Day Teacher Staffing on Student Outcomes . 161 Table E5.15: Annual Effect of Serving a Higher Need Population on Student Outcomes . 162 Table E5.16: Mean Effects Comparing Centers Targeting Students Who Are at Risk and Centers That Did Not: Site Visit Sample . 163 Table E5.17. Mean Effects Comparing Centers Obtaining Youth Input and Centers That Did Not: Site Visit Sample . 163 Table E5.18: Mean Effects Comparing Centers Using a Quality Tool and Centers That Did Not: Site Visit Sample . 164 Table E5.19: Mean Effects Comparing Centers With Higher Program Quality Assessment (PQA) Scores and Centers With Lower PQA Scores: Site Visit Sample . 164 Table F1: Analytic Approach by Data Source . 165 Table F1 (Continued): Analytic Approach by Data Source . 166 Table F1 (Continued): Analytic Approach by Data Source . 167 Table I1: Local Evaluation Timeline for Implementation Year 1: 2017–18 . 174 Table I1 (Continued): Local Evaluation Timeline for Implementation Year 1: 2017–18 . 175 Table I2: Affiliations and Roles of Local Evaluation Advisory Group Participants, 2017–18 . 176 Table I3: Participating Grantees and Centers in the Local Evaluation Support Initiative, 2017–18 . 177 Table I4: Timeline. 182 Table I5: Examples of Key Performance Indicators . 189 Table I6: Overview of Common Quality Assessment Measures. 194 Table I7: Robust Measurement Tool Criteria . 197 Table I8: Available Training and Resources Aligned to the Measure . 197 Table I9: Technical Properties Comparison . 197 Table I10: Texas Afterschool Centers on Education 21st Century Community Learning Centers Planning Worksheet (Example) . 200 Table I11: Texas Afterschool Centers on Education 21st Century Community Learning Centers Planning Worksheet . 201 Table I12: Evaluation Planning . 202 Table I12 (Continued): Evaluation Planning . 203 AMERICAN INSTITUTES FOR RESEARCH AIR.ORG ix
21st Century Community Learning Centers: Texas ACEs 2014–15 Through 2016–17 Evaluation Report Figures Page Figure ES.1: Percentage of Texas Afterschool Centers on Education (Texas ACE) Participants’ Time (Hours) Spent on Activities Categorized by Subject in 2016–17 . xix Figure ES.2: Average School-Age Program Quality Assessment and Youth Program Quality Assessment Scores by Program Quality Assessment Domain . xxii Figure 1.1: A Conceptual Framework for How Afterschool Programs Can Have an Impact on Youth Participants . 5 Figure 2.1: Major Objectives Reported by 20 Texas Afterschool Centers on Education in Spring 2017 . 9 Figure 2.2: Texas Afterschool Centers on Education (Texas ACE) by Locale Type From 2014–15 Through 2016–17 . 13 Figure 2.3: Percentage Participation in Texas Afterschool Centers on Education (Texas ACE) as Measured in Days . 17 Figure 3.1: Student Time by Activity Type and Reporting Period During 2016–17 . 21 Figure 3.2: Percent of Texas ACE Participants’ Time (Hours) Spent on Activities Categorized by Subject in 2016–17 . 22 Figure 3.3: Connections to the School Day, Spring 2017 . 26 Figure 3.5: Overview of Family Engagement and Skill-Building Activities Cited by 20 Texas Afterschool Centers on Education (Texas ACE) Programs in Spring 2017. 32 Figure 3.4: Methods of Engaging Families with Texas Afterschool Centers on Education (Texas ACE) . 34 Figure 3.6: Student Data Used to Monitor Performance by 20 Texas ACE Centers, Spring 2017 . 36 Figure 4.1: Minimum and Maximum Youth Program Quality Assessment Scores for a Sample of Eight Middle School and High School Centers . 42 Figure 4.2: Minimum and Maximum School-Age Program Quality Assessment Scores for a Sample of 12 Elementary Centers . 43 Figure 4.3: Overall SAPQA and YPQA Scores by Program Quality Assessment Domain . 44 Figure 4.4: Youth Experiences in Activities Across Higher Scoring Versus Lower Scoring Centers . 46 Figure 4.5: Youth Experiences in Centers Using an External Quality Assessment (QA) Versus Nonusing Centers . 47 Figure 4.6: Youth Experiences in Activities in Middle and High School Programs Versus Elementary Programs . 48 AMERICAN INSTITUTES FOR RESEARCH AIR.ORG x
21st Century Community Learning Centers: Texas AC
Table 2.2: Texas Afterschool Centers on Education Grantees and Centers by Cycle and Programming Period 11 . Table 2.3: Centers by Grantee Maturitya , Cycles 7 and 9 12 . Table 2.4: Texas Afterschool Centers on Education Grantee Organization Type by Cycle 14 . Table 4.1: Youth and School Age Observation Tools: Domains and Dimensions 41
Table 1.1. 21st Century Community Learning Centers Cycles 8 10 Grantees, by Grant Years Represented in This Evaluation Report 2 . Table 1.1. 21st Century Community Learning Centers Cycles 8 10 Grantees, by Grant Years Represented in This Evaluation Report 2 . Table 2.1. Texas ACE Centers Key Measures of Program Implementation 9 . Table 2.2.
concept “21st OR twenty-first century skills”, this has, also, been searched through the terms: “21st OR twenty-first century competencies”, “21st OR twenty-first century literacy”, “21st OR twenty-first century learn*”. The search action returned 116 results. The references sections of these texts were
To understand the 21st Century Life and Career Standards To gain knowledge in the Framework for 21st Century Inter-disciplinary themes To develop skills in planning cross-curricular activities related to the 21st Century skills To learn practical strategies in how to infuse 21st Century skills
21st Century Assessment to Promote 21st Century Learning: The Benefits of Blinking Valerie J. Shute, Vanessa P. Dennen, Yoon-Jeon Kim, Oktay Donmez, & Chen-Yen Wang Abstract What competencies do kids need to succeed in the 21st century, and how do these skills differ from those reflected in current state standards?
The 21st Century Teaching Learning, and Leading (CTLL) Technical Working Group (TWG) developed a set of tools that provide principals a means to understand the shift from a 20th century principal to a 21st Century School Principal. The platform of the 21st CTLL is collaboration throughou t the school,
organizations that seek accreditation (Source: NCCAM) Dialysis centers Endoscopy centers Imaging centers Infusion therapy services Laser centers Lithotripsy services MRI centers Plastic surgery centers Podiatric clinics Radiation/oncology clinics Rehabilitation centers Sleep centers Urgent/emergency care centers
with the definition, demand and delivery of 21st Century Skills, with a particular focus on low- and middle-income countries (LMICs). In looking at definitions of ‘21st Century Skills’, Section 2 of this study finds that there is a broad range of available literature discussing 21st Century Skills, including a number of key synthesis studies.
“21st century skills” and “college and career readiness” have recently become watchwords in education. The Partnership for 21st century skills advocates adoption of local, state, and federal policies that support explicit integration of 21st century skills into instruction for all students (P21, 2009).
