21st Century Community Learning Centers: Texas Afterschool Centers On .
21st Century Community LearningCenters: Texas Afterschool Centerson EducationFinal Evaluation Report: 2016–17 to 2018–19JULY 2021Neil Naftzger Brenda D. ArellanoAmerican Institutes for Research Joseph Shields Danial Hoepfner Alejandro AllenGibson Consulting GroupDan DiehlDiehl Consulting GroupSubmitted to the Texas Education Agency
Funding StatementThis project was funded in its entirety from the federal Elementary and Secondary Education Act (ESEA), Title IV,Part B, Nita M. Lowey 21st Century Community Learning Centers grant through a contract with the Texas EducationAgency. The ESEA was reauthorized in 2015 by the Every Student Succeeds Act.Copyright Notice. The materials are copyrighted and trademarked as the property of the Texas EducationAgency (TEA) and may not be reproduced without the express written permission of TEA, except under the followingconditions: (1) Texas public school districts, charter schools, and Education Service Centers may reproduce and usecopies of the Materials and Related Materials for the districts' and schools' educational use without obtainingpermission from TEA; (2) residents of the state of Texas may reproduce and use copies of the Materials and RelatedMaterials for individual personal use only without obtaining written permission of TEA; (3) any portion reproducedmust be reproduced in its entirety and remain unedited, unaltered and unchanged in any way; and (4) no monetarycharge can be made for the reproduced materials or any document containing them; however, a reasonable chargeto cover only the cost of reproduction and distribution may be charged. Private entities or persons located in Texasthat are not Texas public school districts, Texas Education Service Centers, or Texas charter schools or any entity,whether public or private, educational or non-educational, located outside the state of Texas MUST obtain writtenapproval from TEA and will be required to enter into a license agreement that may involve the payment of a licensingfee or a royalty. For information contact: Copyrights Office, Texas Education Agency, 1701 N. Congress Ave., Austin,TX 78701-1494; phone 512-463-9041; email: copyrights@tea.texas.gov.The State of Texas Assessments of Academic Readiness (STAAR ) and Texas Afterschool Centers on Education (Texas ACE ) are registered trademarks of the Texas Education Agency. Other product and company namesmentioned in this report may be the trademarks of their respective owners.Notice of Trademark: “American Institutes for Research” and “AIR” are registered trademarks. All other brand,product, or company names are trademarks or registered trademarks of their respective owners.
This page intentionally left blank.
21st Century Community Learning Centers: Texas Afterschool Centers on Education Evaluation ReportContentsPageList of Abbreviations .xiExecutive Summary . xiiChapter 1: Introduction. 1Evaluation Objectives . 1Summary of Findings from Previous Reports . 2Evaluation Questions . 4Organization of the Report . 6Chapter 2: Representation of the Site Visit Samples . 8Introduction . 8Methods for Selecting Texas ACE Centers for Site Visits . 9Profile and Representativeness of Site Visit Centers . 10Chapter 3. Center-Level Characteristics, Program Attendance, and School-Related Outcomes . 21Introduction . 21Center Characteristics Examined . 22Texas ACE Attendance Outcomes . 23School-Related Outcomes . 26Data on Observed Quality . 29Organizational Processes . 40Data on Program Activity Practices . 53Data on Youth Experiences in Programming . 61Intermediate Youth-Reported Outcomes . 71Chapter Summary. 76Chapter 4: The Impact of Texas Afterschool Centers on Education on Youth Outcomes . 80Introduction . 80APT-O Mathematics Practices and STAAR Scores . 83Active Forms of Learning and Disciplinary Incidents. 86Chapter Summary. 90Chapter 5: Local Evaluation Summary . 93Introduction . 93Local Program Evaluation Concept . 93Chapter 6. Summary of Findings and Recommendations . 101Results from Correlational Analyses . 102Active Forms of Learning and Disciplinary Incidents. 105Recommendations . 105References . 107iv
21st Century Community Learning Centers: Texas Afterschool Centers on Education Evaluation ReportAppendix A. Chapter 2: Site Visit Sample Selection. 111Appendix B. Chapter 2: Additional Tables . 115Appendix C. Chapter 3: Summary of HLM and Regression Analyses . 121Appendix D. Description of Propensity Score Matching, Hierarchical Linear Modeling and RaschAnalysis . 148Appendix E. Chapter 3 Youth Experiences in Programming . 154Appendix F. Data Sources . 157Appendix G. Site Visit Methodology . 160Appendix H. Chapter 5 Local Evaluation Artifacts . 164v
21st Century Community Learning Centers: Texas Afterschool Centers on Education Evaluation ReportTablesPageTable 1.1. 21st Century Community Learning Centers Cycles 8–10 Grantees, by Grant YearsRepresented in This Evaluation Report . 2Table 2.1. Texas ACE Centers Key Measures of Program Implementation . 9Table 2.2. Texas ACE Centers by Locale Type, Centers Visited Compared to Statewide Centers2017–2019, by Year . 11Table 2.3. Percentage of Participants’ Time (Hours) in Texas ACE by Activity Type during theRegular School Year, 2017–2019 . 12Table 2.4. Percentage of Participants’ Time (Hours) in Texas ACE by Activity Type during theSummer, 2017–2019. 13Table 2.5. Percentage of Participants’ Time (Hours) in Texas ACE Spent on Various ContentAreas during the Regular School Year, 2017–2019 . 14Table 2.6. Percentage of Participants’ Time (Hours) in Texas ACE Spent on Various ContentAreas during Summer, 2017–2019 . 15Table 2.7. Percentage of Students Participating in Texas ACE in 2017–2019 during the RegularSchool Year, by Number of Days Attended . 16Table 2.8. Percentage of Students Participating in Texas ACE in 2017–2019 during the Summer,by Number of Days Attended . 16Table 2.9. Percentage of Texas ACE Centers Staff during 2017–2019, by Position Type . 17Table 2.10. Texas ACE Students Demographic Characteristics during 2017–2019 . 18Table 2.11. Texas ACE, Substantive Differences between Site Visit Samples and Full Domain ofCenters, 2017–2019. 19Table 3.1. Average Number of Hours Students Attended Texas ACE Programming during theSchool Year – Site Visit Samples . 23Table 3.2. Average Number of Calendar Days between the First and Last Day of Attendance inProgramming during the School Year– Site Visit Samples. 24Table 3.3. Average Number of Different Texas ACE Activities Attended during the School Year –Site Visit Samples . 24Table 3.4. Percentage of Students Attending Texas ACE Programming Who Also Attended ACEProgramming in the Preceding Summer or School Year at the Same Center . 25Table 3.5. Summary of Average Center-Level Effects across All Site Visit Centers . 27Table 3.6. Percentage of Centers, Average Effect, and Range of Effects by Centers Having Eithera Positive or Negative Effect on Student Outcomes: 1 Year of Participation . 28Table 3.7. Percentage of Centers, Average Effect, and Range of Effects by Centers Having Eithera Positive or Negative Effect on Student Outcomes: 2 Years of Participation . 29Table 3.8. Summary of Assessment of Program Practices Tool – Observation (APT-O) ContentScales and Example Items. 32vi
21st Century Community Learning Centers: Texas Afterschool Centers on Education Evaluation ReportTable 3.9. Point-of-Service Quality Areas Positively Associated with Texas ACE ProgramAttendance Outcomes. 35Table 3.10. Point-of-Service Quality Areas Positively Associated with Center-Level Effects byLevel of Texas ACE Participation and Outcome . 39Table 3.11. Organizational Processes Examined by Connection to Texas ACE BlueprintComponents . 40Table 3.12. Organizational Processes Positively Associated with Texas ACE AttendanceOutcomes . 48Table 3.13. Organizational Processes Positively Associated with Center-Level Effects by Level ofTexas ACE Participation and Outcome . 51Table 3.14. List of Activity Practices Examined on the Activity Leader Survey . 54Table 3.15. Activity Practices Positively Associated with Texas ACE Attendance Outcomes . 57Table 3.16. Activity Practices Positively Associated with Center-Level Effects by Texas ACEParticipation and Outcome . 59Table 3.17. Youth Experiences Positively Associated with Texas ACE Attendance Outcomes . 68Table 3.18. Youth Experiences Positively Associated with Center-Level Effects by Level of TexasACE Participation and Outcome . 70Table 3.19. Youth-Reported Outcomes Positively Associated with Texas ACE AttendanceOutcomes . 75Table 4.1 Average Center-Level Effects on Disciplinary Incidents by Activity Practice . 81Table 4.2. Summary of Effectiveness Analyses Performed. 83Table A2.1. Summary of Key Performance Indicators by Category . 111Table A2.1. (continued) Summary of Key Performance Indicators by Category . 112Table A2.1. (continued) Summary of Key Performance Indicators by Category . 113Table B2.1. Texas ACE Grantees by Grantee Organization Type: Grantees Visited Compared toStatewide Grantees 2017–2019, by Year . 115Table B2.2. Activities Offered in Texas ACE Centers by Activity Type during the Regular SchoolYear, 2017–2019. 115Table B2.3. Activities Offered in Texas ACE Centers by Activity Type during Summer, 2017–2019 . 116Table B2.4. Content Area of Activities Offered in Texas ACE Centers during the Regular SchoolYear, 2017–2019. 116Table B2.5. Content Area of Activities Offered in Texas ACE Centers during Summer, 2017–2019 . 117Table B2.6. Grades Served at Texas ACE Centers during 2017–2019 . 118Table B2.7. Texas ACE Students Achieving State of Texas Assessments of Academic Readiness(STAAR) Passing Standard in Reading, Mathematics, and End-of-Course (EOC) Examinations in2017–2019 . 119vii
21st Century Community Learning Centers: Texas Afterschool Centers on Education Evaluation ReportTable B2.8. Texas ACE School Characteristics during 2017–2019 . 120Table C3.1. Student Characteristics Included in Texas ACE Program Attendance Models byOutcome Examined. 124Table D1. Outcomes and Operationalizations . 151Table E3.1. Percentage of Responses by Response Category: Perceptions of Activity LeadersScale . 154Table E3.2. Percentage of Responses by Response Category: Perceptions of Other Youth Scale. 154Table E3.3. Percentage of Responses by Response Category: Opportunities for Agency Scale . 155Table F1. Analytic Approach by Data Source . 157Table G1. Youth and School-Age Observation Tool, Domains and Dimensions . 161Table G2. Overview of Data Collected During 2017, 2018, and 2019 Texas Afterschool Center(Texas ACE) Site Visits . 161Table H1. Participating Grantees and Centers in the Local Evaluation Support Initiative, 2019–20. 164Table H2. Texas ACE Local Evaluation Tutorials 2019–20 . 164Table H3. Local Evaluation Timeline for 2019–20 . 165viii
21st Century Community Learning Centers: Texas Afterschool Centers on Education Evaluation ReportFiguresPageFigure ES.1. Center Characteristics Found to Be Significantly and Positively Associated with TexasACE Program Attendance Outcomes . xviFigure ES.2. Center Characteristics Found to Be Significantly and Positively Associated withSchool-Related Outcomes . xviiiFigure ES.3. Variables Found to Be Significantly and Positively Associated with More Than OneTexas ACE Program Attendance and/or School-Related Outcome . xxivFigure 3.1. A Conceptual Framework for How Afterschool Programs Can Have an Impact onYouth Participants . 22Figure 3.2. Average Program Quality Assessment (PQA) Scores by Scale for CentersRepresented in the Site Visit Sample . 31Figure 3.3. Prevalence of Assessment of Program Practices Tool – Observation (APT-O)Practices within Observed Centers, by Content Area . 33Figure 3.4. Prevalence of Staff-Based Assessment of Program Practices Tool – Observation (APTO) Practices within Observed Centers, by Content Area . 33Figure 3.5. Prevalence of Youth-Based Assessment of Program Practices Tool – Observation(APT-O) Practices within Observed Centers, by Content Area . 34Figure 3.6. Percentage of Centers Indicating a Given Program Goal . 41Figure 3.7. Percentage of Centers Indicating a Given Target Population Definition . 42Figure 3.8. Percentage of Centers Indicating a Given Role for the Advisory Board. 43Figure 3.9. Percentage of Centers Providing High School Equivalency (HSE) or English as aSecond Language (ESL) Programming for Parents and Adult Family Members . 44Figure 3.10. Percentage of Centers Indicating a Data Use or Evaluation Practice . 45Figure 3.11. Percentage of Centers Indicating a Particular Staffing or Operational Practice. 46Figure 3.12. Average Percentage of Activities with a Given Practice for Centers Represented inthe Site Visit Sample . 56Figure 3.13. Survey Items Measuring Perceptions of Activity Leaders . 62Figure 3.14. Survey Items Measuring Perceptions of Other Youth . 62Figure 3.15. Perceptions of Activity Leaders and Other Youth Scales: Average Percentage ofStudents by Response Category for Centers Represented in the Site Visit Sample . 63Figure 3.16. Survey Items Measuring Opportunities for Agency . 64Figure 3.17. Opportunities for Agency: Average Percentage of Students by Response Categoryfor Centers Represented in the Site Visit Sample . 65Figure 3.18. Summary of Responses to Key Constructs from the End-of-Session Survey: AveragePercentage of Students by Response Category for Centers Represented in the Site Visit Sample . 67ix
21st Century Community Learning Centers: Texas Afterschool Centers on Education Evaluation ReportFigure 3.19. Average Percentage of Students Indicating a Particular Program Impact for CentersRepresented in the Site Visit Sample . 74Figure 3.20. Center Characteristics Found to Be Significantly and Positively Associated withTexas ACE Program Attendance Outcomes . 77Figure 3.21. Center Characteristics Found to Be Significantly and Positively Associated withSchool-Related Outcomes . 79Figure 4.1. Effect of Texas ACE Participation on Mathematics: Grades 4–8 in Centers withGreater Adoption of Mathematics Practices Outlined in the Assessment of Program PracticesObservation Tool (APT-O) . 84Figure 4.2. Effect of Texas ACE Participation on Reading: Grades 4–8 in Centers with GreaterAdoption of Mathematics Practices Outlined in the Assessment of Program Practices ObservationTool (APT-O) . 85Figure 4.3. Effect of Texas ACE Participation on Mathematics and Reading: Grades 4–8 inCenters with Greater Adoption of Mathematics Practices Outlined in the Assessment of ProgramPractices Observation Tool (APT-O) . 86Figure 4.5. Effect of Texas ACE Participation on Disciplinary Incidents: Grades 1–8 in CentersWith Greater Adoption of Active Forms of Learning . 87Figure 4.6. Effect of Texas ACE Participation on Disciplinary Incidents: Grades 1–5 in Centerswith Greater Adoption of Active Forms of Learning . 88Figure 4.7. Effect of Texas ACE Participation on Disciplinary Incidents: Grades 6–8 in Centerswith Greater Adoption of Active Forms of Learning . 89Figure 4.8. Effect of Texas ACE Participation on Disciplinary Incidents: Grades 1–8 in Centerswith Greater Adoption of Active Forms of Learning . 90Figure 5.1. Overview of Local Evaluation Key Principles . 94Figure 6.1. Variables Found to Be Significantly and Positively Associated with More Than OneTexas ACE Program Attendance and/or School-Related Outcome . 104x
21st Century Community Learning Centers: Texas Afterschool Centers on Education Evaluation ReportList of Abbreviations21st CCLC21st Century Community Learning CentersAIRAmerican Institutes for ResearchAPT-OAssessing Afterschool Program Practices Observation ToolCTEcareer and technical educationEDU.S. Department of EducationESexecutive summaryEOCend-of-courseESEAElementary and Secondary Education ActESLEnglish as a second languageESSAEvery Student Succeeds ActHLMhierarchical linear modelingKPIkey performance indicatorLEAGLocal Evaluation Advisory GroupLESILocal Evaluation Support InitiativeNCESNational Center for Education StatisticsNIOSTNational Institute on Out-of-School TimeNYSANNew York State Association of NeuropsychologyPEIMSPublic Education Information Management SystemPQAProgram Quality AssessmentPSMpropensity score matchingRFArequest for applicationsSAYOSurvey of Academic Youth OutcomesSACERSSchool-Age Care Environment Rating ScaleSAPQASchool-Age Program Quality AssessmentSTAAR State of Texas Assessments of Academic Readiness STEMscience, technology, engineering, and mathematicsTAPRTexas Academic Performance ReportTEATexas Education AgencyTexas ACETexas Afterschool Centers on EducationTx21stTexas 21st Student Tracking SystemYPQAYouth Program Quality Assessmentxi
21st Century Community Learning Centers: Texas Afterschool Centers on Education Evaluation ReportExecutive SummaryThe 21st Century Community Learning Centers (21st CCLC) program, funded by Title IV, Part B of theElementary and Secondary Education Act, as renewed by the Every Student Succeeds Act (ESSA),provides grant funding to states to support “academic enrichment opportunities during non-school hoursfor children, particularly students who attend high-poverty and low-performing schools” (U.S. Departmentof Education [ED], 2018). By means of state-level subgrant competitions, states allocate this funding toschools, community-based organizations, faith-based institutions, and other agencies to provide thisprogramming in their communities. Community learning centers are meant to “offer students a broadarray of additional services, programs, and activities that are designed to reinforce and complement theregular academic program of participating students” (ED, 2015, p. 233). 1Since 2002, the Texas Education Agency (TEA) has provided 21st CCLC funding to hundreds of granteesand supported thousands of community learning centers, also known as Texas Afterschool Centers onEducation (Texas ACE), across the state. This evaluation report focuses on a sample of 60 Texas ACEcenters that the evaluation team visited in spring 2017, spring 2018, and spring 2019. The focus of this reportis to examine how key center characteristics associated with the 60 Texas ACE centers represented in the sitevisit samples were associated with Texas ACE program attendance and school-related outcomes.Evaluation ObjectivesThis report is the culminating product of a 4-year evaluation of the Texas ACE program undertaken by theAmerican Institutes for Research (AIR), in collaboration with the Gibson Consulting Group and the DiehlConsulting Group. The evaluation of the Texas ACE program was designed to address the following sixobjectives: Objective 1. Cond
Table 1.1. 21st Century Community Learning Centers Cycles 8 10 Grantees, by Grant Years Represented in This Evaluation Report 2 . Table 1.1. 21st Century Community Learning Centers Cycles 8 10 Grantees, by Grant Years Represented in This Evaluation Report 2 . Table 2.1. Texas ACE Centers Key Measures of Program Implementation 9 . Table 2.2.
concept “21st OR twenty-first century skills”, this has, also, been searched through the terms: “21st OR twenty-first century competencies”, “21st OR twenty-first century literacy”, “21st OR twenty-first century learn*”. The search action returned 116 results. The references sections of these texts were
To understand the 21st Century Life and Career Standards To gain knowledge in the Framework for 21st Century Inter-disciplinary themes To develop skills in planning cross-curricular activities related to the 21st Century skills To learn practical strategies in how to infuse 21st Century skills
21st Century Assessment to Promote 21st Century Learning: The Benefits of Blinking Valerie J. Shute, Vanessa P. Dennen, Yoon-Jeon Kim, Oktay Donmez, & Chen-Yen Wang Abstract What competencies do kids need to succeed in the 21st century, and how do these skills differ from those reflected in current state standards?
The 21st Century Teaching Learning, and Leading (CTLL) Technical Working Group (TWG) developed a set of tools that provide principals a means to understand the shift from a 20th century principal to a 21st Century School Principal. The platform of the 21st CTLL is collaboration throughou t the school,
organizations that seek accreditation (Source: NCCAM) Dialysis centers Endoscopy centers Imaging centers Infusion therapy services Laser centers Lithotripsy services MRI centers Plastic surgery centers Podiatric clinics Radiation/oncology clinics Rehabilitation centers Sleep centers Urgent/emergency care centers
with the definition, demand and delivery of 21st Century Skills, with a particular focus on low- and middle-income countries (LMICs). In looking at definitions of ‘21st Century Skills’, Section 2 of this study finds that there is a broad range of available literature discussing 21st Century Skills, including a number of key synthesis studies.
“21st century skills” and “college and career readiness” have recently become watchwords in education. The Partnership for 21st century skills advocates adoption of local, state, and federal policies that support explicit integration of 21st century skills into instruction for all students (P21, 2009).
2nd Grade ELA-Reading Curriculum . Course Description: In 2. nd. grade, readers continue to focus on print with a heavier emphasis on meaning. Students rely on strategies to figure out words, understand author’s craft, and build ideas about the books they read. Students learn from books through informational reading on familiar topics while continuing to build word solving strategies .
