Redefining The Measurement Of Early Childhood Program .

However, debates continue about the concrete componentsof high quality and the precise combination of elementsthat result in consistent, significant, and lasting positiveimpacts for children. In fact, research suggests that currentdefinitions and measures of process quality are too narrowlydefined (Burchinal, 2018).The ChallengeThere is little data available through studies on whatconstitutes high quality programs beyond QRIS validationresearch, and QRIS programs across different states oftendo not share multiple common measures of quality (Toutet al., 2017; Wechsler et al., 2018). In fact, early childhoodprograms are negatively affected by the multiple layers ofattempts to define program quality (National Early ChildhoodAccountability Task Force, 2013; Wechsler et al., 2018). Forexample, state licensing requirements include high qualityindicators in some states, and focus solely on basic health andsafety in other states. Across states, licensing requirementsmay significantly differ from the quality indicators identifiedby QRIS programs, and accreditation standards fromnational accreditors sometimes further conflict with bothlicensing and QRIS. Early childhood programs are caught inthe middle, trying to navigate the various requirements thatare supposed to help them achieve and maintain high quality.States are challenged to balance access to early childhoodeducation for all children while simultaneously ensuringquality across programs.How do we define programquality?The National Institute for Early Education Research(NIEER) has defined common quality measures for staterequirements of publicly-funded early childhood programsthrough its quality standards benchmarks (Friedman-Krausset al, 2019). These benchmarks include: Statewide early learning and development standards Strong curriculum with supports Teacher education and specialized training Ongoing professional development Maximum class sizes and ratios Child screening and referral A continuous improvement systemSimilarly, Minervino’s 15 essential elements framework(Weisenfeld, Frede, & Barnett, 2018) provides guidanceto states on how to implement high quality preschoolprograms through an enabling environment of political willand support, rigorous and articulated early learning policies,and strong program practices. The early learning policiesaddress the need for well-educated and compensatedteachers; maximum class size and ratios; full school dayimplementation; early learning standards; effective andsupported curriculum; and the inclusion of children withspecial needs and dual language learners. Strong programpractices include: High quality teaching Ongoing professional development Child assessments Data-driven decision making Integrated systems of standards, curriculum, assessment,professional development and evaluation.These two models for program quality elements align wellwith the body of commonly accepted research on highquality programs (Meloy et al., 2019; Wechsler et al, 2018).The challenge remains to identify measures that reliablyand consistently identify high quality programs across thepreviously-discussed elements, as well as the componentson which early childhood programs can focus improvementefforts to reach higher levels of quality (Leal, Gamelas,Barros, & Pessanha, 2018). Recent studies examining theimpact of NIEER benchmarks or the 15 essential elementshave found that progress across states has been uneven,and many states’ publicly-funded programs are still notmeeting these benchmarks of quality (Wechsler et al., 2018).Many studies have focused exclusively on publicly-fundedprograms, which leave out a significant population of childrenin early education settings (Phillips et al, 2017). This is due tochallenges in accessing information from non-publicly fundedprograms, funding for validation studies and research, andRedefining the Measurement of Early Childhood Program Quality and Child Outcomes4

the fragmentation of the early childhood system (NationalEarly Childhood Accountability Task Force, 2013).In addition, studies that correlate quality measureswith children’s development and learning outcomeshave demonstrated limited and mixed results (Barnett,2011; Burchinal, 2017; Mashburn et al., 2008; Weisenfeldet al., 2018). Traditionally, program evaluation has focused onmetrics of program quality, practices, and inputs rather thanon child outcomes (National Early Childhood AccountabilityTask Force, 2013). The most common measures of childoutcomes focus on foundational academic skills, executivefunction, and physical development at the current pointin time (Elango, Garcia, Heckman, & Hojman, 2015). Onlya few studies (Barnett et al., 2013; Campbell & Pungello,2006; Ou et al., 2019; Schweinhart et al., 2005) have beenable to capture longitudinal data on child outcomes andthese rarely examine the correlation between participationin early childhood programs of varying quality or includecontrol groups (Cannon et al., 2017). Longitudinal outcomeshave often reviewed measures of readiness for Kindergarten(Camilli, Vargas, Ryan, & Barnett, 2010), retained academicadvantage in 3rd grade (Burchinal, 2018; McCoy et al., 2017),and rates of high school completion (McCoy et al, 2017; Phillipset al., 2017; Schweinhart et al., 2005). These longitudinaloutcomes studies documented impressive improvementsin learning while children were attending early learningprograms. They also showcase the long-term impacts onyoung adult outcomes such as increased graduation rate,lower grade retention, lower involvement with the justicesystem, decreased teen pregnancies. Adult outcomes arealso reviewed, such as better health, higher earnings, stablehousing and more (Elango et al., 2015; Phillips et al., 2017;Rolnick & Grunewald, 2003).Recent research efforts have focused on defining quality andthen trying to demonstrate a correlation with child outcomes(Burchinal, 2018; Cannon et al., 2017). Unfortunately, theseassociations have been inconsistent and modest whencorrelating current measures of quality with children’soutcomes. None of NIEER’s benchmarks have been foundto significantly relate to child outcomes (Friedman-Krausset al., 2019; Mashburn et al., 2008; Phillips et al., 2017).Multiple studies have found a lack of significant associationswith child outcomes for generally accepted qualityelements such as overall classroom quality as measuredby environmental rating scales (Burchinal, 2018; Mashburnet al., 2008), teacher degree and specialized training inearly childhood education (Burchinal, 2018; Mashburnet al., 2008), and ratios and group sizes (Burchinal, 2018). Inaddition, there is a dearth of research on sustained positiveacademic effects, and the evidence of “fade out” of theacademic gains children who experience early educationhave upon Kindergarten entry should be further explored(Barnett, 2013; Cannon et al., 2017; Elango et al., 2015; Phillipset al., 2017; Tout et al., 2017; Weisenfeld et al., 2018).Most current publicly-funded programs are too new tomeasure long-term outcomes for children that have beendocumented by the early demonstration programs of the1960s and 1970s. There is also the question of how much wecan draw from an earlier generation of programs to guidetoday’s programs (Phillips et al., 2017), as “we do not knowwhat it was precisely about these programs that producedpositive outcomes nearly 20 years later” (p. 19). Severalresearchers have discussed the problem of the “blackbox” of the early childhood classroom (Burchinal, 2018;Camilli et al., 2010; Cannon et al., 2017; Meloy et al., 2019;Phillips et al., 2017) and the resulting difficulty to isolatethe contribution of a given element of quality to the overalleffects of a program on child outcomes (Barnett, 2011; Camilliet al., 2010; Cannon et al., 2017).Gathering meaningful data that can be analyzed for decisionmaking presents multiple challenges that further cloudthe process. Studies that examine quality often applyinconsistent documentation of program elements (Elangoet al., 2015; Tout et al., 2017) which increases the difficultyof comparing results from one program to another. Fundingand regulatory requirements may require different programmeasures than those commonly used to measure programquality, or they may require so much data collection thatthe process becomes burdensome to programs. QualityRating and Improvement Systems (QRIS) frequently takea compliance-oriented checklist approach to measuringquality, examining the presence or absence of individualquality elements without looking at the whole picture of aprogram’s quality (Tout et al., 2017). Some QRIS use a singlemeasure of quality, such as the Classroom Assessment andScoring System (CLASS), while others combine multiplemeasures including policy review, observations, programself-assessments and stakeholder feedback surveys (Toutet al., 2017). The most commonly used observational tools—CLASS and the Environmental Rating Scales (ERS)—have overall been found to lack significant associationswith child outcomes (Mashburn et al., 2008). Programsfrequently have limited capacity to collect and analyzedata related to program performance, child outcomes, andongoing continuous improvement (LiBetti & Mead, 2019;Tout et al., 2017). Even the federal Head Start program,Redefining the Measurement of Early Childhood Program Quality and Child Outcomes5

which spends a great deal of time and money collectingdocumentation of program effectiveness, has found that“much of the information collected is not used in ways thathelp individual grantees or the field as a whole to improveperformance” (LiBetti & Mead, 2019, p. 32).Where Quality ImpactsChild OutcomesResearch has documented some key elements of programpractice that do significantly correlate with child outcomes.Intentional teaching that focused on scaffolding higher orderskills in individualized one-on-one interactions and smallgroups predicted later academic success more successfullythan direct instruction focused on rote learning (Barnett,2011; Barnett, 2013; Burchinal, 2018; Camilli et al., 2010).Evidence-based curricula, supported by aligned training andongoing support for teachers, was found to have a small butsignificant effect on children’s literacy skills (Burchinal, 2018).Curricula that targets specific skills, rather than a moreglobal curricula, has been found to more strongly impactchildren’s cognitive and social-emotional gains (Burchinal,2018; Yoshikawa et al., 2013). However, the curricula must beused in a developmentally appropriate approach for thesegains to persist over time (Yoshikawa et al., 2013).Children’s experience of early childhood educationthrough engaging activities and environments was alsolinked to short-term cognitive gains and long-term socialgains (Barnett, 2011; Yoshikawa et al., 2013). Experiencesbenefit children the most when they are rich in contentand stimulation while being emotionally supportive(Mashburn et al., 2008; Phillips et al., 2017). Higher qualityteacher-child instructional interactions had the mostconsistent and strongest associations with developmentof cognitive and language skills, while strong emotionalsupport from teachers positively influenced children’s socialcompetence and reduced challenging behaviors (Mashburnet al., 2008). Elango et al. (2015) found that short-termeffects on noncognitive skills, such as social-emotionaldevelopment and approaches to learning, are importantdeterminants of desirable long-term outcomes such asself-regulation, perseverance, level of education attained,stable employment, better health outcomes, and reducedcrime. Motivation, perseverance, and tenacity were foundto be better predictors of children’s long-term success thanstandardized test scores (Meloy et al., 2019).The challenge continues to be effectively measuring bothchild outcomes in the early years and characteristicsrelated to process and interactions, which have historicallybeen difficult to measure and regulate (Phillips et al., 2017).Our current measures of quality do not assess the contentof what is taught or dimensions of quality such as theeffectiveness of curricula and implementation, scaffoldedlearning, differentiated instruction or engagement of childrenand families (Burchinal, 2018). The relatively nascent workin measuring “soft skills” (Lefkowitz, 2018, p. 5), socialemotional learning, and children’s approaches to learningpresents additional challenges as there are few valid andreliable measures for these outcomes (Meloy et al., 2019).“There are too few assessment tools that capture the fullrange of children’s abilities” (National Early ChildhoodAccountability Task Force, 2013, p. 26), and using a suite ofassessments that do capture a holistic picture of a child tendto be costly and time-intensive in their implementation.Ground quality improvement workin changes that have the potentialto promote meaningful gains inchildren’s skills and competenciesacross developmental domains.”—Tout, Soli, Epstein, & Lowe, 2015, p. 18Redefining the Measurement of Early Childhood Program Quality and Child Outcomes6

The SolutionResearch to this point has focused on measuring agreedupon indicators of quality and then looking at the impacts onchildren’s development and outcomes. Instead, this paperproposes that research needs to examine programs wheredesired child outcomes are consistently strong and examinethe practices in those programs to identify correlatedindicators of quality. This reverse approach may assistthe early childhood field to identify critical componentsof quality that have not yet been considered or have beenonly minimally studied. Then the field can focus on helpingprograms improve those components rather than continuingto measure and rate elements of quality that have previouslybeen identified, but don’t strongly correlate with childoutcomes. Developing a shared understanding of desiredchild outcomes that are associated with children’s long-termacademic, career, and life success, along with subsequentidentification of a common set of indicators observed in highquality programs that correlate to these outcomes, wouldenable early childhood programs to become more consistentto accurately assess meaningful elements of quality and toidentify specific actions for continuous improvement.Valid and reliable measurement tools aligned to theseoutcomes and indicators (LiBetti & Mead, 2019; Phillipset al., 2017) would support internal and external stakeholdersto examine quality and assess outcomes. While existingpolicy papers (Friese, Lin, Forry, & Tout, 2017; National EarlyChildhood Accountability Task Force, 2013; Weisenfeldet al., 2018) have recommended a unified set of standardsand assessments for early childhood program quality andchild outcomes, a major barrier continues to be the vastdifferences in state definitions of quality as well as the layersof differences within state licensing, QRIS, and nationalaccreditation requirements (Wechsler et al., 2018).The changing context of our environment—from advancesin technology to the evolving needs of a future workforce—exacerbate the need for different skills sets and differentiatedmeans of learning in early childhood and throughout theeducation system. The current system is not yet responsiveenough to these changing needs. Given that employersconsistently state that “soft skills” such as collaboration,openness to feedback, critical thinking, and social IQ are asimportant as academic excellence (Lefkowitz, 2018), are wemeasuring child outcomes, and the quality indicators leadingto them, that reflect this emerging focus in the later years ofschooling and workforce? Social-emotional foundations and21st Century skills should be included along with academicperformance to ensure that high quality early childhoodprograms are impacting children’s long-term preparationfor and success in school and life (Elango et al., 2015; Phillipset al., 2017; Tout et al., 2017). In fact, “many of the long-termoutcomes that preschool programs hope to impact mayactually be derived from social-emotional or behavioralskills, such as motivation, perseverance, and tenacity”(Meloy et al., 2019, p. 25). Evidence-based measurementtools are still in development in these newer fields of study,such as those from the assessment workgroup effortsof the Collaborative for Academic, Social and EmotionalLearning (CASEL) and Oregon’s development of a measureof children’s engagement in the early learning setting (Toutet al., 2017).In addition, investments should be made in furtherlongitudinal studies to track child outcomes throughout theireducation career and into adulthood (Camilli et al., 2010;Cannon et al., 2017). Studies must focus on current qualityindicators to examine whether they are in fact making along-term positive impact on children. These studies needto expand beyond the “low hanging fruit” of publicly-fundedprograms that serve 3- and 4-year-olds to include privatecenter-based and family child care programs that serve: Infants and toddlers Children with special needs Children who speak languages other than English (Toutet al., 2017).Ideally, such studies would include control groups toovercome the limitations of quasi-experimental comparisonsamong programs that provide early childhood education(Burchinal, 2018; Camilli et al., 2010; Cannon et al., 2017).Child outcomes should include measures of readiness forfuture academic success, coupled with social-emotionalfoundations and 21st Century skills. The outcomes shouldRedefining the Measurement of Early Childhood Program Quality and Child Outcomes7

look at children’s learning, growth, and developmentwhile participating in early childhood programs as wellas longitudinal outcomes such as learner engagement,academic achievement, persistence, critical thinking,and education completion. Meloy et al. (2019) stated thatresearchers should design measures to “capture likelyprecursors of school progress and longer term success,which may include children’s self-competency, learningorientation, and scholastic motivation” (p. 25). This alignswith current trends in later school grades to shift towardcompetency-based learning and the development of critical“soft skills.” Establishing a common set of desired outcomesand the correlated indicators of quality should result fromblending academic, theoretical research with practitionerexpertise and engagementRecommendationsAnalysis of the current body of research leads to thesespecific recommendations for a research and policyapproach.Define a common set of desired child outcomesincluding both short-term success for schoolreadiness with sustained academic and socialgains; and long-term success in career and life.Robust set of child outcomes should include, at a minimum,cognitive and academic skills; social-emotional skills suchas self-awareness, self-management, social awareness,relational skills and decision-making; 21st Century “softskills” such as critical thinking, creativity, collaboration,communication, flexibility, and information and technologyliteracy; and approaches to learning such as initiative,perseverance, attention, and problem-solving. Long-termoutcomes should further include indicators of academic,health, and economic well-being, such as graduation rates,post-secondary educati

Redefining the Measurement of Early Childhood Program Quality and Child Outcomes 5 the fragmentation of the early childhood system (National Early Childhood Accountability Task Force, 2013). In addition, studies that correlate quality measures with children’s development and learning outco