LITERACY DESIGN COLLABORATIVE 2018–2019

Across all 3 years of the study, teacher, teacher leader, and administrator participantsreported very positive attitudes toward LDC. Teachers appreciated the opportunity tocollaborate and share practice, generally reported that the range of in-person anddigital supports was helpful, and agreed that participating in LDC positively impactedtheir practice, in areas such as engaging students in complex text, locating evidence ofstandards in final student work, and engaging students in understanding theperformance task assignment and standards-rubric. Administrators generally saw LDCas a productive tool for meeting school instructional goals, but had some concernswith the intensive weekly time commitment needed to implement the program, withsome administrators choosing to discontinue participation or dedicate less time to theprogram in favor of other priorities.Professional Learning Community and Teacher Collaboration Nearly all LDC teachers participated in LDC-oriented PLCs. The frequency with whichPLCs met in 2018 2019, however, varied greatly across schools, with the number ofrecorded PLC meetings ranging between four and 20 times in the year, and averaging12.6 times. The average teacher’s individual attendance rate was 78%, but again therewas great variation. About two thirds of teachers met the teacher-level fidelitythreshold of 80% attendance. About half of schools met the program goal of threequarters of PLC participants attending 80% of sessions, with some schoolsexperiencing challenges related to protecting planning time and ensuring that teacherparticipants attended PLC meetings. Both teacher surveys and PLC reflection formsindicated that PLC meetings typically lasted 45 to 59 minutes or an hour or more, andtherefore met the LDC expectation. Teachers valued the collaborative nature of LDC and its PLCs. A large majority ofteachers credited LDC with making them more likely to collaborate with otherteachers, not only within their grade levels and content areas but outside of them aswell.LDC Training and Support Teachers were nearly uniform in their positive attitudes about the value of their PLCparticipation. They found the PLCs a safe space for sharing instructional plans,problem solving, and learning to develop modules. Teacher leaders were almost universally reported to be highly approachable,supportive, knowledgeable, and helpful. Teacher leaders reported high satisfaction regarding the support they received fromcoaches, professional development offerings, and how the teacher leader role allowedthem to be instructional leaders in their schools. Overall, LDC coaches received positive feedback on the survey, with 96% of teachersand 97% of teacher leaders reporting that their coaches gave them appropriate andtimely feedback and support. Data suggest room for improvement when it comes tofrequency and usefulness of coach feedback. While coaches were somewhat morevii

likely to meet fidelity thresholds on module comments in 2018 2019 than 2017 2018,peer review was used even less frequently in 2018 2019 than it was used in2017 2018. Most teachers rated CoreTools positively, which demonstrated the success of changesLDC made prior to the 2017 2018 school year including refinement of the content,sequencing, delivery of CoreTools’ instructional content, and streamlining ofparticipants’ learning process. Overall, teachers were enthusiastic about much of theonline learning content, with more than 4 out of 5 teachers rating most aspects asgood or excellent. Almost a third of teachers, however, were concerned with the easeof use of the online course materials. Despite teachers’ positive attitudes, analysis ofCoreTools data revealed that teachers were being exposed to a less online coursecontent than program goals intended. There was some improvement in this area from2017 2018 to 2018 2019, with a small number of schools meeting fidelity ofimplementation on exposure to the instructional content, but still exposure rates wereconsiderably lower than fidelity thresholds. It is possible that there wasn’t sufficientPLC time to cover the content with which LDC intended participants to engage.LDC Implementation Teachers typically reported adapting/creating and implementing two LDC modulesacross the school year, which meets LDC program expectations. Analysis of program data suggests that while nearly all participants were engagingwith the module-building platform to some extent, the level of engagement variedgreatly across individuals and across subgroups (role, cohort, school level, contentarea) as evidenced by the wide range in the number of views, edits, and commentsacross teachers. More in-depth analysis of the portions of the modules that teachersedited indicated that engagement varied greatly across teachers’ user accounts, withabout half of user accounts associated with engagement at a basic level (editing theteaching task), and other accounts associated with deeper engagement (editingmultiple portions of modules). It is important to note that in response to teacherscollaborating on and implementing common modules in schools, LDC refined its datacollection in the summer of 2018 to include tracking engagement data on modulesthat teachers collaborated on. The fidelity matrix and CRESST’s analyses of editingdata were not designed to capture this shift toward a collaborative model ofinstructional design in CoreTools, and therefore may not fully capture engagement inthe design process. The majority of teachers (79% to 93%) reported success in nine key areas of LDCmodule development. Teachers were most confident in selecting focus standards,creating the writing assignment, identifying skills needed in the module, and makingwriting assignments relevant and engaging. The module analysis, however, suggeststhat the materials adapted and created by PLC members varied in levels of completionand quality.viii

With regard to their classroom implementation of LDC modules, the majority ofteachers reported success with all six key areas queried (86% to 92%). Teachers weremost confident with engaging students in complex text, locating evidence of standardsin final student work, and engaging students in understanding the assignment andrubric.Leadership Support Almost all teachers and teacher leaders reported that their administrators encouragedLDC participation at the school. The majority of teachers and teacher leaders agreedwith administrators that they allocated resources to ensure that LDC teachers couldparticipate in meetings. Administrators generally voiced strong support for LDC, butvaried in their level of direct engagement with the work, with some taking a hands-onapproach by attending many PLC meetings, and some not. Overall, most administrators and teacher leaders took advantage of in-personmeetings offered by LDC. There was less consistency in terms of the frequency ofteacher leader/coach planning calls, with about 40% of schools meeting the fidelitygoal, and teacher leaders and coaches meeting over the phone less frequently in theremainder of the schools.Impact on Teacher Practice The majority of teachers reported improving their practice in seven LDC-related skills(79% to 88%). Teachers were a bit more likely to report impact on selecting focusstandards for an assignment, creating standards-driven writing assignments, andidentifying skills that students need in writing assignments (skills concentrated at thebeginning of the LDC learning cycle). Over 80% of teachers agreed that participating in LDC raised their expectations forstudents’ writing, helped them incorporate writing assignments into their existingcurriculum, and made them more likely to collaborate with other teachers ondesigning instruction. Among teachers who completed the survey in both 2017 2018 and 2018 2019,attitudes around impact on teacher skills and practices were on average even morepositive in 2018 2019 than 2017 2018. Overall, CRESST’s ratings of module quality decreased from 2017 2018 to 2018 2019,when looking at all rated modules and those modules created by a subpopulation ofteachers who were present in both years. This change contrasts with an increase inthe quality of modules scored by CRESST from 2016 2017 to 2017 2018. Please note,however, that sample sizes are small and thus comparisons across years areexploratory in nature.Sustainability Survey respondents generally reported confidence that the LDC program would besustained in their schools. Analysis of attrition patterns, however, reveals concernsix

about sustaining the LDC initiative in schools. Even with the substantial supportsprovided by LDC, a number of administrators decided over the course of the 3 yearsthat they did not have the resources (particularly staff time) to remain in the program.Competition for time between different reform efforts and school priorities seemed toplay a role in whether schools remained in the LDC program. Based on the exit interviews conducted by LDC staff, school leadership teamsindicated that LDC practice was likely to continue at some level in the majority ofschools. Some principals and assistant principals were committed to protectingcommon planning time and expanding the use of the LDC planning process to othergrades and subjects. A handful of schools demonstrated strong commitment to LDCpractice by purchasing LDC product licenses. In other cases, administrators wereunlikely to provide support into the future, and therefore practice was less likely to beconsistent and to spread across the school.Impact on Student Learning Quasi-experimental analyses revealed a statistically significant positive impact ofmiddle school teachers with 2 years of LDC experience on student ELA scores. Thedosage-dependent version of our model also suggests that students who wereexposed to LDC instruction in a greater number of core content classes benefitedmore from the program. Supplemental analysis also indicates that impact was greatestfor students who were exposed to LDC in half or more of their class time in coresubjects (ELA, social studies, and science). The study does not provide evidence of an impact of LDC on student ELA scores at theelementary school level after teachers participated for 2 consecutive years. It isimportant to note that it was only possible to test the impact of LDC in upperelementary grades, so the finding is not generalizable to the full population ofelementary teachers implementing LDC and their students. Also, an analysis of Cohort1 middle school teachers in their third year of LDC implementation did not detect animpact, with limited sample size due to attrition likely a factor. Survey respondents were nearly uniform in perceiving a range of positive impacts ofLDC on student learning. Among teachers who completed the survey in both2017 2018 and 2018 2019, attitudes about LDC impact on students were on averageeven higher in 2018 2019 than in 2017 2018.ConclusionsUCLA CRESST’s multiyear mixed methods evaluation of LDC as implemented in a largeurban West Coast school district provides evidence that LDC is an effective tool for increasingstudent learning in English language arts. LDC’s theory of action predicted that teachers wouldneed to participate in the program for at least 2 years to effectively deliver LDC-infusedinstruction, and positively impact student learning, as measured by student assessment scores.Our quasi-experimental design analyses provide confirmation for that hypothesis, particularlyat the middle school level, with students under teachers with 2 consecutive years of LDCx

participation performing at significantly higher levels than the comparison group of matchedstudents in matched schools in the same district. That effect translates to 9.4 months ofadditional learning for students exposed to LDC in all their core content area, and 4.1 months ofadditional learning for the average observed student in the study. Our dosage-dependentmodel also suggests that middle school students with higher levels of exposure to LDC (by beingexposed to multiple teachers in different content areas implementing LDC) benefited morefrom the program. A positive statistically significant impact was also found for Cohort 2 middleschool students after one year, which provides further evidence of the efficacy of LDC at themiddle school level.A similar effect was not found at the elementary school level with the present data,although attrition and the inability to include lower grade elementary teachers and students inthe analyses may have played a role. As noted above, our study also suggests that more LDC isbetter; since most elementary students were exposed to only one teacher implementing LDC,the program expectation of each teacher implementing two modules per year may not have ledto elementary students receiving a sufficient dosage of LDC to affect their learning. Furtherinvestigation of LDC’s impact at the elementary level would be beneficial.Implementation data demonstrated great appreciation for LDC among teachers, teacherleaders, and administrators. Educators found LDC to be a helpful tool for fosteringcollaboration, creating a safe space for sharing practice, and increasing teacher skills andknowledge around literacy instructional design and teaching. Despite broad support, however,many schools and teachers did not meet fidelity thresholds related to attendance, exposure toinstructional content, and principal observation of LDC instruction. The findings demonstratethe importance of school leaders and teachers understanding and committing to the programin advance of implementation, and school leaders dedicating substantial resources to theprogram, particularly time for common planning. Under these ideal conditions, LDC may showeven greater impacts on teacher practice and student learning.In summary, our multiyear mixed methods evaluation of the LDC as implemented in alarge urban West Coast school district provides evidence that students exposed to LDCinstruction made significant gains in learning, particularly at the middle school level.Furthermore, teachers were nearly uniform in their positive attitudes about the value of theirPLC participation. They valued the collaborative nature of LDC and its PLCs and reportedimproving their practice in LDC-related skills and improved student learning. Despite thechallenges in implementing LDC as intended, the stude

The Literacy Design Collaborative (LDC) was created in 2009 to support teachers in implementing Common Core State Standards (CCSS) and embedding literacy skill development throughout content area curriculum. Engaged in the evaluation of LDC tools since June 2011,