2y ago

15 Views

2 Downloads

672.11 KB

13 Pages

Transcription

The IXL EffectMeasuring the Impact of IXL on the MassachusettsComprehensive Assessment System1IntroductionPrevious research has shown that the use of IXL has a significant impact on studentachievement for an individual school (Empirical Education, 2013). In this study, weexplore IXL usage across the entire state of Massachusetts. Examining such a largesample of schools allows us to quantify the impact of IXL Math and IXL English LanguageArts (ELA) on school performance as measured by Massachusetts state exams.AbstractThis study investigated hundreds of public schools in the state of Massachusetts thatused IXL Math or IXL ELA between 2017 and 2018. Using data from the 2018 NextGeneration Massachusetts Comprehensive Assessment System (Next-Gen MCAS),researchers examined student achievement in both IXL schools and non-IXL schools.Scores from the 2017 Next-Gen MCAS were used as the control for schools’ performanceprior to using IXL. IXL usage by the schools in this study ranged from less than oneminute per student, per week, to nearly 80 minutes per student, per week. Even withthe wide range in student usage, our researchers found a strong positive correlationbetween IXL usage and school performance. These results are statistically significant.Key FindingsMassachusetts schools using IXL outperformed schools without IXL in both math and ELA.

The IXL EffectThe IXL effect was even larger at urban schools and low-performing schools.The Effect of IXL Math(Urban Schools)The Effect of IXL Math(Low-Performing Schools)The Effect of IXL ELA(Urban Schools)The Effect of IXL ELA(Low-Performing Schools)The IXL Effect in Massachusetts SchoolsMAY 2 0 , 2 0 1 9Study DesignOur researchers wanted to determine the effect of IXL on student achievement atthe school level, as measured by the percentage of students in the school meetingproficiency goals set by the state. To do this, we looked at state test results for schoolsbefore and after implementing IXL. We used schools not implementing IXL as a control.This study used a pretest-posttest control group design (see Figure 1) to measure theimpact of IXL. This type of study evaluates the treatment effect by comparing theperformance of the treatment group and the control group on the posttest, afteradjusting for their performance on the pretest. The treatment group included schoolsthat started using IXL in the 2017-18 school year. The control group consisted of schoolsthat did not use IXL in the 2016-17 or 2017-18 school years.2016-17SCHOOL YEARTREATMENT GROUP:IXL SCHOOLSNot using IXLCONTROL GROUP:NON-IXL SCHOOLSNot using IXL2017SPRING2018SPRINGTreatment:Start using IXLPretest:2017Next-GenMCASFigure 1. Study Design22017-18SCHOOL YEARPosttest:2018Next-GenMCAS

The IXL EffectThe Next-Generation Massachusetts Comprehensive Assessment System (Next-Gen MCAS)was used as the pretest and the posttest in this study to determine the performancefor all schools. The Next-Gen MCAS is an updated version of the previous MCAS and wasdesigned to assess students on the Massachusetts learning standards in English languagearts (ELA) and mathematics. Students in grades 3 through 8 have been taking theNext-Gen MCAS tests since 2017. The academic performance of each grade level withineach school is evaluated based on the percentage of students who met or exceededexpectations (referred to as “percent proficient”).MethodologyThe study analyzed data from 1,365 public schools in Massachusetts, including bothtraditional public schools and charter schools. A total of 571 public schools used IXLMath and/or IXL ELA during the 2017-18 school year. As the number of students whoused IXL ranged from a single classroom to the entire school, this study defined a schoolas an “IXL school” at each grade level rather than the school level. A grade level cohortis identified as an IXL school if at least 70 percent of the students enrolled in the gradelevel practiced on IXL (see Appendix A for details on school selection and classification).Based on this criteria, 176 grade level cohorts from 116 schools were identified as IXLschools for IXL Math, and 58 grade level cohorts from 38 schools were identified as IXLschools for IXL ELA. Appendix B shows the characteristics of IXL schools and the stateaverages. The school performance and enrollment data were obtained from the statedepartment of education websites and the Institute of Education Sciences.Our researchers used multilevel linear models to calculate the IXL effect—i.e., theperformance difference between IXL schools and non-IXL schools on the 2018 NextGen MCAS, controlling for factors such as prior performance, school size, percentageof English language learners, percentage of economically disadvantaged students,percentage of students in special education programs, and school location. Similarmultilevel linear models were applied to elementary school levels (i.e., grades 3-5),middle school levels (i.e., grades 6-8), low-performing schools (i.e., schools with2017 Next-Gen MCAS scores below the state average), and urban schools (i.e., schoolslocated in urban areas). Another multilevel linear model was applied to compare theperformance difference between IXL schools with different amounts of IXL usage. (SeeAppendix C for a detailed explanation of analytical methods.)This form of analysis allowed us to answer three key questions:1. What is the IXL effect on student achievement? In other words, did IXL schoolsperform better on the 2018 Next-Gen MCAS tests than non-IXL schools?2. What is the IXL effect for elementary schools, middle schools, low-performingschools, and urban schools?3. What is the association between IXL usage and school performance?3

The IXL EffectResultsThe Efficacyof IXL MathAnalysis of the data showed that the use of IXL had positive and statistically significanteffects on school performance on the 2018 Next-Gen MCAS tests in both math andELA, indicating there is a high probability that similar schools using IXL would achievesimilar results. The IXL effect was even larger for low-performing schools and urbanschools. Our analysis also showed a positive correlation between IXL usage and schoolperformance. In particular, on the 2018 Next-Gen MCAS tests, schools with at leasttwo IXL skills proficient per student, per week, outperformed schools with fewer skillsproficient on IXL.The implementation of IXL Math showed a statistically significant effect on schools’performance on the 2018 Next-Gen MCAS math tests across grades 3 through 8 (seeAppendix D, Table D1 for details).Figure 2 shows that the adjusted percent proficient1 was 47.06 for non-IXL schools and48.66 for IXL schools. The 1.60 percent difference corresponds to a percentile gain of3 points in school ranking. That is, if an average non-IXL school (at the 50th percentile)had begun using IXL Math in the 2017-18 school year, the school’s percent proficientwould be expected to increase 1.60 percent, putting the school at the 53rd percentile.Figure 2. The Effect of IXL Math on the 2018 Next-Gen MCAS41Adjusted percent proficient: the percentage of students who met or exceeded expectations on the 2018 Next-Gen MCAS, after adjustingfor differences in prior performance and school characteristics between IXL schools and non-IXL schools.

The IXL EffectFigure 3 shows the effect of IXL Math at the elementary school level (i.e., grades 3-5)and at the middle school level (i.e., grades 6-8). For elementary schools, the IXL effectis 1.08 points, corresponding to a 2-point percentile gain. For middle schools, the IXLeffect is 1.95 points, corresponding to a 4-point percentile gain.Figure 3. The Effect of IXL Math at the Elementary and Middle School LevelsFigure 4 shows the effect of IXL Math at urban schools and low-performing schools. Forurban schools, the IXL effect is 2.40 points, corresponding to a 5-point percentile gain.For low-performing schools, the IXL effect is 2.16 points, corresponding to a 6-pointpercentile gain.Figure 4. The Effect of IXL Math at Urban Schools and Low-Performing Schools5

The IXL EffectThe Efficacyof IXL ELAThe implementation of IXL ELA showed a positive effect on schools’ performance on the2018 Next-Gen MCAS ELA tests across grades 3 through 8 (see Appendix D, Table D2 fordetails).Figure 5 shows that the adjusted percent proficient was 50.88 for non-IXL schools and54.06 for IXL schools. The 3.18 percent difference corresponds to a percentile gain of6 points in school ranking. That is, if an average non-IXL school (at the 50th percentile)had begun using IXL ELA in the 2017-18 school year, the school’s percent proficientwould be expected to increase 3.18 percent, putting the school at the 56th percentile.Figure 5. The Effect of IXL ELA on the 2018 Next-Gen MCASFigure 6 shows the effect of IXL ELA at the elementary school level (i.e., grades 3-5)and at the middle school level (i.e., grades 6-8). For elementary schools, the IXL effectis 4.23 points, corresponding to a 9-point percentile gain. For middle schools, the IXLeffect is 1.75 points, corresponding to a 3-point percentile gain.Figure 6. The Effect of IXL ELA at the Elementary and Middle School Levels6

The IXL EffectFigure 7 shows the effect of IXL ELA at urban schools and low-performing schools. Forurban schools, the IXL effect is 3.80 points, corresponding to a 8-point percentile gain.For low-performing schools, the IXL effect is 3.92 points, corresponding to a 10-pointpercentile gain.Figure 7. The Effect of IXL ELA at Urban Schools and Low-Performing SchoolsThe UsageEffect of IXLFor schools that used IXL Math in the 2017-18 school year, our analyses found a positiveand statistically significant association between IXL Math usage and schools’ performanceon the 2018 Next-Gen MCAS math tests (see Appendix D, Table D3 for details).Figure 8 shows the adjusted percent proficient for schools with different numbers of skillsproficient2 on IXL. More skills proficient is associated with a greater IXL effect. Schoolswith at least two IXL math skills proficient per student, per week, had 5.59 percent morestudents meeting and exceeding expectations on the 2018 Next-Gen MCAS math teststhan schools with less than one skill proficient per student, per week.Figure 8. The Usage Effect of IXL MathThe number of schools with at least one IXL ELA skill proficient per student, per week,was not large enough to conduct a usage effect analysis.72Skill proficiency on IXL is measured by IXL’s proprietary SmartScore. The SmartScore starts at 0, increases as students answer questionscorrectly, and decreases if questions are answered incorrectly. A student is considered proficient in a skill when they reach a SmartScore of 80.

The IXL EffectConclusionIn sum, the study results indicated that schools using IXL outperformed schools withoutusing IXL. It is concluded that IXL is an effective program for schools seeking to increasestudent achievement in both math and ELA. It was also found that the IXL effectwas even larger at urban schools and low-performing schools. These findings can begeneralized to other public schools, especially for those who are using the MCAS orsimilar assessments (e.g., the new Rhode Island Comprehensive Assessment System)—they are likely to achieve similar results by using IXL.ReferencesEmpirical Education (2013). A Study of Student Achievement, Teacher Perceptions, andIXL Math. Retrieved from 13.pdfWhat Works Clearinghouse (2014). What Works Clearinghouse procedures and standardshandbook (Version 3.0). Retrieved from wwc procedures v3 0 standards handbook.pdfAppendix A:IXL SchoolIdentificationThis study determined whether a school is an IXL school based only on the number ofstudents using IXL. Because a school may choose to use IXL in only a few classroomsor across the entire school, this study defined IXL schools at each testing grade level3rather than the school level. The group of students at the same grade level within thesame school is referred to as a grade level cohort.A school is identified as an IXL school for a certain grade level in a certain school year if:1) the school has an active IXL account within the school year, and 2) at least 70 percentof the enrolled students at this grade level have practiced on IXL within the school year.A school is identified as a non-IXL school for a certain grade level in a certain schoolyear if less than 70 percent of the students at this grade level have practiced on IXLwithin this school year.For example, suppose a K-6 school had an active IXL account within the 2016-17 schoolyear, and over 70 percent of students in grades K-4 had practiced on IXL. Less than70 percent of students in grades 5 and 6 practiced on IXL during that year. This schoolwould be defined as an IXL school for the 3rd and 4th grade level cohorts and as a nonIXL school for the 5th and 6th grade level cohorts. Students in grades K-2 are excludedfrom the analysis because they do not take the state standardized tests.83Testing grade level: a grade level in which students are required to take the Next-Gen MCAS tests.

The IXL EffectAppendix B:Schools’BackgroundInformationTable B1 shows the background information for all public schools in Massachusetts andfor IXL schools. IXL schools performed slightly better than the state average on theNext-Gen MCAS tests in 2017 and 2018. IXL schools had more schools located in citiescompared to the state average.Table B1. Background Information for Massachusetts state and IXL schoolsIXL schoolsStateaverageIXLMathELANumber of schools1,36511638Number of grade level cohorts4,2811765849%50%-49%51%-2017 Next-Gen MCAS ELA percent proficient50%-55%2018 Next-Gen MCAS ELA percent proficient50%-57%% of economically disadvantaged students32%30%30%% of English language learners10%10%9%% of students in special education programs17%16%16%% of schools in cities18%19%29%% of schools in suburbs68%66%48%% of schools in towns2%3%6%% of schools in rural areas11%11%16%2017 Next-Gen MCAS math percentproficient2018 Next-Gen MCAS math percentproficient9IXL

The IXL EffectAppendix C:AnalyticalMethodsA three-level linear model was used to calculate the IXL effect on Next-Gen MCASperformance (i.e., the performance difference between IXL schools and non-IXL schoolson the 2018 Next-Gen MCAS), after adjusting for schools’ prior academic performance(i.e., 2017 Next-Gen MCAS percent proficient), cohort size (i.e., the number of enrolledstudents in the grade level cohort), school size (i.e., the number of enrolled students inthe school), percentage of economically disadvantaged students, percentage of Englishlanguage learners, percentage of students in special education programs, and schoollocation (i.e., city, suburb, town, or rural as defined by the Institute of EducationSciences). The units of analysis of the three-level model are grade level cohorts (i.e.,level 1). Grade level cohorts are nested within school districts (i.e., level 2), whichare further nested within states (i.e., level 3). Similar multilevel linear models wereapplied to the urban grade level cohorts only (i.e., cohorts within schools located inurban areas) and low-performing grade level cohorts only (i.e., cohorts with the 2017Next-Gen MCAS percent proficient below the state average) to calculate the IXL effecton these two types of schools separately. To assist in the interpretation of the IXLeffect, we reported statistical significance, effect size, and percentile gain. Statisticalsignificance, also referred to as p-value, is the probability that the IXL effect is zero.A small p-value (e.g., less than 0.05) indicates strong evidence that the IXL effect isnot zero. Effect size is the mean difference in standard deviation units and is known asHedges’ g. In this study, effect size is computed using adjusted mean and unadjustedstandard deviations. Percentile gain is the expected change in percentile rank foran average non-IXL school if the school had used IXL. It is calculated based on theeffect size. More details about these analytical methods can be found in What WorksClearinghouse (2014).We applied another three-level linear model to compare the performance differencebetween IXL schools with different amounts of IXL usage. We set benchmarks for low,medium, and high IXL usage based on the number of skills proficient (SmartScore 80)per student per week. The model was very similar to the first model described in thisappendix, but the model included the IXL usage group as an independent variable andthe sample only included schools that used IXL during the 2017-18 school year.10

The IXL EffectAppendix D:Data TablesTable D1. The Effect of IXL Math on the 2018 Next-Gen MCAS Math TestsOverall(allschoolsacrossgrades3-8)ES level(grades3-5)MS 692843575Number of grade levelcohorts at non-IXL schools3,5202,3571,1637771,801The IXL effect1.60*1.081.952.402.17Effect size0.080.060.090.120.14Percentile gain3.172.193.684.655.63Adjusted 2018 NextGen MCAS math percentproficient for IXL schools48.66%48.89%47.44%37.54%35.67%Adjusted 2018 NextGen MCAS math percentproficient for Number of grade levelcohorts at IXL schoolsNote: 1) *: significant at .05 level2) ES: elementary school; MS: middle school11

The IXL EffectTable D2. The Effect of IXL ELA on the 2018 Next-Gen MCAS ELA TestsOverall(allschoolsacrossgrades3-8)ES level(grades3-5)MS 31271724Number of grade levelcohorts at non-IXL schools4,0062,6081,3988801,946The IXL effect3.18*4.231.753.803.92Effect size0.160.230.090.190.26Percentile gain6.498.953.457.6710.13Adjusted 2018 NextGen MCAS ELA percentproficient for IXL schools54.06%57.36%49.10%41.59%41.78%Adjusted 2018 Next-GenMCAS ELA percent proficientfor non-IXL schools50.88%53.13%47.35%37.79%37.86%ValuesNumber of grade levelcohorts at IXL schoolsNote: 1) ***: significant at .001 level; *: significant at .05 level2) ES: elementary school; MS: middle school; HS: high school12

The IXL EffectTable D3. The Usage Effect of IXL MathValues 1 skillproficientper student perweek1-2 skillsproficientper student perweek 2 skillsproficientper student perweekNumber of grade level cohorts used IXLin the 2017-18 school year331102311.095.58**0.060.2951.72%56.21%IXL usage effectN/AEffect sizeAdjusted 2018 Next-Gen MCAS mathpercent proficientNote: **: significant at .01 level1350.63%

Results Analysis of the data showed that the use of IXL had positive and statistically significant effects on school performance on the 2018 Next-Gen MCAS tests in both math and ELA, indicating there is a high probability that similar schools using IXL would achieve similar results. The IXL effect was even larger for low-performing schools and .

Related Documents: