A Games Part II Case Studies - A-Games The A-Games Project

A REPORT OF THE A-GAMES PROJECT: ANALYZING GAMES FOR ASSESSMENT IN MATH, ELA/SOCIAL STUDIES, AND SCIENCEannotation tools, and sometimes dashboards.Whether provided by game developers, oraccessed via portals or game aggregators, thesefeatures can be valuable for supporting teachers’use of games for formative assessment in atleast three ways:»providing support for teaching with games;»providing materials teachers can use forassessment around games; and»providing continuity in assessment experiencesacross games.ABOUT THESE CASESTUDIESIn order for any particular formative assessmentapproach to be valuable, it must be both useful andused. The goal of the A-GAMES study is to explore howteachers employ various features of games to supportformative assessment of student learning.The educational games observed in these case studiesall contain features that are intended to support teachersin monitoring and understanding student learning andprogress. Sometimes these features are simple, suchas an end-of-game score. Sometimes they are complex,such as a dashboard to track student progress towardsacademic standards across a range of in-game metrics.We examined how teachers interpretand use these features, and whatissues and challenges arise that limitA unique aspect of our casestudies is that they focuson the features within thegames—not on individualteachers or games.their use in hopes of providing valuable information forthe future development of games that support learning.A line of inquiry of special interest to this project isconcerned with the design of assessment mechanics(Plass et al., 2011; Plass et al., 2012), i.e., patterns ofspecialized activities that have assessment as the primaryobjective, to enhance the abilities of games to provideinsights on student progress toward learning goals.In early 2014, we conducted case studies with 30volunteer teachers in the New York City area. All taughtin grades 5-8, in public and private schools. We selectedteachers across the subject areas of English/languagearts (ELA), social studies/history, math, and science.The teachers were varied in terms of their experienceswith games, both personally and as educational tools(See Appendix A for more information about case studydemographics and methodology). Each teacher agreedto use one of 11 educational video games accessed viaBrainPOP’s GameUp portal.4 We selected GameUpbecause it provides a single point-of-access for gamesby a variety of leading game designers spanning a rangeof content areas, as well as additional curricular andassessment supports for using the games. Someof these supports provide teachers with formativeassessment tools that augment features already inthe games, and these are also described in this report.(See Appendix B for a list of the games included in thestudy). Case study teachers participated in a half-dayEMPOWERING EDUCATORS: Supporting Student Progress in the Classroom with Digital Games PART 2: Case Studies5

A REPORT OF THE A-GAMES PROJECT: ANALYZING GAMES FOR ASSESSMENT IN MATH, ELA/SOCIAL STUDIES, AND SCIENCEprofessional development session on how to use GameUpto access their chosen game prior to entering the study.This professional development included a general introduction to gameplay and features of their chosen game.We then arranged to visit each teacher’s classroomto observe them using the game with students, andfollowed each classroom visit with a phone interview todiscuss their experience using the game and how it fitinto their formative assessment practices.Our goal is to shed light onthe different ways that formativeassessment features areimplemented in games, andhow teachers recognize, interpret,and utilize those features.A unique aspect of our case studies is that they focuson the features within the games—not on individualteachers or games. This report should not be viewed asa critique of or commentary on any particular game orinstructional practice, or as a comparison of games orfeatures. Our goal is to shed light on the different waysthat formative assessment features are implementedin games, and how teachers recognize, interpret, andutilize those features.We observed the following features of games thatsupport formative assessment practice:»Points/Scores/Stars»Other Forms of Player Feedback»Dashboards of Player Progress»Screen Capture/Annotations»Essential Questions»Review Questions»Quizzes»Less Prominent Formative AssessmentFeatures– Replayability– Ability to Unlock Levels– Graphic Organizers– Game GuidesFor each feature, we briefly describe its role and use,give examples of how it “played out” in the classroomswe observed, and describe reasons why it might nothave realized its potential in the classroom learningenvironments we observed.In addition, we offer some general comments on thevalue of an aggregator or portal for the game-usingteacher.EMPOWERING EDUCATORS: Supporting Student Progress in the Classroom with Digital Games PART 2: Case Studies6

A REPORT OF THE A-GAMES PROJECT: ANALYZING GAMES FOR ASSESSMENT IN MATH, ELA/SOCIAL STUDIES, AND SCIENCEFORMATIVE ASSESSMENT FEATURES IN DIGITAL VIDEO GAMES AND HOW THEY ARE USEDFEATUREHOW FEATURE “PLAYED OUT”BARRIERSPOINTS/SCORES/STARS»students monitored own progress orperformance»“Quantified” outcome marking overallprogress or proficiency within a level»increased motivation or persistencelack of clarity about what theseindicators mean with respect tostudent learning»determined whether or not a levelshould be re-played for greaterproficiency»may not be related to contentlearning objectives»often displayed only at the end ofa level (not displayed persistently)OTHER FORMS OFPLAYER FEEDBACK»asked students to pay attention tothe in-game feedback they werereceiving»students might not perceive orunderstand the information theywere being givenGame responses to playeractions to encourage ordiscourage future action»allowed over-the-shoulderobservations»»focused each student’s attentionon areas of weakness before theyreplayed the gamedifficult for teachers to assessstudents’ learning and in-gameinteractions in larger classesDASHBOARDS»focused students in their further playand learning»technical complications involvingstudent logins»reviewed the report with the student»»helped guide post-game discussionwith the entire classmay be better with longer-formgames or when displaying progressacross games»created a snapshot at the end ofevery level»can interrupt the “flow ofexcitement”»got a sense of overall classperformance»used the comment function of featureto encourage student reflection andto gain insight into students’ thinkingduring game play»documented that students hadcompleted the assignment»sent feedback to students»used essential questions at varioustimes during a lesson»teachers often create theirown assessment questions»focused students on what theylearned from game playInformation display that allows oneto quickly scan a range of informationpertaining to a taskSCREEN CAPTURE/ANNOTATIONSCapture ‘snapshots’ of the screen at anymoment during a game play experience,add text annotation as explanation orreflectionESSENTIAL QUESTIONSHelp teachers focus students’gameplay on key concepts andideas; could be used at thebeginning or end of game playEMPOWERING EDUCATORS: Supporting Student Progress in the Classroom with Digital Games PART 2: Case Studies7

A REPORT OF THE A-GAMES PROJECT: ANALYZING GAMES FOR ASSESSMENT IN MATH, ELA/SOCIAL STUDIES, AND SCIENCEFORMATIVE ASSESSMENT FEATURES IN DIGITAL VIDEO GAMES AND HOW THEY ARE USEDFEATUREHOW FEATURE “PLAYED OUT”BARRIERSREVIEW QUESTIONS»students received feedbackabout their answer or were shownthe correct answer»no direct way for teachers tosee students’ progress (withoutdashboard)»students were asked to check offa list of objectives met by theirwriting before moving on to the nexttask in the game»students were sometimes notmotivated to spend enough timeon them»kept track of what students learned»provided informal checks on learning»»set ‘cut scores’ for whether studentsshould replay a level or move onnot aware of the available quizfeatures»»offered a “quick read” on howstudents were doingusefulness of a quiz depended onthe purpose of game use»»offered assessment data that couldbe used immediately to informinstructionthe extent to which quiz contentwas aligned to the important gamecontent»games where therewas no “right” answer hadhigher re-playability»games with one “right”or “best” answer have no orlimited replayability»the majority of a class played andwas assessed on content at an entrylevel, an advanced student could playand be assessed on more advancedcontent»generally associatedwith replayability»generated traces of student workthat the teacher could review»teachers sometimes preferredtheir own materials»elicited student reflection inadvance of the group discussion»concerns about printing»students work at different paces»reduced cognitive load for studentsas they played the game»provided overviews of game»»prompted small-groupdiscussion activities»provided support for peerfeedbackbecause game designers cannotalways anticipate context, theextent of guidance for how togather and interpret informationabout student learning is limitedReview questions or objectives at theend of a levelABCDQUIZZESQuiz features provided by the game designers,including videos with accompanying quizzesand/or a tool for generating quizzes providedby portalREPLAYABILITYGame can be played multiple times forpractice; new experience each timeABILITY TO UNLOCK LEVELSA code that unlocks all levels of the gameat once; allows teacher to tailor difficultyfor studentsGRAPHIC ORGANIZERSSupplemental worksheets anddocuments that accompanygame experiencesGAME GUIDESSupplements to lesson plans thathelp teachers integrate games into theircurricula; include guidance on how toassess student learning with the gameEMPOWERING EDUCATORS: Supporting Student Progress in the Classroom with Digital Games PART 2: Case Studies8

A REPORT OF THE A-GAMES PROJECT: ANALYZING GAMES FOR ASSESSMENT IN MATH, ELA/SOCIAL STUDIES, AND SCIENCEFEATURESPOINTS/SCORES/STARSSalen & Zimmerman (2003) define games as “a systemin which players engage in an artificial conflict, definedby rules, that results in a quantifiable outcome” (p. 80,italics added). In many digital video games, the outcome is “quantified” as a score, a report of points, or adisplay of stars. These displays take a number of forms,but in general they are a report to the player about accomplishment or proficiency. Because stars and pointsare common elements of video games, students mayfind such scoring mechanisms to be a natural and expected part of using an educational video game. Manyof the games in our study employ one or more of theseelements. Points, scores, and stars have the potential tobe a key to teachers’ formative assessment of studentperformance and/or learning from game play, servingas an objective indicator of progress and proficiency.When a game reports points or scores, such as thescore reporting in the game Quandary by LearningGames Network, the points may represent the player’soverall level of accomplishment during a round of play.Points are earned for correct moves, implying boththat there is a maximum possible or “most correct” setof responses or moves in a game, as well as a way toimprove one’s score in future play. Players can com-Scoring, like that in Battleship Numberline, has the potential to be a key toteachers’ formative assessment of student performance. Please note: not allgames can be used on tablets, please check BrainPop’s website to learn more.4Points, scores, and stars have thepotential to be a key to teachers’formative assessment of studentperformance and/or learning fromgame play, serving as an objectiveindicator of progress and proficiency.pete against other players for the best score or againstthemselves to improve their score. In other games, asin Battleship Numberline by Playpower, the points arereported as a more abstract average accuracy, in whichthe implied goal is to make progress toward 100%accuracy with each response.Stars function in a similar fashion to points, byindicating how completely a player has accomplishedthe goals of a particular level. For instance, in MonsterSchool Bus by New Mexico State University LearningGames Lab, players are encouraged to plot a schoolbus route to pick up combinations of students thatresult in both a “full bus” and “rock out buildings.” It ispossible to complete a level without having earned thefull “three stars” if both goals are not completely met.Students can continue on through the levels of game,but can also go back and improve their game performance (and demonstrate greater mastery) by earningall three stars. Like points, stars can represent achieve-Monster School Bus uses stars in a similar fashion to points, indicating howcompletely a player has accomplished the goals of a particular level.EMPOWERING EDUCATORS: Supporting Student Progress in the Classroom with Digital Games PART 2: Case Studies9

A REPORT OF THE A-GAMES PROJECT: ANALYZING GAMES FOR ASSESSMENT IN MATH, ELA/SOCIAL STUDIES, AND SCIENCEment of multiple types of goals. In the game PlanetMechanic by Filament Games, for example, one star isawarded for a correct solution to the overall problemin each level on the first attempt and another star isawarded for correctly answering a “challenge” questionabout the science content in the level.How did this feature “play out” as a formativeassessment tool?Points, scores, and stars can play a key role in formativeassessment, and in particular in student self-assessment. In our interviews, teachers indicated that theywould draw students’ attention to these displays inorder to:»monitor their own progress or performance;»increase their motivation or persistence tocomplete the task or level; and»determine whether or not a level should be replayed for greater proficiency.engaged with each other. There were kids that were notsharing a computer that were actually helping each otherout.” Another teacher used students’ scores to spurin-class competition between the boys and girls.What can prevent this feature from realizing itspotential?Points, scores, and stars are clearly important gamedesign features, especially for educational games whereprogress and mastery are important. However, weobserved a range of implementation-related issues thatinhibited the utility of this game feature. The greatestlimiting factor is the clarity with which game designerscommunicate with teachers what these indicators meanwith respect to student learning. This understanding iscritical for teachers to assess learning related to gameplay, and can affect students’ views of the game. According to one teacher, “I did notice [the points], and the kidsnoticed it. I thought that I didn’t understand the relationship between the points earned and the task that the kids“My only surprise is how engaged they were instead of having a tantrum[when they did not succeed], they went back to try and gain stars . [Studentswere] engaged with each other. There were kids that were not sharing acomputer that were actually helping each other out.” — case study teacherTeachers used points, scores, and stars as criteriafor determining when acceptable performance wasachieved by students. They also used them to checkthat students had complied with a game-relatedassignment, such as playing a game for homework, orreplaying a level for greater mastery. In some cases,teachers could use points or stars to make comparisonsof progress or mastery across students, to identifywhich students needed extra support, or to challengestudents. Points, scores, and stars provide motivationfor students, both for accomplishing the goal and as away of sharing their progress with others. As one teacherfound, “My only surprise is how engaged they were instead of having a tantrum [when they did not succeed],they went back to try and gain stars . [Students were]were performing to earn the points . It seemed like thepoints were kind of an afterthought . If there isn’t a [clear]rationale behind the points, it can give the impression thatthe whole enterprise is somehow fraudulent or fake .”Some teachers found “teachable moments” when agame’s point display was unclear. In one case, a teacherin our study had students engage in deductive reasoning alongside their game play to try to determine howpoints translated into bonuses or feature unlocks in thegame. In contrast, one teacher expressed a concernthat points could potentially distract students fromaspects of game play that directly promote learning,for instance, when points are needed to unlock gamefeatures that are not in direct line with the learningobjectives. Whereas the first issue calls out the challengeEMPOWERING EDUCATORS: Supporting Student Progress in the Classroom with Digital Games PART 2: Case Studies10

A REPORT OF THE A-GAMES PROJECT: ANALYZING GAMES FOR ASSESSMENT IN MATH, ELA/SOCIAL STUDIES, AND SCIENCEof interpreting points with respect to learning, thesecond highlights the importance of game mechanicsthat closely align with learning objectives. We note thisas a general drawback to points or scores in game-basedlearning, where the metaphors and mechanics of thegame are intended to serve the overall “cover story”or fantasy of the game, and as such may not align wellwith the core learning objectives. Later in this report wediscuss dashboards, which have the potential to helpteachers and students tease apart game feedbackassociated with specific learning goals from overall gameplay feedback.Scores may not provide all of theinformation teachers desire.A further challenge to understanding the meaning ofpoints and stars is that in some games, stars indicatemastery of different goals, some of which may not berelated to the content learning, or clearly related toparticular goals a teacher is focused on. In one mathgame in our study, students could earn three starsfor each level. Mastering the mathematics contentearned students two of the three stars, but the finalstar required students to master a component of gameplay unrelated to the math. Additionally, scores may notprovide all of the information teachers desire. In a gamedesigned to teach argumentative writing, for instance,the score only reported percent completion, withoutany indication of writing quality.Another limiting factor in how useful points or starsare in formative assessment practice relates to theirdisplay. In some games, points and stars are displayedonly at the end of a level, while in others they aredisplayed all the time, and are continuously updated onthe screen. These displays may be useful to the studentwho is continuously engaged with the game, but noteasily accessible by teachers. Teachers in this situationwould resort to informal assessment techniques toaccess students’ scores, including “over the shoulder”approaches, or having students “call out” points whenthey finish a level. This problem is reduced or eliminatedwhen the points or stars are also communicated viaa dashboard or similar feature, where teachers canaccess information about students’ progress over time.As a workaround, many teachers in our study directedstudents to use GameUp’s SnapThought feature,which is a way to capture and annotate screen shots, torecord points or stars when they are displayed only atthe end of a level. We describe SnapThought’s role as aformative assessment support on page 15.OTHER FORMS OF PLAYERFEEDBACKGames provide feedback to players through a varietyof means beyond points, scores, or stars. For our study,we defined feedback as actions or information that occurin response to the students’ actions and decisions inthe game. Feedback plays an important role in bothlearning and game play. In i

research base than short-form games and are focused on higher order thinking skills that align more naturally with new common core standards. These games do not fit as easily into the existing school day or classroom time period, but are the source of new experimentation in the research community and a variety of school