We The People: Elementary Pre-Service Teachers And .

following subsections discuss relevant literatureregarding the cloze procedure as a means tomatch students with leveled text, text complexityin light of readability formulas such as the Lexileframework.Cloze ProcedureThe cloze procedure was developed by Taylor(1953) as a means of measuring the complexitylevel of printed material for individual readers.For his purposes, Taylor defined the clozeprocedure as a “psychological tool for gaugingthe degree of total correspondence between (1)the encoding habits of the transmitter and (2)the decoding habits of the receivers” (p. 415).Taylor suggested that the application of thecloze measure requires the interception of amessage between a transmitter, who is the reader(or writer) and a receiver, who is the reader (orlistener).Taylor (1953) developed measures in whichevery fifth word was deleted. The subjects wereadvised to try to fill in the deleted terms. Taylorreasoned that, based on Gestalt psychologicalprinciples, a person presented with a nearlycompleted circle would perceive the almost-circleas a whole. Taylor determined that the sameprinciples would apply to language. In otherwords, Taylor proposed that ability to completepassages with deletions is an acceptable predictorof how difficult that passage is for the individualwho is charged with the task of completing thecloze measure. For example, Taylor suggested thatupon seeing a sentence such as, “The Americanflag is , white, and blue.” Readers familiarwith the content would “almost instantaneouslyand quite unconsciously close the gaps” (p. 415)by filling in the word “red”. Overall analysis offindings, led Taylor to conclude that readabilityand comprehensibility are synonymous terms.Substantial research has demonstrated thevalidity of cloze measures to determine difficultylevel of particular reading passages for individualSRATE Journalreaders. Bormuth (l966), for example, determinedthat cloze measures assess comprehensionin much the same way as well-constructedtraditional multiple-choice questions. Bormuth(1969) replicated the earlier study and foundsimilar results as Rankin and Culhane (1969). Inthis light, Ransom (1970) compared the readinglevels found by a cloze measure and an informalreading inventory for 178 boys and girls in gradesone through six. Ransom concluded that the clozeprocedure “could aid the teacher in determiningthe appropriate instructional reading level andthe level of material that would be frustrating forchildren to read” (65).Extensive review of research coupled withquantitative research involving more than1,000 subjects led O’Toole and King (2010) toconclude that the cloze procedure provides auseful estimate of the “accessibility of a particulartext for a particular group of readers” (p. 305).A strength of the cloze procedure is ease ofconstruction based directly on reading materialsthat individuals are expected to read, usually inacademic settings. In addition, cloze measures areobjective and not influenced by extraneous factorssuch as the competence of test constructors.Analysis of research on cloze suggests thatthe cloze procedure is a valid measure of textdifficulty. There are several sets of criteria fordetermining reading level based on completionof cloze passages. All of the criteria are close inestimating reading levels. The authors determinedthat the criteria determined by Ransom (1970) fordetermining reading level, though based on datagathered in the early 1970s, are still valid:1. Independent Level: 50% or more correctreplacements2. Instructional Level: 30% to 49% correctreplacements3. Probable Frustration Level: 20% to 29%correct4. Frustration Level: Below 20% correctFall - Winter 2014, Vol. 24, Number 1Page 50

Text ComplexityReadability formulas have existed for years(Klare, 1984), but the debate about the readability formula that accurately predicts a reader’sability to comprehend a text still exists (Benjamin, 2012). Despite this, CCSS urge teachers touse increasingly complex texts with all readers(National Governors Association, 2010). Thisemphasis on text complexity is based on researchthat shows students who perform well on ACTquestions from complex texts are more likely toexperience success in college (ACT, 2006). CCSSpresent three factors to measure text complexity: quantitative evaluation of the text, qualitative evaluation of the text, as well as matchingreaders to texts. Yet, the factor that has receivedthe most attention is quantitative evaluations oftext (Hiebert, 2012). Benjamin problematizesquantitative evaluations of text complexity forearly grades because of the number of texts thatstudents engage with over the course of a yearand because the features of early grade texts haverarely been studied. Still yet, quantitative measures are widely used to measure text complexity.Traditional efforts to measure text complexityhave gained widespread use thanks in part tothe prevalent naming of Lexiles throughout theCCSS. Traditional calculations, such as Lexiles,use sentence length, word length, and wordfrequency (Chall & Dale, 1995; Smith et al.,1989; School Renaissance Inst., Inc. 2000) tomeasure text complexity. The corpus of data onwhich researchers base these calculations suggeststhat word frequency is an established indicatorof how well readers are able to comprehend texts(Just & Carpenter, 1980).Lexile FrameworkThe Lexile Framework was developedby Smith and his colleagues (1989) as adevelopmental scale to monitor how well readerscomprehend texts over time. The company usespassages of text known as “slices” over and overSRATE Journalto determine the average word frequency andaverage sentence length of an entire book. Toreduce sampling error, Kamile (2004) reportedthat Metametrics, Inc., owner of the LexileFramework, calculates their Lexile levels using acorpus of over 300 million words. These averagesare used to derive the Lexile level for books.Students then take one of 25 tests linked to Lexilelevels. Once students obtain their Lexile levelfrom these tests, students may choose from about141,847 leveled books (see Lexile web resource).The Lexile measure provides students the levelat which he or she can successfully comprehend75% of what was read (Smith, et al., 2009). Astatement on the Lexile website states, “thereis no direct correspondence between a specificLexile measure and a specific grade level”.However, Metametrics, Inc., provides overlappinggrade-band Lexile levels. Recently, Metametrics,Inc., established “stretch” grade bands in responseto the CCSS call for students to grapple withcomplex texts (see Table 1).Table 1: Lexile FrameworkGradeCurrentBandLexile 60L–1010L9-10960L–1115L“Stretch”Lexile �1335LMethod & Data CollectionThis study was undertaken to determine thereading difficulty level of the Constitution forpre-service elementary teachers in a large, publicuniversity in the southeastern United States. Allof the students were determined to be capablereaders based on ACT reading scores and hadcompleted at least two years of college level worksatisfactorily with selective admission into theteacher education program. The participants wereall elementary education majors planning to earnteaching licensure in grades K-6.Fall - Winter 2014, Vol. 24, Number 1Page 51

This study began in the fall semester of 2011and continued on through the Spring semesterof 2013. One hundred and fifty-one participantsengaged in the Constitution cloze study.Participants were elementary pre-service teachersenrolled in several reading and social studiesmethods courses. Participants were provided withinstruction in completing a practice cloze measureduring class time prior to the administration ofthe cloze measure on the Constitution. Providinginstruction in completing a cloze measure isimportant since students usually expect to behighly successful when completing a test. In otherwords, they can expect to encounter items thatare very difficult given the structure of a clozemeasure.Participants were advised that the purposeof the measure was to estimate the difficulty ofreading the Constitution and such informationwould be helpful to their own development asreading and social studies teachers and for othersinterested in the teaching of civics. In addition,they were advised that they would in no waybe identified or evaluated on their performance.Participants were given the option to declineparticipation however, all students elected toparticipate.As stated above, participants were givena practice cloze measure based on a children’sstory to familiarize them to the process. Next,students were provided a cloze measure (textand answer sheet) for Article 2, Section 1 of theUnited States Constitution. Participants weregiven one 50-minute class period to complete the50 item cloze measure. Most participants spentless than twenty-five minutes on the completionof the measure. Raw scores were determinedand percent correct tabulated and noted basedon students’ score at the frustrational, probablefrustration, instructional, and independent levels.SRATE JournalResults and ConclusionsBased on the Microsoft Word Reviewfunction, the Flesch-Kincaid Grade Level ofArticle 2, Section 1 of the US Constitution is 14.1.This means that this text would be equivalent toa text used in the first month of the 14th grade,or as a college sophomore. Using the LexileAnalyzer, a software program freely accessibleto the public on the Lexile website, this materialhas a mean sentence length of 24.45, a mean logword frequency of 3.63, and a word count of 269,placing the Lexile level at 1290L. This 1290Lplacement falls above the 11th grade and Collegeand Career Ready band of 1070L-1220L, butwithin the CCSS suggested 11th and 12th grade“stretch” band of 1185L – 1385L. This affirmsour findings that this text would be difficult forcollege sophomores. Indeed, college sophomoresexperienced difficulty with this text.Table 2: Pre-Service Teachers Score onConstitutional ClozeLevelNumberFrustration20% or less4correctProbableFrustration2720 - 29%correctInstructional30 - 49%105correctIndependentgreater than1550% correctn 151Overwhelmingly the data suggests that thereadability of pre-service elementary teachersof the Constitution is at the Probable FrustrationLevel leaning towards the Instructional Level,87% of the raw scores fall into the two levels(see Table 2). Participants generally had a toughFall - Winter 2014, Vol. 24, Number 1Page 52

time completing the cloze measurement andstruggled greatly with readability of the passage.These findings have implications for both readingeducation and social studies education. In termsof reading education, questions remain abouthow to determine the complexity of a text andhow teachers will find the time to qualitativelylevel texts. For social studies education, questionsremain about how these changes will impact thecontinued marginalization of social studies andcivic education in their future classrooms.How to determine the complexity of a text?Quantitative analysis of texts is easy toretrieve, but it is prudent that teachers makedecisions about text complexity using multiplemeasures. Just as there is no one way to helpstruggling readers, there is not one method toadequately measure the complexity of a text(Bailin & Grafstein, 2001). Quantitative analysesfail to measure the qualitative nature of the textor the special understanding that teachers holdregarding their students. Benjamin notes (2012),“selecting appropriate texts for a population ofreaders requires some understanding of both thereader and the text, and different methods maybe more or less appropriate for different types oftexts and different populations of readers” (p. 64).It is not by chance that the authors of this articlechose Section 2, Article 1 of the US Constitution.Our study confirmed our beliefs that Section 2,Article 1 of the US Constitution is a difficulttext. This is despite the fact that the majority ofthe words in this text fall within the first 1000words of Fry’s Instant Words (Fry, 1999), a listoften used by teachers, students, parents, andpublishers of children’s texts to create curriculummaterials and “readable” texts. Almost all textleveling formulas rely on vocabulary and syntaxin their calculations (Hiebert, 2013). To calculatevocabulary, computer programs compare thewords in the text to a corpus of words that occurfrequently in texts. The mean sentence lengthof our text example was calculated at 24.45indicating a high degree of complexity, but stillSRATE Journalwithin the capabilities of college sophomores.Yet, this text proves to be more complex thanwhat the readability level indicates due to thehistorical syntax, syntax largely unfamiliar to thisgroup of students. Indeed, using 46,000 responseson 252 non-publicly available assessment tasks,White (2012) found that “particular text features(e.g., long sentences), for example, are not alwayseasy or difficult in and of themselves; rather theirinfluence on literacy tasks depends on the contextin which they appear; their interface with thecognitive and linguistic demands of the task, andaccordingly, with the required readers’ skills”(p. 161). Thus, if teachers are to make informeddecisions about complex texts, they need timeto critically examine texts for the cognitiveand linguistic demands of the task in light ofknowledge about their students’ skills.Finding Time to Examine TextsWhile CCSS provides specific guidance onchoosing texts using quantitative methods, theirAppendix A (Student Achievement Partners,2012, p. 6) offers general guidance about howto select texts using qualitative methods or howteachers should take the reader and task intoconsideration. The CCSS authors do suggest thatthe qualitative and text/reader considerationsrequire human readers, but fail to name teachersas the qualified individuals capable of makingdecisions about the quality of texts used in theirclassrooms. Instead, CCSS authors state thatqualitative decision making “involves making aninformed decision about the difficulty of a textin terms of one or more factors discernible to ahuman reader applying trained judgment to thetask” (p. 5). The measures suggested are vague atbest, presenting anchor criteria for the followingfactors: 1.) levels of meaning and purpose;2.) structure; 3.) language conventionality andclarity; 4.) knowledge demands: life experiences,cultural/literary knowledge, or content disciplineknowledge.Fall - Winter 2014, Vol. 24, Number 1Page 53

Recently, Hiebert (2013) established asimplified four-step process for teachers toconsider when choosing texts that will challenge,but not frustrate readers. This four-step processknown as the Text Complexity Multiple-Index(TCMI) calls for teachers to:1. Check the Lexile level with other levelingsystems such as guided reading levels(Fountas & Pinnell, 2010). Using thesemethods, teachers should make note of thevocabulary as well as the length of sentencesin the text.2. Compare texts to benchmark texts, includingthose exemplar texts listed as exemplar textsby CCSS.3. Engage colleagues in discussions usingqualitative rubrics such as those suggestedin their Appendix A (Student AchievementPartners, 2012, p. 6).4. Consider readers’ background knowledge,te learning task involved, and the context inwhich the learning task should occur.Using this four-step process to analyze Article2, Section 1 of the US Constitution, the authorsmade the following determinations about the textused in this research:1. Text Levels: The Lexile Level of this textcalculated to 1290L, the Flesch-KincaidGrade Level calculated to 14.1. Thus, there isa difference at least a 1 to 2 year grade leveldifference between these two leveling methods.Yet, they both indicate that this text is difficult.2. Benchmark Text Comparisons: The exemplartexts at the 11th and CCR band are as follows,along with their Lexile levels: Common Sense by Thomas Paine (1776) –1330LSRATE Journal Walden by Henry David Thoreau (1854) –1200L “Society and Solitude” by Ralph WaldoEmerson (1857) – unknown on Lexile website “The Fallacy of Success” by G. K. Chesterton(1909) – unknown on Lexile website Black Boy by Richard Wright (1945) – 950L “Politics and the English Language” byGeorge Orwell (1946) – unknown on Lexilewebsite “Take the Tortillas Out of Your Poetry” byRudolfo Anaya (1995) – unknown on Lexilewebsite3. Qualitative Rubric: The authors used thequalitative rubric in Appendix A of the CCSS, toanalyze Article 2, Section 1: Levels of Meaning: The purpose of thedocument is explicitly stated, to lay out howthe states would elect a president. Structure: The writing is specific to thediscipline of History, but this is the only pointthat can be addressed in this category. Language Conventionality and Clarity:The language is archaic and otherwiseunfamiliar and specific to the domain ofHistory. Knowledge Demands: Content/DisciplineKnowledge (chiefly informational texts): Tounderstand Article 2, Section 1, readers needto have familiarity with the role of Senators,the House of Representatives, and electors inthe voting process.4. The students in our classes demonstratecapability to read complex texts, but report a lackof recreational reading. Troubling as it is, someprofessors speculate that these same students failto read class assignments. Thus, we felt that thesestudents would experience difficulty with this textdue to the historical nature of the text.We noted that only by completing aqualitative analysis of the text were we able todetermine what instructional practices teachersFall - Winter 2014, Vol. 24, Number 1Page 54

might need to scaffold student learning of thisdocument. Yet given the sheer volume of booksread by elementary students, we fear the dauntingtask of qualitative analysis of text complexity foreach document will further marginalize socialstudies instruction in the elementary grades fromboth the teacher’s perspectives as well as thechallenges faced by the students.Continued marginalization of social studies andactive civic educationLevstik (2008) states that the impact oftesting and mandates on social studies teachersare stark. As a result, elementary social studieseducation has taken a far backseat to the testedsubject areas of reading and mathematics.Integrated instruction attempts face challenges aswell. Levstik (2008) indicated that teachers willeither be hindered by their inability to plan qualityintegrated instruction or that the surface levelclaims of integrated instruction will continue tomask the real reduction in the time given to socialstudies education (p. 53).With token time given to social studieseducation and civic education, K-12 students missout on a vital and critical understanding about ourworld and the people in it. We fear that studentswill continue to grow into adulthood withoutthe civic values and skills needed to contributeto society in a meaningful way. Certainly theirability to read and analyze primary sourcedocuments will be impacted.However, we do feel that here lays animportant limitation of the CCSS push formore work with historical documents. Reismanand Wineburg (2012) seem to concur whenconsidering the usage of President Polk’s 1846message to Congress and th

the reading difficulty level of the Constitution for pre-service elementary teachers using a cloze procedure and to argue that the cloze assessment is a quick, simple, valid, low-cost means to measure students’ comprehension of complex texts. Most pre-service teachers in our