Textbooks. The Lack Of Emphasis On Applicatn, Analysis .

DOCUMENT RESUMESE 052 783ED 344 770AUTHORTITLERisner, Gregory P.; And OthersLevels of Questioning in Current ElementaryTextbooks: What the Future Holds.PUB DATENOTE91PUB TYPEEDRS PRICEDESCRIPTORSIDENTIFIERS18p.; Paper presented at the Annual Meeting of theMid-South Educational Research Association(Lexington, KY, 1991).Reports - Research/Technical (143) -Speeches/Conference Papers (150)MF01/PC01 Plus Postage.Grade 5; Intermediate Grades; Item Banks; *KnowledgeLevel; Science Education; *Science Materials; ScienceTests; *Test Items; *Textbook Evaluation; *ThinkingSkills*Blooms TaxonomyABSTRACTIn an attempt to determine the level of questioningaccording to Bloom's taxonomy, generated by test items that accompanyselected fifth-grade science textbooks, an analysis of the testsprovided by three of the most commonly used elementary sciencetextbooks was completed. Additionally this analysis examined theextent to which each series emphasized the higher levels ofquestioning and what significant differences, if any, existed betweenthe old and new publications in the number of knowledge-level andabove-knowledge-level test items included in end-of-chapter tests.One hundred items from end-of-chapter tests were analyzed from eachedition (old and new) of the textbook series (200 per series), whichrepresented a total sample consisting of 600 items. Interrateragreement on training materials reached the 96% level. The 600 itemswere randomly selected and categorized by the researchers and oneexperienced classroom teacher, according to the six major categoriesincluded in Bloom's taxonomy: knowledge, comprehension, application,analysis, synthesis, and evaluation. Based on the assumption that thethree science textbooks selected for study are representative of allscience books, one conclusion is that higher order thinking skillsare not elicited by end-of-chapter tests that accompany sciencetextbooks. The lack of emphasis on applicatn, analysis, synthesis,and evaluation levels of questioning indicates a gan between thestated goals of elementary science textbooks and actual publishedmaterial. *************************Reproductions supplied by EDRS are the best that can be madefrom the original ******************************

Levels of Questioning in Current Elementary Textbooks:What the Future HoldsPrepared by: Gregory P. RisnerJanice I. NicholsonJanice G. MyhanPresented to 1991 Annual Meeting ofMid-South Educational Research AssociationLexington, KentuckyPERMISSION TO REPRODUCE THISMATERIAL HAS BEEN GRANTED BYGregory P. RisnerU.S. !APARTMENT OF EDUCiTIONOffice of Educational Research and improvementEDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)IC This document has been reproduced asreceived from the person or organizationoriginating it1' Minor changes have been made to improvereproduction qualityroTHE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC).-Points of view or opinions stated in this documant do nol necessarily represent officialOm position or policyNIL

Levels of Questioning in Current Elementary Textbooks:What the Future HoldsGregory P. RisnerJanice I. NicholsonJanice G. Myhanby:Critical thinking maintains a prominent position among desiredlearning outcomes in elementary science. Since the textbook science,students' thinking most often parallels the cognitive level(s) ofthe textbooks being used.Questioning has beeit labeled by Hilda Taba as the "mostinfluential teaching act,"and research verifies the role ofhigher-cognitive questioning in eliciting critical thinking (Andre,1979; Doak, 1970; Hunkins, 1968; Redfield and Rousseau, 1981; Taba,Levine, and Elzey, 1964).Because teachers use textbooks as the primary source ofinstruction (Smith, 1980, cited by Mechling & Oliver, 1983), it isnecessary rials(e.g.,textbookspublishedand theirtests)todetermine their capacity to develop higher-level thinking mpanyingmaterials such as study guides, chapter and unit tests, posters,parent letters and other materials.Many textbook publishersinclude scope and sequence charts which emphasize the fact that thetext and accompanying materials emphasize divergent, open-endedquestions which facilitate critical thinking skills.

Most of the research studies related to questioning have beenlimited to questions contained in textbooks, and teachers' manuals.The research literature does not contain studies concerning thecognitive levels of test items that accompany elementary sciencetextbooks.This research study compares the levels of questionsfrom science tests accompanying textbooks published in the 1980s tothose that will be used in schools in the 1990s.ANALYZING TEXTBOOK TESTSIn an attempt to determine the level of questioning accordingto Bloom's (1956) taxonomy, generated by test items that accompanyselected fifth-grade science textbooks, an analysis of the testsprovided by three of the most commonly used elementary sciencetextbooks was completed.Additionally this analysis examined theextent to which each series emphasized the higher levels ofquestioning and what significant differences,ifany,existedbetween the old and new publications in the number of knowledge-level and above-knowledge-level test items.Moreover,it washypothesized that there would be no significant difference betweenthe old and new textbook editions in the distribution of knowledge-level and above-knowledge-level test items included in end-ofchapter tests.PROCEDURESTextbook adoption information was obtained from the statedepartments of education of Alabama, California, Tennessee, andTexas to obtain a rank ordering of the fifth-grade series used mostfrequently for all states combined.Three of the most widely used24

textbooks, both old and new editions, publ shed in the 1980s wereanalyzed:Accent on Science (1983) and Merrill Science (1989),both published by Charles E. Merrill Company: Holt Science (1984)and Holt Science (1989), both published by Holt, Rinehart, andWinston Company: and ScottForesman Science (1983) and Scott.s.Foresman Science (1989), both published by Scott, Foresman andCompany.One hundred items from end-of-chapter tests were analyzed fromeach edition (old and new) of the textbook series (200 per 0items.Interrater agreement on training materials reached the 9E% level.The 600 items were randomly selected and categorized by theresearchers and one experienced classroom teacher, according to my:knowledge, comprehension, application, analysis, synthesis, andevaluation.Results of the classificatiois were analyzed andpresented in frequency distribution tables and a series of circlegraphs.The chi-square statistic was selected to test the proposedhypotheses.FINDINGS AND CONCLUSIONSAs indicated in Table I, a total of 173 (58%) of the 300 testitems in the old editions represented the knowledgecategory.These series contained 109 (36%) comprehension items and only 6%application items.Test items reflective of the -nalysis andsynthesis categories were nonexistent in the distribution; only oneevaluation item existed.3

Among the new editions (see Tab.1,II), the knowledge categoryreceived even greater representatici collectively.Of the 300items analyzed, 209 (70%) items were judged as knowledge,85 (28%)as comprehension, and only 6 items as application.This analysis also examined the extent to which the test itemsin both the cld and new editions emphasized the higher levels ofquestioning.For this purpose, levels two through six of Bloom's(1956) taxonomy were combined and percentages were calculated.Bycollapsing the upper five levels into one major category (i.e.,higher level questions), both new and old textbook editions wereexamined according to their emphasis on understanding or applyingscience information (higher-level)(lower-level).as compared to memorizationThe resulting dichotomies of the old publicationsas shown in Figures 1-3, indicated that two series, Merrill andHolt, possess generally balanced disbursements of lower and higherlevel questions on the published tests, thereby exposing studentsto cognitive stages other than mere recall.Closer examination ofthe higher level questions reveals that Merrill contains moreapplication items than Holt, however both series demonstrated apreponderance of comprehension items.While providing cognitiveexercise other than memorization; Merrill and Holt focus primarilyon the lowest level of understanding.As indicated by Figure 3,the older edition published by Scott, Foresman contained the lowestpercentage of higher level questions.The higher level category inthis series possessed 17 comprehension items and one applicationitem.One obvious conclusion is that chapter-end science tests in46

this edition predominantly emphasize mere recall of isolated bitsof scientific vealaninteresting trend with regard to the emphasis of higher levelquestions.As shown in Figures 4-6, recent publications of bothMerrill and Holt place less emphasis on the higher level questionsthatdidoldereditions;publications are striking:thesimilaritiesamongthesenewMerrill and Holt possessed only 26% and30% higher level questions respectively.On the other hand, Scott,Foresman demonstrated more higher-level test items in the neweditions.In order to determine if changes had occurred over time, thestudy compared old and new editions for each of these series todetermine if significant differences existed in the distributionsof knowledge and abovE -nowledge-level test items.The chi-square(X2) statistical technique was employed through a serles of 2 x 2contingency tables (Textbook Editions:Old and NewxKnowledge-Level vs. Above-Knowledge Level Questions).Results of these analyses revealed that Merrill containedsignificantly more knowledge-level items in the new editions (X2 23.98; p .001).A similar emphasis on the knowledge category wasdemonstrated by the new Holt science series as indicated by thesignificant (p .001) chi-square (X2) value of 9.15.The Scott,Foresman series however, demonstrated no significant difference (X2 3.37) in the level of questioning between the old and new5

editions, indicating that questioning levels in this text remainedunchanged.DISCUSSIONBased on the assumption that the three science textbooksselected for study are representative of all science books, oneobvious conclusion is that higher order thinking skills are notelicited by end-of-chapter tests that accompany science textbooks.The lack of emphasis on application,analysis,synthesis,andevaluation levels of questioning indicates a gap between the dactualpublishedIt is recommended that science textbook publishersinclude test items that engage students with substantial practicewithcomprfthension,evaluation quetA.ions.application,analysis,Teachers must alsosynthesis,anddevise supplementaryquestions, both written and oral, that facilitate the developmentof higher cognitive processes.Similar research in the area of science is needed to determinethe cognitive levels of questioning utilized in teacher's editionsfor classroom use and pupil's textbooks for chapter-end review.Other recent experimental studies are needed to determine theeffects of higher-level questions on student achievement, attitude,and interest in elementary science.Prepared by:Gregory P. RisnerJanice I. NicholsonJanice G. Myhan6