Multiple Choice Questions That Test Conceptual .
Paper ID #13350Multiple Choice Questions that Test Conceptual Understanding: A Proposalfor Qualitative Two-Tier Exam QuestionsMr. Dion Timmermann, Hamburg University of TechnologyDion Timmermann studied electrical engineering at Hamburg University of Technology, Hamburg, Germany. In his master thesis he worked on simulation methods for the signal and power analysis of highspeed data links. He currently pursues his Ph.D. in the Engineering Education Research Group at Hamburg University of Technology, where he investigates students understanding in introductory electricalengineering.Prof. Christian H Kautz, Hamburg University of TechnologyChristian H. Kautz has a Diplom degree in Physics from University of Hamburg and a Ph.D. in Physics(for work in Physics Education Research) from the University of Washington. Currently, he leads theEngineering Education Research Group at Hamburg University of Technology.c American Society for Engineering Education, 2015
Multiple-Choice Questions that Test Conceptual Understanding:A Proposal for Qualitative Two-Tier Exam Questions1IntroductionThe goal of most university courses is for students to not only learn factual knowledge but toacquire conceptual understanding of the topics taught. Consequently, a course’s assessmentshould at least in part evaluate this conceptual understanding. 1 To achieve this, there are multipleassessment methods that could be used, as for example essays or oral exams. However, many ofthese methods require a very high time investment on the part of the instructor, which is, in manycases, simply not possible. For large classes, multiple-choice tests are among the most efficienttypes of assessment. Although much care has to be taken in their development, machine-basedscoring of multiple-choice tests can significantly reduce an instructor’s work load, freeing uptime for more face to face interaction with students. However, one main point of criticism ofmultiple-choice tests is that they only test factual knowledge and not conceptual understanding. 2,3Ideally, tests for conceptual understanding and multiple-choice questions could be brought together. One possible solution is the use of two-tier multiple-choice questions, where the first tierrequires a fact-based response and the second tier a reasoning for that response. This type ofquestioning was probably first used by Tobin and Capie in 1981 in a test on logical thinking. 4In 1985 Treagust presented an approach for the design of such questions for standardized testson student misconceptions. 5 Since then, this question format has been used in several conceptinventories and similar tests. 6,7,8 Instead of standardized tests, we started using such questionsin exams. The results are very promising. Not only do these questions improve the quality ofthe exams by requiring a higher level of thinking, they also give us access to a large pool of dataabout the conceptual understanding of students at our institution, and thus allow us to monitor theprevalence of student misconceptions.This paper aims to promote this type of two-tier multiple-choice questions, especially for exams,by giving examples and guidelines for their development. We will also show some properties(difficulty and discrimination) of eight sample questions and try to gauge the influence of thesecond tier statistically.
Exam 11th of March, 2014 „Electrical Engineering 101“Exercise 2 (8 points)The circuit at right contains an ideal alternating voltagesource v0(t) with v 0 (t ) V̂0 cos( ω t ϕ 0 ) . The voltages at thev0(t)circuit elements are denoted by vR(t), vL(t) und vC(t); the cor- uresponding currents by iR(t), etc. The current through thevoltage source is denoted by i0(t).page 3 of 9RLCTwo sinusoidal signals have the same phase, if (choosing the respective sign) they reachtheir maximum at the same time.The values for R, L, C, and ω are neither zero nor infinity, but otherwise unknown.Please select for each question the correct answer as well as the corresponding reason.Question 1 (2 points, if both items are correct)Item 1.1AvL(t) and v0(t)are in phase,Bare not in phase,Ccan have the same or a different phase,Item 1.2becauseathe inductance and the source are connected in parallel.bthe inductance and the source are connected in series.cat the inductance the voltage leads.dbetween source and inductance, there is a branch with R and C.ethe phase relation depends on the value of the capacitance.Figure 1: A two-tier multiple-choice question in the format suggested in this paper, used in anexam in 2014.22.1Description and design of our two-tier multiple-choice questionsBasic structure of our two-tier multiple-choice questionsThere are several different ways to formulate two-tier multiple-choice questions. The type ofquestion used by our group is very similar to that used by e. g. Treagust 5 , Lawson 6 , and Chandrasegaran 7 . However, their questions were used in standardized tests and could be improvediteratively after being tested on students, while we use our questions in exams, where we do nothave the possibility to test them on students beforehand. While the types of questions are verysimilar, our reasoning and design approach differs because of the different usage.In the following, we are going to describe a specific type of two-tier question that we have founduseful in the context of examinations. This description will be illustrated by the sample questionpresented in Figure 1. Further similar questions can be found at the end of the article in Figures 3,4, and 5. In designing our questions we follow four central principles:Technische Universität Hamburg-Harburg, Institut für Elektrische Energiesysteme und AutomationProf. Dr.-Ing. G. Ackermann, Eißendorfer Str. 38, 21073 Hamburg
The first tier (item) asks students to apply their conceptual understanding to make a prediction about a specific situation. Thus, we do not ask students to recite a law or rule but ratherto apply a law or rule to a specific situation. The goal of the exam question shown in Figure 1was to assess if the students knew that the voltages of circuit elements connected in parallel areequal, and thus in phase. While we could have asked for this directly, students were asked thequestion shown in Figure 1, which requires them to apply their knowledge. This has two positiveeffects: On the one hand, we do not only expect students to be able to recite the law in question,but rather to be able to apply it. Thus, it is only logical to also test for this. On the other hand,this makes it easy to generate a large amount of different questions about the same topic without the possibility for students to memorize the correct answer. One can simply exchange thesituation the students have to make a prediction about.In the second tier (item), students have to choose one of several statements to justify theirprediction made in the first tier. Ideally, the items in this second tier, i. e. the reasons offered tothe students, are simple, factual statements that are either true or are believed to be true by manystudents. In Figure 1, reasons a and d are true statements. From informal classroom observationswe knew that some students have an incorrect understanding of series connections, which wouldlead them select reason b. From previous research we also knew that a large fraction of studentshas an incorrect understanding of phase relations at reactive circuit elements. 9,10 This promptedus to add reason c. When grading, only a question where both, the first and the second tier, aremarked correctly will be counted as correct. In our opinion, this combination of two items allows to better test students’ conceptual understanding than with just the first item. This will beinvestigated in the following sections.Ideally, there is no obvious relation between the items in both tiers, i. e. for each reason (second item) there is not just one answer (first item) it could apply to. This becomes easier to do ifthe reasons are factual statements. In Figure 1, reasons a, b, and d are simple statements aboutthe topology of the circuit. Reason c is an oversimplified statement about the behavior of one ofthe circuit elements in question. There are two benefits from this: Students cannot simply selectone reason or answer they know is correct and then select the respective item linked to it. Whilethis increases the difficulty of the question and requires higher level thinking, it also increasesthe number of possible answers to the question, which is ideally the product of the number ofanswers and the number of reasons.Distractors should be based on known student misconceptions. This rule is true for all multiplechoice questions. 5 Using known student misconceptions usually increases the difficulty of thequestion, but also increases the value of the answers for the teacher. It is easier to select the correct answer if all other choices are unattractive. However, if a student decides not to select a common misconception, one can be quite certain he does not hold it. Additionally, the percentage ofstudents that have a certain misconception is usually surprisingly constant, at least among similarpopulations. This allows to predict the difficulty of a question, which is especially helpful whendesigning exams where it is usually impossible to test the questions beforehand. An example forthis can be seen in Tables 1 and 2, which we will discuss in more detail later.
2.2Tips for the design of such questionsThere are many guides to the design of multiple-choice questions. 11,12,2,3 While much of the advice given in these is also true for two-tier questions, some aspects are different. In the following,we will point out the most important aspects for designing two-tier multiple-choice questions,focusing especially on the differences to single-tier questions.Ideally, 2 answers and 4 to 5 reasons should be provided. For single-tier multiple-choice questions, 3 to 5 possible answers are suggested. 11,3 If a two-tier question has na answers and nr reasons for students to select from, there are na · nr possible combinations – given there are no obviousconnections between answers and reasons. Thus, if one wanted to have 3 to 5 possible answers intotal, each two-tier question had to have two answers and two reasons. Neither this, nor the otherextreme of 5 answers and 5 reasons seems like a good solution to us. We prefer to give studentsfour to five reasons to choose from. The number of answers to choose from should be lower. Foreach answer, there should be at least two reasons “that make sense”. If there is a strong relationbetween answers and reasons, providing only two answers for students to choose from would bethe logical consequence. The format of 2 answers and 4 to 5 reasons is also used by Treagust. 5Ideas for reasons can be generated from previous open-ended questions or literature. Togenerate statements that contain the possible reasons for students to select, Treagust first askedonly single tier multiple-choice questions but required students to give a written reasoning foreach answer. 5 While this approach is very useful for standardized tests, the first tiers of examquestions can hardly be given to students before their use in an exam. However, different questions about the same concepts but with different problem set-ups can be used. Alternatively, onecan browse through the abundance of papers on known student misconceptions.Reasons can contradict each other. When writing one-tier multiple-choice questions with threeor more answers, one should not provide two answers that are the negation of each other. Thisallows a student to exclude all but these two answers by logic. Consider the question “Whichof the following properties should one-tier multiple-choice questions fulfill?”. If, among theanswers there are the choices “Two answers may be the negation of each other.” and “No twoanswers must be the negation of each other.” one of these two answers has to be true, makingit unnecessary to consider any other possible answers. However, for the reasons of a two-tiermultiple-choice question, this is different. If two reasons are the negation of each other, stillneither could be the correct reason as they do not explain the correct answer.Answers and reasonings that are too precise should be avoided. Many of the known studentmisconceptions sound absurd if they are formulated with precision, for example because theyarise from students not differentiating between two related concepts like velocity and acceleration. Because of this, we often formulate the incorrect reasons quite vaguely. If the correct reasonwas formulated with precision, it would in many cases be the statement with the highest numberof words. In these cases, a fine balance has to be found between not being obvious by too muchprecision, but still being precise enough so the reason is still correct.Questions that require reasoning chains should be avoided. We found that it is difficult toask two-tier multiple-choice questions with reasoning chains, i. e. where the reason consistsof two or three statements that build upon each other or where several conditions have to be
292 J. B. WilliamsAssertionReasonIn a small open economy, if theprevailing world price of a good islower than the domestic price, thequantity supplied by the domesticproducer will be greater than thedomestic quantity demanded,increasing domestic producer surplus.(a)(b)(c)(d)(e)BECAUSEIn a small, open economy, anysurplus in the domestic market willbe absorbed by the rest of the world.This increases domestic consumersurplus.True; True; Correct reasonTrue; True; Incorrect reasonTrue; FalseFalse; TrueFalse; False(The correct answer is (d).)Figure 2.Example ARQ questionFigure 2: Example of an assertion-reason question. Reproduction of Figure 2 from 13example, the logical consistency of written material, the validity of experimentalmet in order for a law to apply. For these problems, other types of two-tier questions might beprocedures or interpretation of data’ (Carneson et al., n.d., Appendix C).more suitable, as for example assertion-reason questions. One example of such a question fromAn examplequestion13 is illustrated in Figure 2. Like traditional MCQs, ARQsa publicationby Williamsis reprinted in Figure 2. With these questions, two statements arepresenta numberof andpossiblesolutions.contrastto traditionalgiven.One studentsconcerningwitha specificsituation,a generalone. ncludeatrue/falseelement(CAACentre,have to decide individually if they are true. If a student decides that both statements are2000).true,each itemtwo statements,an forassertionand aone.reason,that areofheSpecifically,also has to indicateif theconsistsgeneral ofstatementis the reasonthe specificThis formatlinkedquestionby the wordThestudentthen selectsfroma multiple-choicetwo-tierallows‘because’.to ask morecomplexquestions,as thedecisionsstudents havelegendto rshemustdeterminewhetherare more limited. But as Williams points out, long statements require a high level of reading13 is true or false, and then whether the ‘reason’ is true or false. If one, orthe ‘assertion’comprehension.both, of the statements is deemed false, then the answer will be alternative (c), (d), or(e) accordingly. If, on the other hand, both statements are deemed true, a third step3 isAnalysisthe samplerequiredofwherebythequestionsrespondent must determine whether the second statementprovides an accurate explanation for the first.StandardizedtestsMCQsand conceptinventoriesare tested and ARQs,evaluatedTraditionalusuallytest onlyusuallyone issue/concept.onextensivelythe other beforehand,theiruse questionas a measuringinstrument.done byanalyzing theof test cohortstestactualtwo per(the he validityof theof‘because’students. However,usuallynot beandshownto mentexamsin thecaneventassertionreasonare cannotbeanalyzedbeforehand.However,afteritsthe basis that judging the correctness of two statements must be harder than judgingadministration,the resultsexam canbe usedanalyzeit. Frommorethis analysis,instructorsthe correctnessof one,ofitanwouldfollowthattoARQspresentof an intellectualcanat least judgequality of theirquestionswith One afterwardsmight putandforwardthecasegainedthat ingyear.options (a) and (b) require a third step of reasoning (only two steps being required ifFigure 1. Bloom’s taxonomyFigure 2. Example ARQ questionthequestionslearner presentedcorrectly inidentifiesfalsestatement),withofTheFigures 1aand3 weregiven to questions488 studentsin ancorrectexam inanswers2014. These(c), (d)wereor (e)be lesseffective engineeringin terms of andlearningis, assumingstudentsfirstmaysemestermechanicalnaval outcomes.architectureThatstudentsas well asall semesteranswers logistics(a)–(e) andoccurin roughlyequalstudentsproportionsin enrolledan eeringwho werein anstatementwould onlybe thattestedin 40%of theofquestions,depth e basicsdirect and thealternatingcurrentcircuitanalrelatively less the remaining 60% of the time. However, this is hardly a reason for notusing ARQs, as a two-step question is still preferable to a single-step traditional
Exam 11th of March, 2014 „Electrical Engineering 101“page 4 of 9Question 2 (2 points, if both items are correct)Item 2.1AiC(t) and iR(t)are in phase,Bare not in phase,Ccan have the same or a different phase,Item 2.2becauseaat resistors the current is in phase with the voltage.bthe resistor and capacitance are connected in series.cat capacitances the current leads.dthe currents through resistor and source are in phase.ethe phase relation depends on the value of the inductance.Question 3 (2 points, if both items are correct)Item 3.1AvR(t) and v0(t)are in phase,Bare not in phase,Ccan have the same or a different phase,Item 3.2becauseathe resistor and the source are connected in parallel.bat resistors current and voltage are in phase.ca capacitance is connected in series with the resistor.dthe inductance has an influence on the voltage of the source.ethe phase relation depends on the value of the inductance.Question 4 (2 points, if both items are correct)Item 4.1Ai0(t) and v0(t)are in phase,Bare not in phase,Ccan have the same or a different phase,Item 4.2becauseacurrent and voltage are always in phase at ideal sources.ba capacitance and an inductance always compensate each other.cat ideal sources current and voltage always have a 90 phase difference.dthe circuit contains an inductance.ethe phase relation depends on the values of R, L and C.Figure 3: Three two-tier questions from the 2014 exam. All questions concern the problem set-uppresented in Figure 1.Technische Universität Hamburg-Harburg, Institut für Elektrische Energiesysteme und AutomationProf. Dr.-Ing. G. Ackermann, Eißendorfer Str. 38, 21073 Hamburg
ReasonabcdeSumReason(a) Question 1, n 484(b) Question 2, n 481AnswerAnswerABCSumReason49 %6%3%5%1%2%3%16 %4%1%1%0%1%3%5%52 %10 %20 %11 %6%abcde63 %27 %10 %100 %SumABCSum2%68 %1%1%0%2%6%14 %1%1%1%0%0%1%1%5%74 %16 %4%2%72 %24 %4%100 %(c) Question 3, n 481(d) Question 4, n 480AnswerAnswerABCSumReasonABCSumabcde8%15 %2%0%0%2%6%49 %4%2%1%0%4%1%5%11 %22 %55 %5%7%abcde10 %1%1%1%1%0%1%13 %6%17 %0%1%1%1%45 %11 %4%15 %7%63 %Sum25 %64 %11 %100 %Sum15 %37 %47 %100 %Table 1: Responses to two-tier questions from Figures 1 and 3 in an exam. Correct answers areprinted in bold, values below guessing probability in gray.ysis and gives an overview of three-phase current systems. The course is mostly taught in a traditional manner with 90 minutes of lecture and 45 minutes of recita
Figure 1: A two-tier multiple-choice question in the format suggested in this paper, used in an exam in 2014. 2Description and design of our two-tier multiple-choice questions 2.1Basic structure of our two-tier multiple-choice questions There are several different ways to formulate two-tier multiple-choice questions. The type of
Developing multiple choice and other objective style questions 4 2. Writing multiple choice questions THE FORMAT of a multiple choice question Multiple choice questions are the most commonly used format for presenting objective-style questions. A multiple choice question consists of two parts ñ A stem and several options or alternatives.
50 multiple choice. 5. field test 40 multiple choice field test 46 ITEMS/40 POINTS 45 ITEMS/40 POINTS 55 ITEMS/50 POINTS 45 ITEMS/40 POINTS. 12 Students compose two essays one, for each of. two writing prompts. 40. multiple choice. 5. field test. 49. multiple choice. 1. open ended. 6. field test 50 multiple choice. 5. field test 40 multiple .
IAS 36 – LỖ TỔN THẤT TÀI SẢN. xxx KHÔNG áp dụngcho Ápdụngcho x Hàng tồnkho (IAS 2) x . Tài sản tài chính (IFRS 9) x . Quyền lợi người lao động (IAS 19) x . Tài sản thuế hoãn lại (IAS 12) x . Hợp đồng xây dựng (IAS 11) x . Bất động s
Test Blueprint 10 Preparing to Write Items 11 Description of Multiple-Choice Items 12-15 Multiple-Choice Item Writing Guidelines 16-18 Guidelines to Writing Test Items 19 Sample Multiple-Choice Items Related to Bloom’s Taxonomy 21-23 More Sample Multiple-Choice Items 24-25 Levels of Performance and Sample Prototype Items 26 Good versus Poor Multiple-Choice Items 27-28 Activity: Identifying .
For grade 4, the types of test questions are Multiple-Choice questions (MC), Evidence-Based Selected-Response questions (EBSR), Text- Dependent Analysis Questions (TDA), and mode-specific Writing Prompts (WP) . Multiple Choice: Each of this type of test question has four answer choices . Some MC test questions are based on a stimulus reading passage, while other MC test questions are .
Using multiple choice questions in the classroom How multiple choice questions can aid knowledge retention and help teachers diagnose pupils’ understanding. Why might I use multiple choice questions with my students?
Grade 7, Book 1 Question Type Points Strand Content Performance Indicator Answer Key 1 Multiple Choice 1 Number Sense and Operations 7.N.9 C 2 Multiple Choice 1 Algebra 7.A.1 G 3 Multiple Choice 1 Statistics and Probability 6.S.3 A 4 Multiple Choice 1 Number Sense and Operations 7.N.6 H 5 Multiple Choice 1 Geometry 7.G.3 B
Operation Guide for the Mercedes-Benz GLA/CLA This is a basic operation guide for those who are driving the Mercedes-Benz GLA/CLA vehicle for the first time. Please read this guide before you leave the rental office if you are not familiar with operation. For more information about the vehicle, please read the instruction manual. Basic Operations to Note Before Driving the Vehicle Starting .
