R/exams: A One-for-All Exams Generator - ERum
R/exams: A One-for-All Exams GeneratorWritten Exams, Online Tests, and Live Quizzes with RAchim Zeileishttp://www.R-exams.org/
R/exams: A One-for-All Exams GeneratorWritten Exams, Online Tests, and Live Quizzes with RAchim Zeileishttp://www.R-exams.org/
R/exams: A One-for-All Exams GeneratorWritten Exams, Online Tests, and Live Quizzes with RAchim Zeileishttp://www.R-exams.org/
R/exams: A One-for-All Exams GeneratorWritten Exams, Online Tests, and Live Quizzes with RAchim Zeileishttp://www.R-exams.org/
Motivation and challengesMotivation: Many of us teach large lecture courses, also as support for other fields. For example, statistics, probability, or mathematics in curricula such asbusiness and economics, social sciences, psychology, etc. At WU Wien and Universität Innsbruck: Some courses are attended by morethan 1,000 students per semester. Several lecturers teach lectures and tutorials in parallel.Strategy: Individualized organization of learning, feedback, and assessment. The same pool of exercises at the core of all parts of the course.1
Motivation and ssessmentLectureLive streamTextbookScreencastLive quiz( Tutorial)Self test( Forum)Written examOnline test2
Motivation and ssessmentLectureLive streamTextbookScreencastLive quiz( Tutorial)Self test( Forum)Written examOnline testLearning: Standard: Textbook along with presentation slides. Streaming: Videos streamed simultaneously or (pre-)recorded.2
Motivation and ssessmentLectureLive streamTextbookScreencastLive quiz( Tutorial)Self test( Forum)Written examOnline testFeedback & assessment: Scalability: Randomized dynamic exercises required. Feedback: Support for complete correct solutions. Flexibility: Automatic rendering into different assessment formats.2
R package examsExercises: Each exercise is a single file (either .Rmd or .Rnw). Contains question and (optionally) the corresponding solution. Dynamic templates if R code is used for randomization.Answer types: Single choice and multiple choice. Numeric values. Text strings (typically short). Combinations of the above (cloze).3
R package examsOutput: PDF – fully customizable vs. standardized with automatic scanning/evaluation. HTML – fully customizable vs. embedded into exchange formats below. Moodle XML. QTI XML standard (version 1.2 or 2.1), e.g., for OLAT/OpenOLAT. ARSnova, TCExam, LOPS, . . .Infrastructure: Standing on the shoulders of lots of open-source software. . .4
R package examsTypeStatistical computingWriting/reportingReproducible researchDocument conversionImage manipulationWeb technologiesLearning managementSoftwareRLATEX, Markdownknitr,rmarkdown,SweaveTtH/TtM, pandocImageMagick, magick,pngbase64enc, RCurl, . . .Moodle, OpenOLAT,ARSnova, . . .PurposeRandom data generation, computationsText formatting, mathematical notationDynamically tie everything togetherConversion to HTML and beyondEmbedding graphicsEmbedding supplementary filesE-learning infrastructure5
Dynamic Exercises
Dynamic exercisesText file:1Random data generation (optional).2Question.3Solution ce knowledge quiz with shuffled answer alternatives.Which of these institutions already hosted a useR! or eRum conference?Dynamic numeric arithmetic exercise.What is the derivative of f (x) x a e b·x , evaluated at x c?7
Dynamic exercises: .RmdExample: Which of these institutions already hosted a useR! or eRum conference?8
Dynamic exercises: .RmdExample: Which of these institutions already hosted a useR! or eRum conference?Question Which of these institutions already hosted a useR! or eRum conference?Answerlist---------* Uniwersytet Ekonomiczny w Poznaniu* Agrocampus Ouest* Technische Universität Dortmund* Universität Wien* ETH Zürich* Københavns Universitet8
Dynamic exercises: .RmdExample: Which of these institutions already hosted a useR! or eRum conference?Solution The list of useR!/DSC and eRum hosts can be found at https://www.R-project.org/conferences.html and https://erum.io/ , respectively.Answerlist---------* True. eRum 2016 was hosted in Poznan.* True. useR! 2009 was hosted at Agrocampus Ouest, Rennes.* True. useR! 2008 was hosted at TU Dortmund.* False. Universität Wien did not host an R conference yet (only TU Wien and WU Wien).* False. ETH Zürich did not host an R conference yet.* False. Københavns Universitet hosted DSC but not useR! or eRum.9
Dynamic exercises: .RmdExample: Which of these institutions already hosted a useR! or eRum conference?Solution The list of useR!/DSC and eRum hosts can be found at https://www.R-project.org/conferences.html and https://erum.io/ , respectively.Answerlist---------* True. eRum 2016 was hosted in Poznan.* True. useR! 2009 was hosted at Agrocampus Ouest, Rennes.* True. useR! 2008 was hosted at TU Dortmund.* False. Universität Wien did not host an R conference yet (only TU Wien and WU Wien).* False. ETH Zürich did not host an R conference yet.* False. Københavns Universitet hosted DSC but not useR! or eRum.Meta-information exname: R conferencesextype: mchoiceexsolution: 111000exshuffle: 59
Dynamic exercises: .RnwExample: What is the derivative of f (x) x a e b·x , evaluated at x c?10
Dynamic exercises: .RnwExample: What is the derivative of f (x) x a e b·x , evaluated at x c? echo FALSE, results hide ## parametersa - sample(2:9, 1)b - sample(seq(2, 4, 0.1), 1)c - sample(seq(0.5, 0.8, 0.01), 1)## solutionres - exp(b * c) * (a * c (a-1) b * c a)@10
Dynamic exercises: .RnwExample: What is the derivative of f (x) x a e b·x , evaluated at x c? echo FALSE, results hide ## parametersa - sample(2:9, 1)b - sample(seq(2, 4, 0.1), 1)c - sample(seq(0.5, 0.8, 0.01), 1)## solutionres - exp(b * c) * (a * c (a-1) b * c a)@\begin{question}What is the derivative of f(x) x {\Sexpr{a}} e {\Sexpr{b}x} ,evaluated at x \Sexpr{c} ?\end{question}10
Dynamic exercises: .RnwExample: What is the derivative of f (x) x a e b·x , evaluated at x c?\begin{solution}Using the product rule for f(x) g(x) \cdot h(x) , where g(x) : x {\Sexpr{a}} and h(x) : e {\Sexpr{b}x} , we obtain\begin{eqnarray*}f'(x) & & [g(x) \cdot h(x)]' g'(x) \cdot h(x) g(x) \cdot h'(x) \\& & \Sexpr{a} x {\Sexpr{a} - 1} \cdot e {\Sexpr{b}x} .\end{eqnarray*}Evaluated at x \Sexpr{c} , the answer is\[ e {\Sexpr{b}\cdot \Sexpr{c}} \cdot \Sexpr{c} \Sexpr{a-1} \cdot(\Sexpr{a} \Sexpr{b}\cdot \Sexpr{c}) \Sexpr{fmt(res, 6)}. \]Thus, rounded to two digits we have f'(\Sexpr{c}) \Sexpr{fmt(res)} .\end{solution}11
Dynamic exercises: .RnwExample: What is the derivative of f (x) x a e b·x , evaluated at x c?\begin{solution}Using the product rule for f(x) g(x) \cdot h(x) , where g(x) : x {\Sexpr{a}} and h(x) : e {\Sexpr{b}x} , we obtain\begin{eqnarray*}f'(x) & & [g(x) \cdot h(x)]' g'(x) \cdot h(x) g(x) \cdot h'(x) \\& & \Sexpr{a} x {\Sexpr{a} - 1} \cdot e {\Sexpr{b}x} .\end{eqnarray*}Evaluated at x \Sexpr{c} , the answer is\[ e {\Sexpr{b}\cdot \Sexpr{c}} \cdot \Sexpr{c} \Sexpr{a-1} \cdot(\Sexpr{a} \Sexpr{b}\cdot \Sexpr{c}) \Sexpr{fmt(res, 6)}. \]Thus, rounded to two digits we have f'(\Sexpr{c}) \Sexpr{fmt(res)} res)}}\exname{derivative exp}\extol{0.01}11
Dynamic exercises: Single choiceextype: schoiceexsolution: 01012
Dynamic exercises: Single choiceQuestionWhat is the seat of the federal authorities in Switzerland (i.e., the de facto capital)?(a) Bern(b) Lausanne(c) Zurich(d) St. Gallen(e) BaselKnowledge quiz: Shuffled distractors.extype: schoiceexsolution: 01012
Dynamic exercises: Single choiceQuestionWhat is the derivative of f (x) x 3 e 3.3x , evaluated at x 0.85?(a) 45.97(b) 35.82(c) 56.45(d) 69.32(e) 39.31extype: schoiceexsolution: 010Numeric exercises: Distractors are random numbers and/or typical arithmetic mistakes.12
Dynamic exercises: Multiple choiceextype: mchoiceexsolution: 01113
Dynamic exercises: Multiple choiceQuestionWhich of these institutions already hosted a useR! or eRum conference?(a) Agrocampus Ouest(b) Universität Wien(c) ETH Zürich(d) Technische Universität Dortmund(e) Uniwersytet Ekonomiczny w PoznaniuKnowledge quiz: Shuffled true/false statements.extype: mchoiceexsolution: 01113
Dynamic exercises: Multiple choiceextype: mchoiceexsolution: 011 70 65 60 55 50 45 40 35QuestionIn the following figure the distributions of a variable given by two samples (A andB) are represented by parallel boxplots. Which of the following statements arecorrect? (Comment: The statements are either about correct or clearly wrong.)AB(a) The location of both distributions is about the same.(b) Both distributions contain no outliers.(c) The spread in sample A is clearly bigger than in B.(d) The skewness of both samples is similar.Interpretations:Statements that are approxi(e) Distribution B is about symmetric.mately correct or clearly wrong.13
Dynamic exercises: Numericextype: numexsolution: 123.4514
Dynamic exercises: NumericQuestionGiven the following information:extype: numexsolution: 123.45 585 144 177Compute: ?Numeric exercises: Solving arithmetic problems.14
Dynamic exercises: Stringextype: stringexsolution: ANSWER15
Dynamic exercises: StringQuestionWhat is the name of the R function for Poisson regression?Knowledge quiz: Sample a word/phrase froma given vocabulary or list of question/answerpairs.extype: stringexsolution: ANSWER15
Dynamic exercises: Clozeextype: clozeexclozetype: schoice numexsolution: 10 123.4516
Dynamic exercises: ClozeQuestionUsing the data provided in regression.csv estimate a linear regression of y onx and answer the following questions.(a) x and y are not significantly correlated / y increases significantly with x / ydecreases significantly with x(b) Estimated slope with respect to x:Exercises with sub tasks: Several questions basedon same problem setting.extype: clozeexclozetype: schoice numexsolution: 10 123.4516
One-for-All
One-for-all The same exercise can beexported into different formats. Multiple standalone documentsvs. combined exercise pool. Multiple-choice and single-choicesupported in all output formats.18
One-for-AllIdea: An exam is simply a list of exercise templates.R myexam - list( "deriv2.Rnw", "fruit2.Rnw", c("ttest.Rnw", "boxplots.Rnw") )Draw random exams: First randomly select one exercise from each list element. Generate random numbers/input for each selected exercise. Combine all exercises in output file(s) (PDF, HTML, . . . ).19
One-for-AllWritten exam:R exams2nops(myexam, n 3, dir odir, language "hu", institution "eRum 2018")Online test:R exams2moodle(myexam, n 10, dir odir)Live quiz:R exams2arsnova(myexam, n 1, dir odir)Other: exams2pdf(), exams2html(), exams2qti12(), exams2qti21(), . . .20
Written Exams
Written ExamsFlexible: Roll your own. Combination with user-specified template in exams2pdf() andexams2pandoc(). Customizable but typically has to be evaluated “by hand”.Standardized: “NOPS” format. exams2nops() intended for single- and multiple-choice questions. Can be scanned and evaluated automatically within R. Limited support for open-ended questions that have to be marked by a person.22
Written examseRum 2018Exam 2018-05-11Exam: 1805110000111. What is the derivative of f (x) x 7 e 3.7x , evaluated at x 0.83?(a) 49.35Regisztrációs számSzemélyes adatok(b) 87.17Vezetéknév:(c) 71.00Keresztnév:Aláírás:EllenőrizveEz a blokk adminisztrációs célokat szolgál,kérjük ide ne írjon, tartalmát ne módosítsa!TípusDokumentum ID00518051100001A válaszát jelölje egyértelmű x-el: @@ Jelöletlen cella:Variáns0 000112233445566778899Válaszok 1 - 3bcde(e) 55.20 282 137 106Compute:vagyA vizsgalap szkennelése automatikusan történik, ezért kérjük, hogy ne hajtsa össze és a kitöltéshez használjonkék vagy fekete tollat.Kizárólag az egyértelműen és pontosan megjelölt válaszok kerülnek feldolgozásra!a(d) 72.462. Given the following information: ?(a) 106(b) 313(c) 161(d) 232(e) 454123. The waiting time (in minutes) at the cashier of two supermarket chains with different cashiersystems is compared. The following statistical test was performed:3abcdeTwo Sample t-testdata: Waiting by Supermarkett -3.3, df 90, p-value 1alternative hypothesis: true difference in means is greater than 095 percent confidence interval:-3.227Infsample estimates:mean in group Sparag mean in group Consumo4.0456.192Which of the following statements are correct? (Significance level 5%)(a) The absolute value of the test statistic is larger than 1.96.(b) A one-sided alternative was tested.(c) The p value is larger than 0.05.(d) The test shows that the waiting time is longer at Sparag than at Consumo.(e) The test shows that the waiting time is shorter at Sparag than at Consumo.23
Written exams1. Create As illustrated above. Using exams2nops(), create (individual)PDF files for each examinee.24
Written exams1. Create As illustrated above. Using exams2nops(), create (individual)PDF files for each examinee.2. Print Print the PDF exams, e.g., on a standardprinter. . . . or for large exams at a print shop.24
Written exams3. Exam Conduct the exam as usual. Collect the completed exams sheets.25
Written exams4. Scan Scan exam sheets, e.g., on a photocopier. Using nops scan(), process the scannedexam sheets to machine-readable content.26
Written exams4. Scan Scan exam sheets, e.g., on a photocopier. Using nops scan(), process the scannedexam sheets to machine-readable content.5. Evaluate Using nops eval(), evaluate the exam toobtain marks, points, etc. and individualHTML reports for each examinee. Required files: Correct answers (1.), scans(4.), and a participant list in CSV format.26
Written exams27
E-Learning
E-Learning1. Goal Online tests with flexible exercise types. Possibly: Dynamic supplements and/orcomplete correct solution. Random variations of similar exercises toreduce the risk of cheating. Use university’s learning managementsystem, e.g., Moodle, . . .29
E-Learning2. Create Draw random replications from exercisetemplates, e.g., via exams2moodle(), . . . Automatically embed these into exchangefile format (typically via HTML/XML).30
E-Learning2. Create Draw random replications from exercisetemplates, e.g., via exams2moodle(), . . . Automatically embed these into exchangefile format (typically via HTML/XML).3. Import Import in learning management system. From there handling “as usual” in thesystem.30
E-Learning: Online test31
E-Learning: Online test32
E-Learning: Live quiz33
What Else?
What else?Under development: Many volunteers: Internationalization for “NOPS” exams. Nikolaus Umlauf: Exercise “stress tester”. Stefan Coors, Nikolaus Umlauf: Graphical exams manager based on shiny thatcan be used on a local machine or on a server. Achim Zeileis: Reports for lecturers based on IRT models. Niels Smits: Better management of exercise categories. Mirko Birbaumer, Andreas Melillo, Achim Zeileis: Ilias interface based on QTI 1.2.35
Données personnelles5Nom de famille:In this section no changes or modificationsmust be made!Prénom:TypeExam IDInternationalizationR University12518051100001Signature: Exam 2018-05-11Numéro de matricule5660707181829293daJensen, MessnercontrôléThis document is scanned automatically. Please keepclean and dofold. For filling in the document4 not bend or4deZeileisplease use a blue orblackpen.Regisztrációs számSzemélyes adatok5 processed!5Only clearly marked and positionally accurate crosses will beVezetéknév:enZeileisCe champne1doitAnswers- 15pas être modifié.Answers 16 - 21 66esKogelnikabcdeabcd 0e077Keresztnév:161TypeNuméro d’examenfiNordhausen1818212518051100001 17Aláírás:22frAllignol9931833@Merci de cocher soigneusement: @ouNon coché:gswStauffer194EllenőrizveCet examen sera corrigé par un système automatisé.Ne pas plier,à4 corner ni tacher. Merci d’utiliser un stylo 4hrJuraić520billebleu ou noir.55Seul les marques lisibles et bien positionées seront evaluées!huDaróczi, Tótha blokk adminisztrációscélokat 21szolgál,1 - 15Réponses 16 - 2166 Ez Réponses6kérjükne módosítsa!abide nec írjon,d tartalmáteabcdeitZambella 7771Dokumentum ID 16Típus8nlSmits88212518051100001 17999ptCalvão,Dellinger,318vagy10@ Jelöletlen cella:A válaszát jelölje egyértelmű x-el: @Petutschnig (pt-PT/pt-BR)194A vizsgalap szkennelése automatikusan történik, ezért kérjük, hogy ne hajtsa össze és a kitöltéshez használjon115 vagy fekete tollat.20kékroGatuKizárólag az egyértelműen és pontosan megjelölt válaszok kerülnek feldolgozásra!12skFabsic621Válaszok 1 - 15Válaszok 16 - 2113 abcdeabcde7srKecojevic161148trEr21715Please mark the boxes carefully: @@9 a310bcdeNot marked:18or3More contributionswelcome . . .36
Stress testerR s - stresstest exercise("deriv2.Rnw")R plot(s)Histogram of numeric 0Runtimes 0.026 ns37
Stress testerR s - stresstest exercise("deriv2.Rnw")R plot(s)Rank of correct solution200055101015nn152520303525Position of correct solution123Position4512345Rank38
Graphical exams manager39
Graphical exams manager40
Examining examsReport: Exercise difficulty, student performance, unidimensionality, fairness.Methods: Psychometrics, especially item response theory.Example: End-term exam from first-year mathematics course for business andeconomics students at Universität Innsbruck. 729 students (out of 941 registered). 13 single-choice exercises on the basics of analysis, linear algebra, financialmathematics. Two groups with partially different pools of exercise templates.R library("psychotools")R data("MathExam14W", package "psychotools")R mex - subset(MathExam14W, nsolved 0 & nsolved 13)41
it0.0itgeran lag se plic0.0 im10.51.01.50.40.60.8Item difficulty parameters 1.0 0.50.20 trixplanneq squad2.01.0Examining examsItem difficulty: Raw proportions vs. Rasch model.R plot(mex solved, .)R mr - raschmodel(mex solved)R plot(mr, .) 42
Examining examsStudent performance: Points and person-item map.80100R hist(MathExam14W points, .)R piplot(mr)1 lagrange1 60 hesse11 1 equations40planning1 matrix1 payflow1 20annuity1 interest1 integral1 elasticity1 deriv0Frequencyimplicit1 quad 50510Points152025 3 2 10123Latent trait43
Examining examsUnidimensionality: Principal component analysis.R pr - prcomp(mex solved, scale TRUE)R plot(pr, .)R biplot(pr, .) 5015quadpayflowplanning234567Number of components89105integralequationslagrange00.02 5interestelasticitymatriximplicitannuityhesse deriv 0.06 10PC20.00 0.02 0.041100.042.01.51.00.50.0Variances 100.062.5 15 0.04 0.02PC10.0044
Examining examsFairness: Differential item functioning.3R ma - anchortest(solved group, data mex, adjust "single-step")R plot(ma final tests, .) Group 1Group 2quad() ( 1 0 1( (() ( ((itlagrangeesspliche(lagrangeim) )implicit xplanneq inguations() hesse quadplanning) annuitymatrix ) (payflow ) integralinterest ) (elasticity 2Item difficulty parameters2deriv 2 1) ) ) 01Family wise 95% confidence intervals45
RecommendationsIf you want to try: Start with simple exercises before moving to more complex tasks. Focus on content of exercises. Don’t worry about layout/formatting too much. Try to build a team (with lecturers, assistants, etc.). Use exercise types creatively. Don’t be afraid to try stuff, especially in formative assessments. Thorough quality control for dynamic exercises before summative assessments.46
ResourcesSoftware: Zeileis A, Grün B, Leisch F, Umlauf N, Birbaumer M, Ernst D, Keller P, Smits N,Stauffer R (2018). R/exams: Automatic Generation of Exams in R. R package version -exams.org/https://CRAN.R-project.org/package examshttp://R-Forge.R-project.org/forum/?group id msReferences: Zeileis A, Umlauf N, Leisch F (2014). “Flexible Generation of E-Learning Exams in R:Moodle Quizzes, OLAT Assessments, and Beyond.” Journal of Statistical Software,58(1), 1–36. doi:10.18637/jss.v058.i01 Grün B, Zeileis A (2009). “Automatic Generation of Exams in R.” Journal of StatisticalSoftware, 29(10), 1–14. doi:10.18637/jss.v029.i1047
R package exams Type Software Purpose Statistical computing R Random data generation, computations Writing/reporting LATEX, Markdown Text formatting, mathematical notation Reproducible research knitr, rmarkdown, Sweave Dynamically tie everything together Document conversion TtH/TtM, pandoc Conversion to HTML and beyond Image manipulation ImageMagick,magick, png
Bruksanvisning för bilstereo . Bruksanvisning for bilstereo . Instrukcja obsługi samochodowego odtwarzacza stereo . Operating Instructions for Car Stereo . 610-104 . SV . Bruksanvisning i original
10 tips och tricks för att lyckas med ert sap-projekt 20 SAPSANYTT 2/2015 De flesta projektledare känner säkert till Cobb’s paradox. Martin Cobb verkade som CIO för sekretariatet för Treasury Board of Canada 1995 då han ställde frågan
service i Norge och Finland drivs inom ramen för ett enskilt företag (NRK. 1 och Yleisradio), fin ns det i Sverige tre: Ett för tv (Sveriges Television , SVT ), ett för radio (Sveriges Radio , SR ) och ett för utbildnings program (Sveriges Utbildningsradio, UR, vilket till följd av sin begränsade storlek inte återfinns bland de 25 största
Hotell För hotell anges de tre klasserna A/B, C och D. Det betyder att den "normala" standarden C är acceptabel men att motiven för en högre standard är starka. Ljudklass C motsvarar de tidigare normkraven för hotell, ljudklass A/B motsvarar kraven för moderna hotell med hög standard och ljudklass D kan användas vid
LÄS NOGGRANT FÖLJANDE VILLKOR FÖR APPLE DEVELOPER PROGRAM LICENCE . Apple Developer Program License Agreement Syfte Du vill använda Apple-mjukvara (enligt definitionen nedan) för att utveckla en eller flera Applikationer (enligt definitionen nedan) för Apple-märkta produkter. . Applikationer som utvecklas för iOS-produkter, Apple .
och krav. Maskinerna skriver ut upp till fyra tum breda etiketter med direkt termoteknik och termotransferteknik och är lämpliga för en lång rad användningsområden på vertikala marknader. TD-seriens professionella etikettskrivare för . skrivbordet. Brothers nya avancerade 4-tums etikettskrivare för skrivbordet är effektiva och enkla att
Den kanadensiska språkvetaren Jim Cummins har visat i sin forskning från år 1979 att det kan ta 1 till 3 år för att lära sig ett vardagsspråk och mellan 5 till 7 år för att behärska ett akademiskt språk.4 Han införde två begrepp för att beskriva elevernas språkliga kompetens: BI
**Godkänd av MAN för upp till 120 000 km och Mercedes Benz, Volvo och Renault för upp till 100 000 km i enlighet med deras specifikationer. Faktiskt oljebyte beror på motortyp, körförhållanden, servicehistorik, OBD och bränslekvalitet. Se alltid tillverkarens instruktionsbok. Art.Nr. 159CAC Art.Nr. 159CAA Art.Nr. 159CAB Art.Nr. 217B1B
