Using Rubrics In Student Affairs: A Direct Assessment Of .
Using Rubrics in Student Affairs:A Direct Assessment of LearningJoseph D. LevyAssistant Director, Assessment Programs
Overview of Session Overview rubricsExplain steps for implementing rubricsHighlight examples of rubrics in practiceDiscuss recommendations and next steps
What is a rubric? Definition: “a set of criteria specifying thecharacteristics of an outcome and the levelsof achievement in each characteristic.” Benefits– Provides consistency in evaluation of behaviorsand performance– Gathers rich data– Mixed-method– Allows for direct measure of learning
Measuring learningDirect MethodsIndirect MethodsAny process employed togather data whichrequires subjects todisplay their knowledge,behavior, or thoughtprocesses.Any process employed togather data which askssubjects to reflect upontheir knowledge,behaviors, or thoughtprocesses.Where on campus would you go orwho would you consult with if youhad questions about which coursesto register for the fall?I can identify where to go on campus if Ihave questions about which courses toregister for in the fall.Strongly agreeModerately agreeNeither agree nor disagreeModerately disagreeStrongly disagree
Points to Ponder:At your institution, how importantor relevant are learning outcomes?What current methods or effortsare you using to measure learning?
Example Rubric
Why use rubrics?1. Provides both qualitative descriptions ofstudent learning and quantitative results2. Clearly communicates expectations to students3. Provides consistency in evaluation4. Simultaneously provides student feedback andprogrammatic feedback5. Allows for timely and detailed feedback6. Promotes colleague collaboration7. Helps us refine practice
Types of rubrics
Holistic Rubrics Ideal when potential dimensions list is short or difficultto differentiate Appropriate for simpler outcome/area of assessment Difficult to decide on score if student is all over the map More subjective scoring
Analytic Rubrics Can provide scores by dimension Necessary when potential dimensions list is long More objective scoring
Analytic rubric exampleUsing Portfolios to Assess the Outcomes of a Leadership Program, presented June 13, 2008 at International Assessment &Retention Conference by Katie Busby, University of Alabama
Rubrics are used for ObservationoooooPresentationsTeam workTrainingsRole playsPerformanceArtifact AnalysisoooooReflection papersPortfoliosJournalsArt piecesResumes
Example data collection opportunities Students on an Alternative Spring Break blog or journal each dayabout their experiences. College 101 students create a portfolio as part of their classwork. Students in a leadership workshop participate in a group exercise. Outdoor adventure club members take photos and video of theirexperience. RA incident reports are used to assess crisis response learning Observations of mock interviews. Watching student presentations about their service learningexperience.
Points to Ponder:In what ways (if any) have youutilized rubrics in the past?What obstacles do you see in tryingto utilize rubrics in your role?
Steps for implementation1234567Identify the outcomeDetermine how you will obtain evidenceDevelop the rubric toolTest rubricTrain evaluators on rubric useCollect dataAnalyze and report
1. IDENTIFY THE OUTCOME
Learningoutcomes Examine what a student (orother stakeholders) is to do orthink as a result of theprogram, course, service.Programoutcomes Examine what a program orprocess is to do, achieve oraccomplish for its ownimprovement; generallyneeds/satisfaction driven.
Good outcome statements Translate intentions into actionsDescribe what participants should demonstrateor produce (as a result of participation studentswill think, act, feel)Use action verbsAlign with other intentions (institutional,departmental)Map to practicesAre collaboratively authoredReflect/complement existing national criteriaAre measureableMaki, P. L. (2004). Assessing for learning: Building a sustainable commitmentacross the institution.
2. DETERMINE HOW YOUWILL OBTAIN EVIDENCE
Collecting evidenceObservationoooooPresentationsTeam workTrainingsRole playsPerformanceArtifact AnalysisoooooReflection papersPortfoliosJournalsArt piecesResumes
3. DEVELOP RUBRIC
See what is out there . Ask colleagues Rubrics already being utilized on campus (e.g.,employee evaluations, process management) Web search Institutions (e.g., Texas A&M, Hawaii at Manoa) Organizations (e.g., AAC&U, ACPA)
Rubric PreparationResearch andReflectionListing andGrouping Reflect on activity, program, oroutcomes Research theory, principles, orunderpinning concepts List learning objectives orexpectations Group objectives and criteriaApplication Transfer information to gridand Develop descriptionsDevelopment
Determine ScaleCompetency:Frequencyof Behavior:Extent toWhichPerformed:Beginner, Developing, Accomplished, AdvancedUnacceptable, Marginal, Proficient, ExemplaryNovice, Intermediate, Proficient, DistinguishedNot yet competent, Partly competent, Competent,SophisticatedNever, Rarely, Occasionally, Often, AlwaysNever, Once, Twice, Three times, Four times Never, 1 – 3 times, 4 – 6 times, 5 – 7 times, .Not at all, Slightly, Moderately, Considerably, A great dealYes, NoMet, Partially met, Not met
Create DimensionsList the traits you want a student to be able todemonstrateDiscuss this list with others to get feedbackShorten the list by including the most significantelements (if needed, recommended no less than 3 andno more than 8)Edit list to make sure each dimension is concise andclear
Write the Descriptions1 Managingfear2 Apprentice3 Proficient4 DistinguishedActively sought tiesprovided challenge,growth, change,innovation, and/orimprovementDemonstrateswillingness to takeriskSuccessfully faces andmanages the fears(of being different,failure)
Creating a Rubric1 ManagingfearDid not seekleadershipopportunities2 Apprentice3 Proficient4 DistinguishedActively sought ed little tono challenge,growth, tunitiesprovided challenge,growth, change,innovation, and/orimprovementDid notdemonstratewillingness totake riskDemonstrateswillingness to takeriskAvoids beingdifferent, failureSuccessfully faces andmanages the fears(of being different,failure)
Creating a Rubric1 Managingfear2 Apprentice3 Proficient4 DistinguishedDid not seekleadershipopportunitiesResponded to leadershipopportunitiespresented to themSought leadershipopportunitiesActively sought ed little tono challenge,growth, change,innovation,and/orimprovementOpportunities providedminimal challenge,growth, change,innovation, and/orimprovementLeadershipopportunitiesprovided somechallenge, growth,change,innovation, and/orimprovementLeadershipopportunitiesprovided challenge,growth, change,innovation, and/orimprovementDid not demonstratewillingness totake riskDemonstratesunderstanding of risktakingDemonstrateswillingness to takesome risksDemonstrateswillingness to takeriskDoes not successfullyface and manage fear(of being different,failure)Faces and manages thefears (of beingdifferent, failure)Successfully faces andmanages the fears(of being different,failure)Avoids beingdifferent, failureUsing Portfolios to Assess the Outcomes of a Leadership Program, presented June 13, 2008 at InternationalAssessment & Retention Conference by Katie Busby, University of Alabama
1 BeginnerOpportunityseeking Did notseekleadershipopportunities Unable toarticulateinterestsChallengingself Opportunities providedlittle to mentRisk-taking Did notdemonstratewillingnessto take riskManagingfear Avoidsbeingdifferent,failure2 Developing3 Accomplished Responded toleadershipopportunitiespresented to them Aware of interests, butis not intentional inseeking relatedopportunities Opportunities providedminimal challenge,growth, change,innovation, and/orimprovement4 Advanced5 Expert Actively sought specificleadershipopportunities Pairs opportunitieswith interests Leadershipopportunities providedchallenge, growth,change, innovation,and/or improvement Demonstratesunderstanding of risktaking Demonstrateswillingness to take risk Does not successfullyface and manage fear(of being different,failure)Successfully faces andmanages the fears (ofbeing different, failure)
Student Employee Learning Outcomes (SELO)OutcomeSourceLevel(The student employee demonstrates sensitivity to differences.A Developmental Approach to Training for Intercultural Sensitivity, Bennett (1986)321Student evaluates events andStudent is able to empathize with aStudent recognizes and acceptssituations in a cultural context.person of a different culture in adifferences in fundamental(Integration)particular situation.cultural values.(Adaptation)(Acceptance)Observable OutcomesBehavioralIndicators Lacks strong culturalidentificationContent with a self-createdidentitySense of self involvesdynamic processComments: Demonstrates an ability toempathizeQuestions about difference areappropriateEager to apply their knowledge ofcultural difference Enjoys recognition and explorationof differenceTolerates ambiguityQuestions about difference may benaïve, but are geared to learning
4. TEST RUBRIC
Testing your rubric Use a Metarubric to review your etarubric.doc) Peer review- ask one of your peers to review the rubricand provide feedback on content Student review-ask a student to do so as well ifappropriate Test with students - use student work or observations totest the rubric Revise as needed Test again Multiple raters – norm with other raters if appropriate
Metarubric: Rubric for a rubric
Points to Ponder:How often do you currently pilot ortest your assessment instruments?What might prevent you frompiloting or testing your rubrics?
5. TRAIN EVALUATORS
Rater SelectionIdentifycharacteristicsof ratersIdentifypossible ratersthat meetneeds Knowledge of domain Complexity ofdimensions/descriptions Ability to be objective Other staff or faculty Student leaders Students/peers
Rubric rater training erDiscussion
Multiple raters and consistencyOverall goals:1. High inter-rater reliability: Between-raterconsistency Affected by:Initial starting point or approach to scaleInterpretation of descriptionsDomain/content knowledgeIntra-rater consistency2. High intra-rater reliability: Within-raterconsistency Affected by:Internal factors: mood, fatigue, attentionExternal factors: order of evidence, time of day, situationsApplies to both multiple-rater and single-rater situations
6. COLLECT DATA
Reviewing DataCompileaggregate datato examinetrends andoverall learningthat occurredDrill down toindividualdata/scores ysis viaSPPS or othersoftwarePrepare datafor sharing orreporting
7. ANALYZE & REPORT
Reporting ResultsDetermine a timeline for sharing your resultsWrite an executive summary of the dat
Assistant Director, Assessment Programs . Overview of Session Overview rubrics Explain steps for implementing rubrics . o Art pieces o Resumes . Example data collection opportunities Students on an Alternative Sprin
_4.Rubrics can be created in a variety of forms and levels of complexity, but all have two features in common: a list of criteria and gradations of quality. _5.Instructional rubrics are easy to use and to explain. _6.Instructional rubrics focus the teacher helping to clarify the criteria and expectations in specific terms.
Teachers are encouraged to modify the enclosed rubrics or create their own, so that grades are not the only form of student assessment. Checkpoint C classroom teachers may prefer to distribute assessment rubrics at the beginning of a unit, so that students are aware of expectations and understand how their .
3 1. Introduction Overview This document contains information regarding the review criteria and related rubrics used to identify evidence-based PK-6 literacy interventions. The rubrics were applied to specific interventions that included one or more
Advanced Grading with Rubrics 3 Create a New Rubric from a Template After clicking Save and display (see page 1), click Create new grading form from a template. To find Rubrics you have used before, check the include my own forms box in the Grading forms search and type in a word that appears in you
Teacher Evaluation Rubrics by Kim Marshall – Revised January 2, 2014 Organization, Rationale, and Suggestions for Implementation 1. The rubrics have six domains covering all aspects of a teacher’s job performance: A. Planning and Preparation for Learning B. Classroom Management C. Deliver
Principal Evaluation Rubrics by Kim Marshall – Revised August 21, 2011 Rationale and suggestions for implementation 1. These rubrics are organized around six domains covering all aspects of a principal’s
such as C/S (comma splice), Frag (fragment). You can also add your own Quickmarks. Building Rubrics with Turnitin. 46. Building Rubrics with Turnitin. 47 Once you’re done reading and typing comments or putting in Quickmarks, then . PowerPoint Presentation Author: Kasparian, Melanie
Transactions, the National Finance Center report that shows total disbursements by appropriations and schedule number, to the general ledger and to the Government-wide Accounting (GWA) Account Statement for each appropriation. Any differences must be resolved in a timely manner. Section 6.0 Time and Attendance . Time and attendance is to be recorded accurately to ensure that the presence and .
