Kirkpatrick’s Level 3 Evaluation - Boston University

Pushing for Measurement of Impact: NEPHTC’s IncreasingCapability to Measure Training Impact on the WorkforcePerformanceKirkpatrick’s Level 3 EvaluationJuly 12, 201811:30 am

AcknowledgementsThis training and all supporting material was supported by funds made availablefrom the Centers for Disease Control and Prevention, Office for State, Tribal, Localand Territorial Support, under B01OT009024. Additionally, this training wassupported by the Grant Number, 5U90TP116997-10, funded by the Centers forDisease Control and Prevention and the Assistant Secretary for Preparedness andResponse. This project is also supported by the Health Resources and ServicesAdministration (HRSA) of the U.S. Department of Health and Human Services (HHS)under grant number UB6HP27877.The views and opinions expressed as part of the training and all related documentsand course materials are those of the authors and do not necessarily represent theviews and opinions or the official position of, or endorsement by, the Centers forDisease Control and Prevention, its Public Health Emergency PreparednessProgram, the Office of Assistant Secretary for Preparedness and Response (ASPR)Hospital Preparedness Program, or that of HRSA, HHS or the U.S. Government.

“Managing Effectively in Today’s Public Health Environment”NEPHTC’S MANAGEMENTPROGRAM

NEPHTCThe New England Public Health Training Centerdevelops trainings and provides services thatbuild public health workforce ephtc.org/Please visit our website and learning management system!

Management Program - A flagshipfor NEPHTCCohorts run in MA and ME,downloadable materials2017 Impact Award NNPHIRelationship to PH 3.0

Topics and Alignment with PH 3.0Management Training Course TopicPublic Health 3.0 Key ComponentsFrom Manager to LeaderLeading Organizational ChangeRecruiting and HiringOnboarding and CoachingLeadership & WorkforceLabor Law & Collective BargainingGrievances & DisciplineBudgeting & Resources 1Budgeting & Resources 2Flexible & Sustainable FundingGrant WritingManaging Projects & TeamsProgram EvaluationData, Analytics & MetricsQuality ImprovementCommunity CollaborationStrategic PartnershipsGetting Started (Orientation to PH)Setting the StageMarketing Public HealthEssential Infrastructure

Adaptable for Any Health Department

Adaptable for Any Health DepartmentLong, detailed vrs short, fewer topicsMaterials and customization tips

EVALUATION

The Kirkpatrick Model Level 1: Reaction post-training Likert scale ratings of aspects of training to assess satisfaction,engagement, and relevancemid-course survey for QI Level 2: Learning pre/post-training multiple choice quizzesLikert scale ratings to assess trainee perceptions about their ability to perform traininglearning objectives before and after training Level 3: BehaviorTrainees apply what they learned when back on the job Level 4: ResultsTargeted outcomes or impact occur as result of the training10

Level 1 and Level 2Level 3 - Pilot

Level Three EvaluationFull Level 3 Evaluation:12

Example: Level Three Survey

Level Three Behavior Change Results

Level Three Behavior Change Results

Level Three Behavior Change Results

Level Three Quotes “This course has influenced me to pay more attention to how I work as amanager.” “The management course was great in the fact that I take a step back andassess overall issues, situations, problems. [I] deal with the issue at handand look for ways to solve [them] amicably. “After being in my role for a number of years, it [the course] helped merefocus and helped me with developing a plan to move my departmentforward. We are conducting a community health assessment now, haveused the onboarding materials and will be doing some marketing of thedepartment.” “The course also provided an opportunity to understand how the rest ofthe Commission works. I hope to have more opportunity for the bureaus towork together.”

Lessons Learned for BehaviorChange EvaluationRound 1:– Do a pilot to gather themesRound 2:– Develop survey based on themes, courseobjectives, and session content– Conduct survey and use Level 1-3 results for QIRound 3:– Refine, refine, refine based on evaluation results

Contact Information Seth Eckhouse – Karla Todd Hope Kenefick -seckhous@bu.edutoddks@bu.eduhopewk@comcast.net Kirkpatrick Training Evaluation Model available at: NEPHTC Managing Effectively in Today's Public Health -Philosophy/The-New-World-Kirkpatrick-Modelfor Trainers - https://tinyurl.com/TtT-MGMT

Kirkpatrick’s Level 3 Evaluation. Acknowledgements This training and all supporting material was supported by funds made available from the Centers for Disease Control and Prevention, Office for State, Tribal, Local and Territorial Support, under B01OT009024. Additionally, this training was