Evaluation Of MTM’s Signs Of Safety Pilots Appendices

2y ago
28 Views
2 Downloads
1.81 MB
63 Pages
Last View : 1d ago
Last Download : 2m ago
Upload by : Ronan Orellana
Transcription

Evaluation of MTM’sSigns of SafetyPilotsEvaluation report appendicesOctober 2020Mary Baginsky, Ben Hickman, JessHarris, Jill Manthorpe, MichaelSanders, Aoife O’Higgins, EvaSchoenwald and Vicky Clayton

ContentsAppendix 1: Project theory of change4Appendix 2: Evaluation’s logic model6Appendix 3: Self-profiling instrument7Appendix 4: Progress on implementation of Signs of Safety9Appendix 5: Contrast study10Appendix 6: Views of senior leaders in the 2 pilots that exited the SofS project 12Appendix 7: Profiles14Appendix 8: Focus group profile data summary table19Appendix 9: Outcomes analysis21Research objectives21Methods21Selecting comparison groups21Identifying outcome data22Analysis methodology23Changes to evaluation methods24Key challenges24Outcomes24Reduce risk for children and young people: Children in need24Reduce risk for children and young people: Referrals and re-referrals25Reduce risk for children and young people: Child protection plans27Reduce days spent in state care30Increase staff wellbeing31Reduce staff turnover and agency rates31Summary tables32Appendix 10: Difference-in-differences analysisMethods3939Regression specification39Defining pre- and post-treatment40Sensitivity analysis40Secondary analysis412

Main analysis42Matching results42Summary statistics43Analysis results47Conclusion58Appendix 11: Cost study60Methodology60Staffing and training costs60Other direct costs60Indirect costs61Management input for SofS implementation61Ongoing costs613

Appendix 1: Project theory of changeFigure A1.1: Signs of Safety Practice Theory of Change4

Figure A1.2: Signs of Safety Organisational Theory of Change5

Appendix 2: Evaluation’s logic modelTable A2.1: Evaluation’s logic s10 localauthorities inEngland9 had beeninvolved in EIP1and 1 joinedin EIP21 that hadbeen involvedin EIP 1subsequentlyleft the projectOfsted ratingsAssessmentand planningwith the familyBuilding anetwork offamily andsocial supportsUsing theassessmentand planningframework tofocus onanalysis andbuilding thecentral placeof the family s ofassessmentand analysistogether withaction toachieve thegoals of thecase, asassessed bythe family, itsnetwork pportarrangementsfor practiceTrainingSupervisionGroupsupervisionSafety plans:detail actions by the familyand the network members inthe face of identified dangersand identified triggers (redflags) for those dangersinclude contingency actionsshould the planned actionsbe impededthat are written out in detailfor adultsthat are set out in a Wordsand Pictures for the children(and the adults).Case work that should occurin conjunction with a safetyplan:all families beingencouraged, supported andexpected to bring a networkinto the case workmapping* with the family andnetwork including use of theanalysis categories withdanger statements andsafety goals agreed with thefamilyThree Houses work with thechild(ren) and this beingshared with the parentsWords and Picturesexplanation for the child(ren)being prepared with thefamily and this being sharedby the family with thechildren and the network.*Mapping involves:What is worrying (past hurt,danger statement andcomplicating factors)What is working well(existing strengths andexisting safety)What needs to happen(safety goals, friends andfamily safety network,managing safety plan,trajectory, bottom line andscaling)Basic trainingAdvanced trainingCoaching for practiceleadersSupporting thetransition of basictraining in-houseDeveloping specificareas of practice andstaff ntOrganisationalalignment of processesand systems, andmeaningful measures(aligning qualityassurance andinformationmanagement)Families and children feelmore empowered, arebetter able to understandchildren’s services’concerns and requirementsand so are better able toaddress the concerns formore effective outcomesand reduced re-referralsPractitioners report greaterclarity, job satisfaction andcommitment leading toimproved staff retentionand reduced absenteeismThe number of childrenremoved from familiesreduces as the number offamilies being supportedintensively increases,including greaterconfidence to close cases6

Appendix 3: Self-profiling instrumentSelf-profiling instrument: Pilot (Please add authority’sname)Please indicate (with a tick or cross) where you think your authority is in relation to eachof these items:a) at the start of the Signs of Safety projectb) at the end of the Signs of Safety project.The information you return will be held in strictest confidence and will be used as one ofthe many factors contributing to the evaluation of the programme. Not all of the itemswill be exclusively linked to Signs of Safety and it would be helpful if you would placean asterisk next to items that are directly linked with other initiatives in addition toSigns of Safety.Please rate each on a scale of 1–10 where 1 Not at all and 10 FullyComponents of Signs of SafetyOrganisational cultureProgress along the path towards building constructive working relationships between professionalsand family members?Creating a collaborative culture with parents around child protection practice, whilst remaining vigilantand realistic about risk.Providing an effective early help offer to allow intervention at the right timeSpending the necessary time with adults in familiesSpending the necessary time with children in familiesConfidence that your service is intervening at right timeCreating a culture of practice led evidenceCreating a culture where it is permissible to admit you may have it wrongEmbedding an organisational commitment toSigns of SafetyFostering a safe organisation - building confidence that workers will be supported through anxiety,contention and crisesPractice issuesUsing plain language that can be readily understood by familiesCapturing the voice of the child in safeguarding practice and direct workUsing tools to engage children and young people e.g. three houses; fairies and wizardsUsing Words and Pictures explanationsUsing statements focusing on specific observable behavioursSeparating fact from hearsayMapping cases by individual social workersMapping cases in teamsManaging safety plans over time linked with progressionUsing safety plans across initial and review child protection conferences and in all related groupsUsing safety plans developed with familiesDevelopment of appreciative inquiry work with familiesDeveloping family networks from the outsetPlacing Family Network Meetings at the centre of all processesAllowing families to run Family Network Meetings7

LearningBasic training in Signs of Safety is provided in-house as part of compulsory introductory trainingDevelop in house training team to deliver SOFS basic trainingProviding basic 2-day training for all social workersProviding basic 2-day training for all other social care staff working with childrenProviding advanced 5-day training for all managersProviding advanced 5-day training for all social workersEmbedding Signs of Safety approaches and principles across all training for those working inchildren’s social careProcessesAligning paperwork with Signs of Safety practiceAligning Initial Child Protection Conferences with Signs of SafetyAligning Review Child Protection Conferences with Signs of SafetyAchieving consistency in the quality of social work decision making and practiceRevise, negotiate and implement changes to Public Law Outline (PLO) policy, procedure and practiceto fit with Signs of SafetyAligning quality assurance and audit processes with Signs of SafetyAligning IT systems with Signs of SafetyStructuresAchieving manageable caseloads for all practitionersSupporting social workers with administrative tasksRecruiting high quality staffRetaining high quality staffEstablishing practice leadership and supervision processes to support Signs of SafetyCreating a culture of appreciative inquiry across staff interactionsLeadership and ‘staying the journey’Embedding Signs of Safety as the organising framework for all child protection practicePlanning for expected difficultiesPlanning for unexpected difficultiesHaving a vision which is shared and which is sustainable even if key people leaveEmbedding an organisational commitment toSigns of Safety8

Appendix 4: Progress on implementation of Signs ofSafetyIndividuals in focus groups assessed each item on a scale of 1–10 where 1 not at alland 10 completely Plans remain central to practice from assessment through to case closure Plans are regularly reviewed and revised Having sufficient time to spend with families Establishing naturally connected support networks with families Informing, involving and listening to children Clear commitment to Signs of Safety from management It feels like a safe organisation in which to work Family Court engaged in Signs of Safety approach Recording systems aligned with Signs of Safety Quality assurance systems aligned with Signs of Safety Group supervision and appreciative inquiry in place Partner agency involvement9

Appendix 5: Contrast studyThe study was located in one team in 4 authorities – 2 SofS pilots (Pilots 4 and 9) and 2non-SofS authorities. Of the 2 non-SofS sites one had adopted a restorative practiceapproach across children’s services and the other had developed an approach toworking with families based on striving for positive change through a number of routesincluding motivational interviewing and multi-disciplinary teams with specialistpractitioners on domestic abuse, parental mental health and alcohol/substance misuse.In Pilot 9 and in the 2 contrast sites the work took place at approximately 6-monthintervals (T1, T2 and T3). Pilot 4 did not enter the contrast study until summer 2019when it became evident that the work could not continue in Pilot 7, so the fieldwork wasconducted between August 2019 and February 2020.One team had experienced a change of manager and there was considerable staffturnover and staff shortages in 2 others, although all 4 teams were working underconsiderable pressure. At one point in one contrast site, a team that should haveconsisted of a manager and 6 social workers plus a drug worker and domestic violenceworker consisted of a manager, one part-time experienced social worker, a one-yearpost-qualification social worker and another social worker who was in the assessed andsupported year, plus the domestic violence worker.Given the limitations of resources devoted to the evaluation that had to be spreadacross the 4 areas at 3 time points there was a heavy reliance on social workers’ cooperation which, for the most part, was extremely good. Nonetheless, it is important torecognise that, for some social workers, however much it was stressed that individualswould not be judged, that was how it would have been perceived. In a few casesparents did not agree to participate but it was evident that in some teams the biggerchallenge was the reluctance on the part of social workers to suggest participation.Understandably some cases were excluded because they were at a particularlychallenging point or because a social worker said a family would never agree or that theresearcher’s presence would change their relationship with the family. Overall, as far asit was possible to judge, a high proportion of families who were approached agreed totake part and there was a high level of engagement in providing feedback.All observations were conducted by the same evaluator. While this brought consistency,if there were biases there was no one else to provide challenge. While interesting datawere collected it must primarily be viewed as a trial for how to conduct a subsequentstudy as many lessons were learnt, not least the benefits that would attach toembedding researchers in teams for a period of time while the work was concluded.10

The instruments used during the contrast study were:Organisational social context tool: The Organizational Social Context (OSC)Measure is a nationally normed (for use in 2 settings: child welfare and mental health)and psychometrically proven 105-item scale that measures the cultures and climates ofchild welfare and mental health organisations. It can be administered online or usingpaper scan forms. The OSC Measure is completed by front-line staff (rather thanmanagers or leaders) to obtain the most representative view of an organisation’s cultureand climate. Reliability coefficients for OSC dimensions range from .78 to .94.Clinical Competence-Based Behavioral Checklist (CCBC): The CCBC is a tool forassessing performance in social work practice that consists of 4 categories: (1)interviewing skills; (2) cultural empathy; (3) assessment and intervention strategies; and(4) comprehensive evaluation (see Lu et al., 2011). To increase the reliability of themeasure, an ‘overall score’ category was added. This assesses a broader set of skillsthan individual categories. Regehr et al.(1999) have reported that the scores for anoverall assessment category are at least as reliable as the scores for individualcategories and in some cases more valid.Families and social workersWorking Alliance Inventory (WAI) WAI – short version for SWs WAI – short version for familiesSee https://wai.profhorvath.comFamiliesHampshire County Council Children's Services Family FeedbackFamily Chart developed at Round 1 to collect feedback on children’s social careFamily Feedback Scaling ChartPractice Elements of Signs of Safety ChartClient Engagement in Child Protection Services (Yatchmenoff)Social workersSurvey for attached social workers in SofS sites11

Appendix 6: Views of senior leaders in the 2 pilots thatexited the SofS projectSenior leaders in both pilots had joined the authorities after Round 1. The children’sservice department in the pilot that did not take part in Round 2 had been judgedinadequate by Ofsted in April 2017 but by June 2019 was found to have improvedsubstantially on all aspects and was then judged to be ‘good’. The pilot that left theproject half-way through Round 2 had been found to require improvement at the end ofRound 1 but by June 2018 it received an ‘inadequate’ judgement. The new seniormanagement team did not consider that SofS, in the form that was advocated duringEIP, fitted with the improvement journey the authority was following.Both inspection reports that led to the ‘inadequate’ judgements had highlighted the factthat children were being left at ‘risk of significant harm’. While there were severalcontributory factors in both departments, the question of how SofS dealt with risk was atthe heart of the decision to move away from it.The discussion in both former pilots was very similar. SofS was seen to have become asubstitute for basic social work skills and the ‘3 columns’ 1 was seen as the assessmentrather than a tool to aid analysis. It was being used superficially and as a result wasleading to risky practice because it encouraged an overly strengths-based approachwith a tendency to minimise risk. There was an over-reliance on parental self-reportingand the production of plans was based on what families said they would do rather thanon an understanding of what was required, what was needed to achieve change and anassessment of whether that change was sustainable:There is a basic flaw with it, which is that you rely on people who may not be ableto do the right thing, to do the right thing and in so doing it minimises professionaljudgement.The approach facilitated the ‘rule of optimism’ which, in the (authority’s) contextwas at best superficial and at worst meant risk was missed.It was said to have been used as a way of processing cases more quickly to deal withthe demand and to have been imposed from above, depending on a small number ofPart of Signs of Safety assessment and planning framework that identifies what children’s services areworried about (past harm, future danger, complicating factors), what is working well (existing strengthsand existing safety) and what needs to happen (family and child protection authority safety goals and nextsteps for future safety)112

advocates, rather than implemented from the bottom, taking staff along and listening totheir response.Alongside these concerns was one focusing on the electronic recording systems thathad evolved to reflect SofS but which, in the opinion of these informants, did not reflectthe statutory framework within which work must take place. It was reported to fail toreflect the basic statutory functions in terms of information sharing, chronology, historyas an indicator for the future, and arriving at clear conclusions about risk and strengththat are based on balance.Restorative practice had been introduced into both former pilots. One informantsummed up the reasons for adopting this approach:Restorative, from my point of view, provides an overarching umbrella aroundsome ways of being; it's more about language and approach. It helps withinterventions, it helps with behaviours and culture, from my point of view. What itgives people, I think, is an ability in equal measure to support and challenge;very, very simple and what it does, I think, is provide a focus on expectation andhelp. So where it's really helpful from a safeguarding point of view is when you'reapproaching a family in need of help, as most of our families are, but where youalso have to shift up a gear quite quickly, it helps staff to think about how we dothis without alienating and causing conflict And it easily over-layers onto thestatutory framework.13

Appendix 7: ProfilesThe profiling exercise asked strategic leads in the 9 pilots to rate their progress (1–10)at the beginning (T1) and the end (T3) of Round 2 on 50 items organised into 5categories: organisational culture; practice issues; learning; processes and leadership.All pilot scores were converted to T-scores based on the mean average response of allpilots. A score of 50 represents the mean and a difference of 10 from the meanindicated a difference of one standard deviation.Pilot 1 had slightly higher than average scores for culture and practice at T1, withaverage scores for the remaining categories. At T2 its scores had decreased slightly inboth culture and practice, although they remained just above average, and hadimproved in learning.Figure A7.1: Pilot 1 T-Scores for profile of SofS progress at T1 and T314

At T1, Pilot 2 was slightly below average in each of the categories other than practice(where it was just above average). By T2, its scores had decreased in every categoryand were one standard deviation below the average in culture, processes andleadership, and 2 standard deviations below in learning.Figure A7.2: Pilot 2 T-Scores for profile of SofS progress at T1 and T3Pilot 3 was below average in all of the categories at T1 and was one standard deviationbelow in culture and processes. By T2 it had improved in each of the categories butremained below average in all.Figure A7.3: Pilot 3 T-Scores for profile of SofS progress at T1 and T315

At T1, Pilot 4 was below average in learning and leadership and above average inprocesses. It had improved in each of the categories other than processes by T2, tobecome slightly above average in all categories.Figure A7.4: Pilot 4 T-Scores for profile of SofS progress at T1 and T3Pilot 5 was above average in all categories at T1, and was one standard deviationabove in learning, processes and leadership. It remained above average in allcategories at T2, although only processes remained one standard deviation above theaverage.Figure A7.5: Pilot 5 T-Scores for profile of SofS progress at T1 and T316

Pilot 6 was slightly below average in all categories other than learning at T1. By T2 ithad improved in every category to become slightly above average in all categories otherthan processes, where it remained just below ave

Changes to evaluation methods 24 Key challenges 24 Outcomes 24 Reduce risk for children and young people: Children in need 24 Reduce risk for children and young people: Referrals and re-referrals 25 Reduce risk for children and young people: Child protection plans 27 Reduce days spent in state care 30 Increase staff wellbeing 31 Reduce staff turnover and agency rates 31 Summary tables 32 .

Related Documents:

Dimensions, including handle/bumpers (W x H x D): ONT-601 MTM 481 x 41 x 385 mm Weight: ONT-601 MTM approx. 5.5 kg Instrument Operation Remote GUI operation via LAN Remote control for test automation via LAN Local GUI via integrated touchscreen (optional for ONT-603/606) Optical Test Interface The MTM uses pluggable XFP optics.

5 Figure 1. Asian LICs and EMEs: MTM Gains/Losses III. HOW SIGNIFICANT WAS THE IMPACT OF THE CRISIS THROUGH EXPOSURE TO MARKET RISKS? 7. LIC banks continued their strong mark-to-market (MTM) gains despite the recent global financial crisis.2 However, EME banks registered a modest degree of MTM 2 For the purpose of this paper, MTM gains are defined as a sum of (i) realized gains on securities .

2 Construction Signs, & Parking Signs 3 Resident Parking Signs 4-8 Regulatory Signs 9-10 Street Cleaning Signs 11 Street Cleaning/Snow Emergency Signs 12-14 Tow Zone Signs 15-17 Warning Signs 18 Guide signs & Street Name Signs December 2013 NOTES: 1. This document is a guide and provides recommended minimum sign sizes.

Signs Chapter 2 Page 2-2 WSDOT Traffic Manual M 51-02.05 April 2011 Chapter 2L Changeable Message Signs Chapter 2M Recreational and Cultural Interest Signs Chapter 2N Emergency Management Signs Part 6 Work Zone Signs Part 7 School Area Signs Part 8 Railroad and Light Rail Signs Part 9 Bicycle Facility Signs 2.2 Sign Design

describes six channels of the MTM. Section IV provides a review of the empirical literature of MTM in the EAC. Section V describes the various VAR methodologies. Section VI describes the data. Section VII presents the empirical results, including an evaluation of relative strengths and weaknesses of various channels of the MTM in each EAC country.

monitoring measures CMS will use to assess the Enhanced MTM model. Although monitoring . safety, patient population, and other demographic and clinical factors. Subject to CMS approval, sponsors can also provide various incentives , based on each beneficiary’s risk level and individual . presentelements that capture enhanced MTM services .

4. Where a RSW is seeking competency levels 2 and 3 then specific MTM “domain knowledge” such as the following are obtained. a. MTM signal principles b. MTM signal standards, c. train dynamics, d. headways e. line speeds f. curves and gradients knowledge, g. Pass Assets h.

When students complete an MTM, the general education courses in the "Core Transfer Map" portion of the MTM, for which minimum required grades have been earned, are guaranteed to transfer into . The Biology MTM group will work with the AP/IB Policy Workgroup to resolve any outstanding AP/IB issues by December 31, 2020. 6. Please note that .