Whilst outcomes frameworks exist in many areas of children’s social care, some organisations require a bespoke solution to assess, analyse or the impact of their services.
For the National Youth Advocacy Service, we developed and implemented a bespoke outcomes framework, that would reliably evidence the impact of advocacy interventions and support the promotion and protection of children’s rights.
All services in children’s social care must assess and report on the outcomes of service. For most service areas valid and reliable assessment tools exist to contribute to outcome assessment. However, this is not always the case meaning bespoke assessment tools must be developed. This was the case when the Centre for Outcomes of Care were commissioned to produce an outcome framework for the National Youth Advocacy Service (NYAS). They required an evidence-based outcome framework and assessment tool to:
· Demonstrate and evidence the difference NYAS makes to people’s lives
· Provide evidence for improving services
· Keep NYAS relevant in an increasingly competitive marketplace
We have an established evidence led approach to developing outcome frameworks. As with all commissions the process starts with a scoping review of the peer reviewed literature to assess if a validated outcome framework and validated indicators appropriate to the service exist. No suitable outcome framework was found so we were required to co-produce a bespoke outcome framework with NYAS
Our approach followed established research processes and standards in the development of the outcome frameworks. As with all bespoke evidence based frameworks we undertook an evidence review using a systematic approach to identify appropriate constructs to inform the outcomes chosen.
GRAPHIC 5 CIRCLES - BELOW TEXT AS GUIDE
Explore; Evidence review and stakeholder engagement
Define: Development of logic model
Develop: Development of impact measurement tool
Validate: Pilot exercise & statistical analysis
Reveal: Reliable results
The project was broken down in to the following phases:
1. Evidence review to:
a. Define the type of ‘advocacy’ NYAS provides and identify appropriate constructs
b. Methods and models of assessing impact of advocacy to suitable to use as progress indicators
2. Development of logic model based on the service NYAS provides
3. Design of impact measure based on the evidence review and service NYAS provides
4. Piloting and tool validation (3 iterations)
5. Statistical analysis to assess validity and reliability of results
In the first 12 months we produced an evidence based framework informed by the values of NYAS and explicit aims for the advocacy service provided. The results of the evidence review and stakeholder engagement underpinned the outcome framework and new assessment process. The outcomes were focused on four core domains: Knowledge, Participation, Empowerment and Support. Central to the framework were two separate bespoke assessment measures (under 11 and over 11) to assess progress. The measures were designed for assessment at the beginning (pre) and after (post) advocacy on three of the domains.
The measure was subject to validity testing, piloting and 3 iterations and extensive analysis by our statisticians over a 12 month period. The pilot phase consisted of nearly 1000 children and young people giving suffice sample size for rigorous analysis. This results confirmed validity and internal consistency correctly identified 3 factors that collectively explained over 65% of the variance Pre and Post intervention. The results confirmed the NYAS measure to be valid and reliable to assess the impact of advocacy.
The reliability of the tool now allows NYAS to demonstrate the impact of its service with high confidence and hard data in support. As well as helping advocates assess progress as a service NYAS are able to collect valuable data to contribute to our understanding of what works for children and young people. Following the national role out of the tool NYAS were able to evidence that children were overwhelmingly happy about the service they had received.