Evaluation

Background

The Project

1.         The project was to set up a Community Intervention Programme in the three priority neighbourhoods of Barne Barton, North Prospect and Stonehouse to help reduce particularly youth related crime by modifying young people’s behaviour. It was hosted by the Tamar View Community Complex and delivered by Brighter Futures, who designed the program for young people. It was meant to be a two year project but in effect is an eighteen month one running to end March 2008, with a total cost including matched funding of £212,861. The individual counselling finished in January but the group work continued to the end of the programme and hopefully beyond.

Objectives

2.         The Bid document describes the Objectives in Section 2.4 as set out below. These have not been changed within the lifetime of the project.

Ø       One to one specialist directive counselling for perpetrators of Crime, Violent Crime, bullying, neighbourhood nuisance, abusive, aggressive and violent behaviour, anti-social behaviour and those excluded from school, to provide opportunities to manage their emotions and modify their offending behaviour. The service will be available to young people from the age of eight to twenty one years old,  both male and female through a referral system with existing service providers and agencies, including schools

Ø       Providing Anger Management Workshops for parents with children who have behavioural issues and provide training for professionals/workers, in the priority areas through partnership working with the referring agencies and organisations, taking a collaborative approach to reduce crime.

Ø       Publicising the service in existing community publications and using the press and media to maximise local knowledge and acknowledge the investment by funders

Ø       Report to existing steering groups of partners and referring agencies to ensure the aims and objectives are being met.

Ø       Work in a holistic and collaborative manner in partnership to find the most appropriate support and positive outcome for victims and perpetrators of crime, including attendance at multi-agency meetings and referring to other agencies where appropriate

Ø       Monitor and evaluate the effectiveness of the service and provide monthly and quarterly reports and an evaluation of the program at the end of the funding period

Ø       The Community Intervention Program will work safely within an ethical framework (RESPECT and BACP Principles), and receive regular professional external supervision.

The Evaluation

3.         The Bid document built in an ongoing evaluation element from almost the very start and with specific points in February, May, September and December 2007 and then a Final report in March 2008.

4.         There were two Interim reports in April and June 2007. The first Interim Report was very much process focussed as there were few outputs to report. It did however reassure that the project was underway and on track and being carried out and managed in an appropriate manner. The second was around a one day conference arranged by Brighter Futures to publicise the project – particularly its need of financial support going forward – and set out both the methodology and the results to date which were very positive. SBD Associates presented the report at the meeting.

5.         The evaluation framework and methodology were agreed with the project principals mainly based on “before” and “after” questionnaires. (Section 4 of this report looks at the Outcomes in detail.) The information for this Client Assessment Tool (CAT1) to be filled in for each participating individual is listed in Table 4 in Section 4. This tool measures the behaviour of the individual in 57 categories both before the counselling and immediately after. We also tried to measure it 3 months after the end of the counselling to see if any beneficial effects lasted. This has proved more problematic but we believe we have robust enough figures to form a tentative conclusion.

6.         Group sessions were introduced in response to demand from the referrers once the programme was underway and the Group participants had to be measured on a different basis given the lighter touch counselling they received. We decided on a simple measure of whether they managed to stay at school or not as all were at risk of being excluded.

7.         The Bid Document (Para 2.5) did anticipate that the individual format might not be appropriate for all and therefore the change was readily agreed.

                                                                                                                               “We would not wish to restrict the format of the work to the above as being flexible to demand is critical to the success of the project.  For example, we already know that this project may be oversubscribed, from the feedback that we have received. Not all clients may require all twelve counselling sessions, therefore we could utilise the counselling hours to enable additional clients to access the service. We would therefore, utilise the time as effectively as possible to deliver either more counselling sessions or more workshops and/or training, where-ever there is greater priority. “

          Outputs

            General

8.         The programme had a basic set of Direct Outputs which it was expected to achieve and a subset to be achieved by March 2007. They related to those affected by the project and were quite straightforward to measure. Because of the late start the end March 2007 targets were not fully met, but we were relaxed about that.

Table 1: Direct Outputs
Output	Plan		Actual
	Mar 07	Mar 08	Mar 07	Mar 08
No of young people given counselling	8	50	2*
No of workshops for parents	4	10	2
No of parents attending workshops	16	50	15
No of workshops for professionals & workers	1	2	1
No of professionals & workers attending Anger Management Training	8	16	22
No of Full Time Staff employed	1.5	1.5	1.5
* 2 individuals completed 12 hours of counselling

9.         What soon became apparent was that the referring agencies were also seeking to use the service to help those who are not so extreme as to need the full 12 hour sessions. Many of these could be handled in group sessions of 3 or 6 hours. The project therefore responded to this by increasing its group activities. Some individuals may be identified from the groups for further one-on-one assistance but given the limited resources it was thought that the individual numbers had to fall.

10.       Given that this was a demand led pilot we felt it entirely reasonable that the format of the project should change to reflect what the professionals on the ground in the referring agencies were saying was needed. We felt that particularly in terms of sustainability if the project demonstrated that it could respond to and meet changing demands it was much more likely to be supported by the agencies after this first round of funding ended.

11.       We therefore recommended that the formal targets were modified to reflect this change in working and that was accepted. The new targets are set out in Table 2 below. As it happened the project was able to accommodate the increase in group working without reducing the individual counselling numbers to the revised target, which has consequently been well met even with the January cut off (see below).

Table 2: Revised Outputs
Output	Plan		Actual
	June 07	Mar 08	June 07	Jan 08
No of young people given counselling	5	25	6	38*
No of workshops for parents	1	4	2	3
No of parents attending workshops	15	60	15	17
No of workshops for professionals & workers	1	2	1	2**
No of professionals & workers attending Anger Management Training	20	40	22	40
No of Full Time Staff employed	1.5	1.5	1.5	1.5
No of young people attending a group session	28	56	46	62
Number of groups sessions	7	14	10	15
* number completed or terminated
** three more planned for February and March 2008

            Individual Targets

Young People Individually Counselled

12.       The figures for individuals counselled are complicated by the nature of the cohort in that some started and did not finish; some were waiting to start the programme; and some finished but we did not acquire the CAT1 forms (see Section 4 on Outcomes for more detail). The target was revised downwards following the introduction of group sessions from 50 to 25. We discuss the various ways of measuring the output in Section 4 but with 38 individuals starting the course and 26 completing it the revised target has been met.

13.       A wide variety of agencies referred young people but most are under the umbrella of Plymouth City Council. The One Year summary produced by Brighter Futures in support of its bids for funding to maintain the programme after March 2008 estimated that for both Individual and Group work 93% of the referrals came from Plymouth City Council departments. Referring organisations included -

·        Social Services

·        CAST Team – Plymouth City Council

·        Young Runaways Team – PCC

·        Youth Offending Team – PCC

·        Youth Inclusion and Support Program – PCC

·        Youth Inclusion Program – Youth Offending Team – PCC

·        Secondary Improvement Program – PCC

·        Schools (Primary and Secondary through Excellence Cluster, SENCOs and EWOs)

·        Parent Support Advisors – PCC

·        Positive Futures – PCC

·        Frederick Street Centre – Young People’s Services – PCC

·        Connexions

·        Harbour Young People’s Service

Workshops for Parents

14.       The response to the workshops for parents/carers/guardians has been disappointing and the numbers are well below target. This is not due to lack of effort by the team but to parents’ reactions. The first two workshops were reasonably well attended with 15 parents attending. A third one in late 2008 had 14 confirmed attendees but only two turned up on the day. It was therefore decided that with limited resources there was little point in trying to organise another and the effort was put into the professional workshops.

15.       The number of workshops target has therefore also not been met. We are relatively relaxed about the numbers as these workshops are voluntary and the team cannot force anyone to attend. It is nonetheless a disappointing response to a part of the programme that was designed to reinforce the change in attitude and behaviour of the young people counselled. Ironically many of the positive comments about that change are from the parents who recognise the impact made on their daily lives. We had intended to analyse the feedback forms from the parents’ workshops but because of the numbers this has not been possible.

Workshops for Professionals and Workers

16.       This is very much a success story. The first workshop had 22 attendees with a waiting list for another. That took place in December 2007 with 18 attendees. Thus both the number of workshops and number of attendees project targets have been met.

17.       More people and agencies have expressed an interest and more workshops are planned. The one planned for February already has 15 signed up and the project is confident of reaching 20. This is all the more encouraging as the workshop places are not free and now cost £85 per place as opposed to the £20 charged for the first one. From the feedback forms there is a demand for a higher level workshop i.e. taking it to the next stage and this is planned for March as is another starter level one so this target will be well oversubscribed.

18.       The organisations/agencies that attend the workshops are asked to feedback their views and these are unvaryingly positive. The more advanced workshop came out of these responses and there was also a call for one working with adults. Typical comments were –

            “The course was very well delivered and extremely relevant”

            “I feel more confident in my ability to respond to people with emotional/anger issues”

            “I think this course would be of enormous benefit for teachers and support staff in schools”

“Really found the day’s work tools useful.”

19.       Typical attending agencies/organisations were -

Public Health – NHS

Positive Futures – Plymouth City Council

YISP – Youth Intensive Support Programme

YIP – Youth Inclusion Programme

Plymouth City Council - Trainers

Parent Support Advisors Project

White Gold Project

Starting Point Project

ISSP – Intensive Support & Supervision Program (Youth Offending Team) SALT – Sexual Abuse Counselling Agency

Money Advice Plymouth

Behaviour Support Team

Group Sessions

20.       Group numbers were added in the revised outputs. Fifteen group sessions have been undertaken involving 62 separate individuals. Some were involved in more than one group depending on their circumstances – one was involved in 4 – so there 75 attendees in total. On either count the target of 56 has been comfortably met.

Full Time Staff Employed

21.       This target has been met with the appointment of Danny Biscombe as the Outreach Worker and Neil Thacker in administration.

          Outcomes

            Impact on the Individuals

Table 3: Client CAT1 scores
Our reference	Client reference	Start	End	3 mths

1	001	112	64	C
2	002	68	21	32
3	003	42	14	C
4	004	66	28	38
5	013	137	47	C
6	014	101	T	T
7	015	84	30	C
8	016	108	108	98
9	017	56	32	C
10	018	83	C	C
11	022	31	T	T
12	023	128	T	T
13	024	106	T	T
14	025	78	T	T
15	031	42	17.5	21
16	032	111	50.5	23.5
17	033	116	29	32
18	034	84	39.5	46
19	039	168	W	W
20	040	64	19	C
21	041	100	W	W
22	042	108	64	C
23	043	124	21	28
24	068	116	69	C
25	073	91	T	T
26	074	61	18	C
27	075	121	86	C
28	076	103	31	46
29	089	142	T	T
30	090	36	36	C
31	091	50	T	T
32	092	178	W	W
33	093	75	75	T
34	094	66	W	W
35	095	155	T	T

36	096	110	W	W
37	097	72	17	C
38	098	79	W	W
39	099	64	T	T
40	100	78	40	C
41	101	80	W	W
42	102	76	40	C
43	103	112	18.5	C
44	117	106	W	W
45	118	47	W	W
46	119	163	W	W
47	120	105	W	W
48	121	117	W	W
49	122	77	W	W
50	123	54	T	T
51	124	69
52	125	48	W	W
53	126	156	W	W
54	127	86	W	W
55	128	51
56	129	91	W	W
57	133	33	T	T
58	134	84	W	W
59	135	68	W	W
60	136	43	W	W
61	137	139
62	143	69	W	W
C = completed but no other CAT scores
T = terminated for various reasons
W = were waiting for a place
Source: CAT 1 forms and Brighter Futures database

22.       The Outputs from the programme are set out in the previous Section but it was also felt appropriate to try to measure the effect on the individuals taking part in the programme as it is their behaviour that will affect the contribution to the Floor targets. We therefore measured impact through a Criteria Assessment Tool (CAT) which had been especially devised for the programme by its Outreach Counsellor Danny Biscombe. This was not however an untried tool as it is heavily based on the Ahimsa Model already used in Plymouth. The maximum an individual could score is 228. The scoring system is described in Paragraph 31 below.

23.       As can be seen from Table 4 below it consists of a series of physical and emotional activities with an assessment of how prone the individual is to them. The intention was to score them at the beginning of the counselling; immediately after it finished and then again after three months to see if the hoped for improvements had lasted. We did not have direct access to the young people and were dependent on the referring agencies and the cooperation of the young people to make this work. We believe however it is a powerful measurement tool.

24.       In our view, in a programme such as this it was unrealistic to expect that everyone who was referred or started individual counselling would complete the full 12 sessions. Also it was possible that a number of sessions were prone to be missed, prolonging the counselling for some. This could be for very valid reasons such as sickness, holidays, exams, or simple non attendance. Some however were deliberate and judgement was then used to determine whether the individual actually wanted to be involved. Where the conclusion was negative then the course was stopped. That is shown in the above Table as Terminated.

25.       Table 3 above sets out the individuals selected to take part in the programme. The total of 62 is well above even the original target but 21of these (marked with a W) had not started when the individual counselling programme was finished in January. We have no information on three individuals identified but believe they did not participate leaving 38 as the number where counselling began. Of those 12 were terminated for the reasons given above with the most extreme being that the individual was imprisoned leaving 26 that completed the course – in line with the revised target.

26.       We have CAT1 scores for 25 of these at course completion enabling a direct comparison to be made with their situation before the course started. Eight also filled in a CAT1 three months after the course completed so we can make some tentative judgements about the longer term impact.

27.       What is clearly shown from the figures (and backed by the reports on the individuals by referrers, parents etc) is that the course had a dramatic effect on the behaviour and attitude of virtually all the individuals that took part to the end. As we would expect the impact varied from individual to individual but it is significant in every case; with the exception of Client 16, who only had three weeks of counselling and Client 90 where we suspect the form has gone awry.

28.       Falls of well over 50% are common between the beginning and the end of counselling and those forms that we have for three months after the end of counselling show that the improvement in behaviour and attitude was sustained. The majority show an increase from the second CAT1 score but to nowhere near the levels of the first one showing that the benefits last longer than just the counselling period.

29.       79% of the total cohort were male and 58% were known to the police before counselling, which makes the behaviour improvement even more impressive as clearly the programme was dealing with some very challenging individuals.

30.       We also thought it instructive to look at the effect on the identified behaviour or attitude to demonstrate if the improvement was in those areas that the programme was designed to improve. Table 4 sets out the consolidated figures for each of the actions for the cohort where we have first and second CAT scores.

Table 4: Consolidated CAT1 Scores by Issue Before and After Counselling
Issue	Before	After	Issue	Before	After
Impulse control	46	24	Malicious intent	16	6
Physical violence	54	19	General attitude/beliefs	32	18
Physical aggression	62	25	Derogatory statements	21	14.5
Verbal aggression	66	29	Self value	42	17
Reckless behaviour	45	14	Self importance	32	12
Aggressive body language	50	22	Self care	28	9
Intimidating behaviour	30.5	19	Impatient	58	29.5
Destroy/damage objects	49	13	Respecting others	48	19.5
Punch walls/throw objects	59	16.5	Minimisation of behaviour	44	16.5
Shout/scream	68	29.5	Victim empathy/remorse	35	11
Swearing/abusive language	56	33.5	Accepting responsibility for self	46	17
Disruptive behaviour	47	33	Dealing with emotions	51	21.5
Discriminatory behaviour	18	8	Stress management	59	20
Demand own way	54	22	Sexually harass others	0	0
Controlling of others	39	15	Distorted/fantasy thinking	11	15
Manipulate/exploit others	22	10.5	Motivation to change	25	8
Name calling/put downs	35	22.5	Tolerate vulnerability	26	22.5
Break agreements	40	21	Self reflection/understanding	33	12
Sabotage relationships	22	10	Listening/pay attention	45	21
Withdrawn	19	8	Relationships with others	31	14
Threaten self harm	18	3	Engage with support offered	22	6
Threaten harm to others	48	15	Domestic problems	41	9
Denial	43	19	Attendance record	17	8
Accepting consequences	48	20	Drug use	18	5
Lying/honesty	40	16	Alcohol use	18	9
Blaming others	41	21.5	Anti social behaviour	24	11
Sulk	36	16	Criminal activity/known to police	15	7
Gives in to peer pressure	27	15	Carrying/use of weapons	5	0
Resentment	29	17
For the 23 individuals that completed both sets of CAT1 forms

31.       Each of the traits is given a figure of between 0 and 4 for the scale of concern viz –

Ø      0 = no concern

Ø      1 = some concerns

Ø      2 = often a concern

Ø      3 = constantly a concern

Ø      4 = serious concerns

32.       What Table 4 shows is those areas that are of the greatest concern identified through this tool – they have the highest starting numbers – and the impact of the counselling on them. The other point of this consolidation is that while the figures for any individual depend on the accuracy of those compiling it, consolidation helps remove any individual blips and shows the overall effect on specific behaviour and attitudes that the programme was intending to address.

33.       The results clearly show that the right people were referred in that the programme was designed to deal with anger, aggression and violence issues and they score the highest before counselling. In total the drop between before and after counselling is 56% but the highest rated show quite a large variation with Punch walls/Throw objects down a massive 72% but Swearing/Abusive language only down 40%.

34.       These consolidated figures therefore reinforce the ones in Table 3 on the individuals by showing that the improvement was in those areas of greatest concern and those the programme was designed to affect.

35.       Because of the closure of the programme we only obtained eight 3 month detailed forms. While the comparisons therefore cannot as robust as those above there is sufficient data for us to draw tentative conclusions. Table 5 below compares the highest rated attitudes/behaviours over the three CAT1 forms.

Table 5: Before, After and 3 month CAT score comparisons: Highest Rated Issues
Issue	Before	After	3 months
Physical violence	17	7	3
Physical aggression	21	8	6
Verbal aggression	24	8.5	9
Aggressive body language	14	3	5
Punch walls/throw objects	20	5.5	2
Shout/scream	26	11	9
Swearing/abusive language	24	11	9
Demand own way	20	4	4
Impatient	24	10.5	8
Dealing with emotions	20	7	5
Stress management	22	6	7
TOTAL	222	81.5	67
For the 9 individuals that completed three CAT1 forms

36.       While 8 out of 26 (31%) would normally be considered a sufficient response rate for the results to be regarded as robust there is one caveat we must express about these results. The eight are not fully representative of the full cohort in that while the full cohort’s scores fell by 56% before and after counselling, the eight fell by 67%.

37.       Furthermore it can be seen from the above Table that the scores for the highest rated issues continued to fall while the total scores for this cohort rose between the second and third counts. The above fell by 18% whereas the overall total for the 57 issues rose by 13%. We are therefore wary of reading too much into these figures other than the general point that the considerable improvement shown was largely maintained three months after the end of counselling; and for the eight at least the scores for those issues rated the most problematic in the cohort as a whole continued to fall.

38.       If that is a reflection of the cohort as a whole it is a powerful testament to the ongoing effect of the counselling as while the less heavy and more peripheral to the counselling issues moved back up after three months the relevant ones continued to fall. As we state above we are reluctant to read too much into such a small response and would have liked to have been able to follow up with all 26 to verify these results.

39.       Our conclusion is therefore that the programme addressed the concerns it was designed to address and had a beneficial impact on the individuals that took part and stayed the course, which lasted after the course had finished.

            Group Sessions

40.       The original targets contained no numbers for group sessions as they were not envisaged as being part of the programme. When they were introduced a target of 56 was inserted which we believed to be very reasonable given the fall in individual counselling. This target has been met with 62 people involved in group counselling. It was clear however that those involved in the group sessions cannot be measured in the same way as those receiving individual counselling.

41.       All of those involved in the group sessions are of school age and have either been excluded or identified as in danger of being excluded. As the school is the common factor we decided in consultation with the project that the measurement be whether they have either returned to school if excluded at the moment or remained in school if not. The same principle of following up at a later date to ascertain the current situation - e.g. whether they returned to school and then been excluded again – could be applied.

42.       Fifteen groups totalling 75 from 62 different students have been through the counselling process. The largest group consisted of 8 members and the smallest two. 69% were male and 53% were definitely known to the police so again they were a challenging set of individuals. As of the end of the December 2007 term 50 (81%) were still at school with 6 (10%) having legitimately left or moved away.

43.       This seems to us to be a very respectable retention rate as with 2 others (3%) attending off site education only 4 (6%) have been excluded or are refusing to attend school. We are therefore relaxed about the change in emphasis from individual to group as it is self evident that if the individuals are in school then they are not on the streets potentially contributing to the problems identified in the Floor Targets below.

            Wider Impact

44.       This project had to demonstrate that it would contribute to the Neighbourhood Renewal Themes and help achieve the Floor Targets. Those to which it should contribute are –

Ø      20% reduction in criminal damage (Youth Related)

Ø      20% reduction in Violence (Young People)

Ø      20% reduction in Vehicle Crime (Youth Related)

Ø      5% reduction in attendance at A&E

Ø      20% reduction in Anti-social behaviour by young people in the neighbourhoods.

45.       From an evaluation perspective there are a number of caveats that have to be placed on the above. The first is that the collection of these statistics is not within the remit of the project. Secondly the timeframe for collection will naturally be outside that of the project; and thirdly there is no easy way of showing a causal connection between the output of the project and any movement in the above figures. For instance all the young people involved in the programmes might reduce their contributory behaviour but other factors might cause it to increase from other sources.

46.      We therefore have reservations about attributing movement in these desired outcomes directly to the project, though obviously any improvement in individual behaviour will help.

Programme take-up and Sustainability

47.       We have seen how the emphasis within the project shifted towards group working as a result of demand from the referrers for larger numbers of individuals to be involved. This did not mean that the need for one to one working went gone away and there were people on the waiting list when the project closed.

48.       What has been shown is that demand was exceeding the team’s resources. If the funding was there, Brighter Futures could have provided extra staff, by way of a part time specialist and the demand within the three areas mostly satisfied as well as moving outside the NRF areas - which the project could not handle without additional non tied funding.

49.       There was also an ambition, in the interests of sustainability, to try to move towards being paid for each referral handled but most of the referring organisations stated that they did not have the resources to pay. A Community Interest Company (CIC) was set up to take this and the training workshops forward. Whether a similar counselling programme can be resurrected in the future is outside the scope of this evaluation.

50.       What we can state however is that from the evidence of the CAT1 forms, statements from individuals, referring agencies and parents and feedback forms, the programme was having an impact on curbing anger and aggression with the consequent improvement in attitude and behaviour, and that improvement was being sustained at least in the short term.

Ends

16/01/08