Her Majesty’s Prison & Probation Service has just (22 March 2019) published a useful Toolkit to measure intermediate outcomes to reduce reoffending from arts and mentoring interventions. Authored by Mike Maguire, Emma Disley, Mark Liddle, and Rosie Meek with Nina Burrowes, the main output was a “robust but user-friendly” instrument designed to measure the outcomes of mentoring or arts interventions for offenders. The focus was on ‘intermediate outcomes’, defined as measurable changes in individuals that are directly or indirectly associated with reductions in reoffending. These outcomes may, of course, also produce other social and individual benefits, independent of any demonstrable effect on offending.
A theoretically informed 29-item questionnaire, named the Intermediate Outcomes Measurement Instrument (IOMI), was produced through an iterative process of literature searches, consultation, piloting and analysis. IOMI has not undergone full validity and reliability testing, but the results of initial testing have been encouraging.
The IOMI forms part of a broader toolkit for service providers to assist them in evaluating their work. The toolkit includes: guidance on the administration of IOMI and how it could be used as part of wider evaluations; a data entry tool to facilitate the collection and analysis of IOMI data; and a tool for the collection and presentation of cost data.
While the IOMI was developed for use in arts or mentoring interventions with adult male offenders, it is likely that it will have wider applicability to other intervention or groups. Its main purpose is to provide evidence of offenders’ progress towards desistance, which is not necessarily reflected in standard reoffending rates. This could be particularly valuable for capturing any contribution to desistance made by individual interventions which are not expected to reduce reoffending on their own. You can see a page of the IOMI instrument in the image below.
Context
The preliminary research for the project found strong interest among providers in a tool of this kind, with providers admitting that they relied too much on anecdotal evidence when attempting to convince commissioners of the effectiveness of their work. At the same time, some resistance was encountered, particularly among front-line staff and volunteers, to the idea of introducing ‘measurement tools’. This was largely because of concerns that voluntary agencies often lack the capacity to collect and analyse significant amounts of data. So, while there was clearly an appetite for measurement, there was also a demand for simplicity and succinctness in any evaluation tool.
Validity
The IOMI itself is theoretically informed and is anchored in careful reviews of evidence from research literature, as well as extensive consultation with providers of mentoring and arts programmes. The results of all three stages of testing were highly encouraging. There were strong indications that the instrument has face validity, internal consistency, stability and the potential to reflect change across most dimensions. At the end of the research period, it was concluded that the IOMI is in a form that is likely to require little further amendment, apart from possible changes or additions to the questions in the ‘resilience’ dimension.
The authors, however, do emphasise that an instrument of this kind inevitably requires extensive testing over longer periods of time (including the collection and analysis of reconviction data on those who have completed the questionnaires) before confident claims can be made for its validity and reliability as a measure of psychological change, or for its robustness as a proxy for reoffending.
Potential uses
The authors make clear that the tool is not yet sufficiently validated for it to provide a benchmark to compare provider performance. However, the tool does have considerable potential to be used in the following ways:
- Alongside other evidence, helping bidders to make a case to commissioners for the effectiveness of particular interventions or practices.
- Helping organisations improve their services by finding out more about the outcomes that they are producing for clients.
- Offering an extra resource for research into what works in offender rehabilitation, especially into the effectiveness of small-scale and less mainstream interventions such as arts programmes.
