IMPACT Europe at the European Evaluation Society Conference

IMPACT Europe researchers presented their work in progress and discussed future directions with evaluation experts at the 11th European Evaluation Society Biennial Conference in Dublin, Ireland, on 1st – 3rd October, 2014.

Allard Feddes, Post-doctoral researcher in the Department of Psychology, University of Milano-Bicocca, Italy, presented the initial results of the analysis of the state-of-the-art in evaluating counter-radicalisation interventions that he has carried out with project partners at TNO.

The details of 126 evaluated interventions have been collected and coded. They range from awareness-raising among people at risk of becoming violent, to social work with members of local communities. Across this collection, the interventions focus mostly on Islamic extremism and right-wing extremism, and most address the individual with the aim of long-term prevention, or restoration of the de-radicalised phase. The associated evaluations mainly use cross‐sectional designs and theoretical designs and are mainly theory-based single methods, focussing on the impact and mechanism of the intervention rather than, for example, the process or the economics.  To take the analysis forward, the IMAPCT Europe researchers intend to bring in learning from fields where evaluations are more common, such as gangs and other fields of criminology; find more empirical research; and look at alternative evaluation methods that can deal with complexity and network analysis.

In a separate, round-table discussion session, Joanna Hofman, Senior Analyst from RAND Europe in Cambridge, elaborated on the project’s current thinking about the design of the IMPACT Europe evaluation toolkit.

The framework underlying the toolkit would be a database connecting radicalisation factors to existing interventions, to current evaluation approaches, methods, and metrics for measuring effect. The toolkit is therefore navigated as a decision tree, where users answer a series of guiding questions, to arrive at an appropriate menu of options for evaluation methods.

Tom Ling, Senior Research Leader at RAND Europe, then outlined the role of evaluation in building a community of practice that over time helps to disseminate good practice and shared values, something that will help to bring together the front-line practitioners in counter-radicalisation.

“Evaluators can begin to unlock potential that isn’t currently being tapped in the field of counter-radicalisation”, said Ling. “Compare this field with, say, healthcare  – in which you have powerful professionals with a strongly shared vocabulary – and you can see that we can gain enormously from embedding an evaluation culture into policy-making in this area.”

The evaluation experts attending the round-table session were asked to comment on risks and challenges involved in developing the toolkit, what factors to take into account, whether the toolkit could be useful more widely to the evaluation community, and whether there were examples that the IMPACT Europe researchers could learn from.

Ling reports: “We had two eminent attendees in our session, Professor Nicoletta Stame, a past president of the EES, and Professor Ray Pawson from the University of Leeds. Neither of them said ‘You’ve got this sorted’! Our audience generally agreed that this is a very challenging project and that the key step at this early stage is to provide a clear conceptual and theoretical framework to underpin the project’s approach. There was a good debate and we got the kind of constructive criticism that we needed to inform our work in the coming months.”

Participants made several useful contributions, pointing out that the range of possible interventions is very wide, and in each case the local context must be taken into account in designing an evaluation.

The researchers are continuing to review the state-of-the-art on what interventions are being used and where they are successful, consider good practice, and prepare workshops with stakeholders who are already experienced in evaluating these types of interventions.  Project partner ITTI in Poland, who are developing the toolkit software, will begin designing the look and functionality of the toolkit so that workshop participants can test the inbuilt framework.

Since the conference, using further learning from the project, the team have developed the plan for the toolkit to include the following elements:

• Front page providing a high-level explanation of what the toolkit is about and a structure that takes users to more detailed information depending on their preferences and interests
• Second-level information could form an inspiration platform and display a list of evaluable interventions
• Third-level information could explain how one can operationalise evaluation and carry it out step-by-step
• Fourth-level information could be dedicated to policy level and meta-analysis for registered users only.

With 2015 declared as the International Year of Evaluation we can be sure that evaluators worldwide will continue to be interested and involved in the progress of IMPACT Europe, and the researchers look forward to presenting their evolving research at future conferences.