Learning from Demand-Side Campaigns
30 November 2016
Norbert Cyrus & Dita Vogel
University of Bremen
Ordinary people’s behaviour matters – and may unconsciously contribute to the continuation of exploitation and trafficking in human beings. This basic rationale informs measures addressing trafficking in human beings from the demand-side - but too few people seem to be aware of it. Anti-trafficking efforts are strengthened by campaigns calling on people to change behaviour and take conscious action. They are asked to abstain from buying goods produced or services delivered under unfair and harmful conditions. Or conversely, they are requested to buy responsibly – goods produced or services delivered under fair conditions. Or they may be called upon to report seemingly suspicious occurrences to competent agencies. However, until now little was known about the working of these campaigns or their impact.
The European Commission’s tender for a research project on demand-side efforts against trafficking in human beings included the quest to analyse campaigns. As part of the DemandAT consortium, we committed ourselves to this task. The most appropriate approach, we thought, would be a thorough review of evaluation reports of demand-side campaigns. We compiled an overview of evaluation efforts in this field in order to create a ground for evidence-based judgements on the working of campaigns. We identified 55 campaigns that referenced a demand-side. Yet among these campaigns further information and related evaluation was hard to obtain. Finally, we could find only four external and four internal evaluation reports although we are unsure whether evaluation reports exist for some of the others. Nevertheless, our experience tallies with the findings of two recent studies commissioned by the European Commission. One study found that only 5% of a sample of 321 reviewed EU-funded anti-trafficking projects had been externally evaluated and that about two-third failed to provide a narrative report informing about results (Walby and others 2016: 93). Another study attested a “lack of evaluation” in the field of preventive anti-trafficking interventions (Deloitte 2015: 9).
Against the background of the EU Commission’s strong commitment to evaluation, this lack of evaluation came as a surprise. Moreover, looking at the few available evaluation reports it was surprising to notice that internal evaluations conducted by members of the campaign-conducting organisations actually provided more critical insights into the working of campaigns than reports by external evaluators. The few available evaluation reports did not apply – with one exception – the standards and norms of evaluation established by professional bodies such as the United Nations Evaluation Group. More concerning, the only report which referred to these norms and standards concluded that the lack of data prevents a sound judgement on the success or failure of the reviewed activities (Berman and Marshall 2011): Campaigns were not evaluable.
The insight that evaluability is a prerequisite for every evaluation effort guided our subsequent efforts. In addition to the preparation of a report on the insights from campaigns, we are developing a manual for evaluability designing. We intend to provide tools which can support project providers in considering the issue of evaluability when designing campaigns.
In order to validate our results and explore the helpfulness of the manual, we organized three focussed workshops with different stakeholders this autumn. A workshop with representatives from six projects performing demand-side anti-trafficking campaigns took place on 21 October 2016 in Berlin. The participants confirmed that evaluation is a rather neglected issue in the field of anti-trafficking interventions. Some NGO-representatives had even made the experience that proposed expenditures for evaluation had been cut by a donor. Others explained that they did not pursue efforts for self-evaluation due to a sense of lacking expertise and because nobody insisted. Organisations expressed concerns about side-effects, for example with regard to campaigns requesting to stop donating to begging children: “To ask people not to donate to children may be harmful when no alternative income opportunities are provided for families”. The concern was raised that some anti-trafficking interventions even continue harmful side-effects like stigmatization of ethnic minorities or women selling sexual services because these effects are not considered.
On 2 November 2016, we met in Brussels with representatives from four philanthropic foundations. Participants experienced in the funding of humanitarian actions expressed their astonishment that donors of anti-trafficking interventions do not effectively require thorough evaluation and thus miss the chance for learning lessons from experiences. Relating to their own experiences, one obstacle mentioned as impeding quality of evaluation was a fear on the side of project providers that openly sharing information on weaknesses and failed objectives will negatively influence the chance of subsequent funding. In contrast, the participants stressed the chance for learning and improvement and shared their experiences how funded projects can be encouraged to share their learning about intended and unintended effects.
The workshop with five evaluation researchers took place on 20 October 2016 in Berlin. The researchers commented on our research findings and their practical implications. It was particularly stressed that the designing of a project requires a proper analysis of the problem addressed. However, in the case of EU-funded interventions, organisations respond to a call for applications. Thus, the problem is already defined by the funder, requiring organisations to find out whether they agree. Another issue raised was the context analysis: the effectiveness of campaigns should be considered in the context of other activities or measures. For example the impact of a campaign to report suspicious observations depends on the effectiveness of the law enforcement system to sanction reported crimes and protect the victims – an issue that is beyond the control of the campaigners but crucial for the ultimately intended effects. While evaluators stressed the merits of external evaluation conducted by well-trained professionals, they agreed that investment in self-evaluation and evaluability will support the work of external evaluators.
The workshops yielded more detailed critical feedback and stimulating suggestions that help us to revise and complement the report and the manual. More than before, we are convinced that systemic evaluation – external as well as internal - can improve learning from demand-side campaigns.
The views expressed in the blog post do not necessarily reflect the views of the DemandAT Consortium as a whole