Cross posted from the Learning Portal for DM&E for Peacebuilding. Find the original post here.
Recently, there has been a heated debate on differing approaches to the design and evaluation of peacebuilding initiatives, and a publication by Reina Neufeldt does an excellent job of exploring the differing approaches and identifies emerging models and practices that mediate tension between the two perspectives. She describes the two camps as framers--those who develop linear, cause-effect thinking, measurable indicators, and empirical findings--and circlers--those who are relationship-focused, context-driven, and adaptive.
Hot Resource! “Frameworkers” and “Circlers”—Exploring Assumptions in Impact Asses... by Reina C. Neufeldt
Neufeld begins by explaining that this debate is not just a superficial one but instead lies on an epistemological and ontological level. In this table, the framers are closely associated with the positivist approach, and the circlers are associated with the interpretive approach:
As this table indicates, framers and circlers often see how the world operates in a very different light, which causes them to subsequently take their approaches to evaluation very differently. It is important to note that not every framer is a positivist and not every circler is interpretive. Often each camp borrows from the other, overlapping in some categories, or at the very least acknowledges the other approach in some circumstances.
To look at these approaches specific to the peacebuilding context is difficult. At first look, circlers and the interpretive approach aligns closer to the underpinnings of peacebuilding: relationship-focused work in complex, fast-changing environments. However, when it comes to evaluation, peacebuilders also need to develop rigorous measurement systems that attempt to find causality, or at the very least, contribution.
This is where it starts to get complicated. How do peacebuilders make sure our systems of evaluation are in fact more rigorous? Often times this means evaluation becomes quantitative in nature, or more in line with the framers perspective. Despite the belief that quantitative and cause-effect questions are generally believed to add rigor to an evaluation, sometimes this may not necessarily be the case.
Take for example, Randomized Control Trials (RCTs), which over the past few years have become the “gold standard” of evaluation. RCTs have been glorified because of their scientific rigor and their supposed ability to test any sort of intervention. However, in the excellent Randomised Control Trials for the Impact Evaluation of Development ..., Carlos Barahona notes two problems: scale and contamination. On one hand, the indicators for RCTs need to be limited to a size in which they can be accurately measured, and on the other, a level of control must be held that nullifies the contamination of outside variables. Often what occurs in RCTs is the question that it seeks to answer is so miniscule that it ends up being a “well that was obvious” moment or they run into context-specific results that fail to generalize. In other words, RCTs can run into many of the same problems of an interpretive approach and therefore may not necessarily add significant rigor to an evaluation.
Hot Tip! Check out this TED Talk by Esther Duflo, founder of the Jameel Poverty Action Lab at MIT, on why RCTs are important in international development.
This is not to say that RCTs and Framer approaches do not have a place in peacebuilding evaluation. Rather, Framer and Circler approaches are just that – a place among other approaches, not the methodological approach to all evaluation contexts. RCTs are not a “one size fits all” evaluation approach. Instead of defining credible evidence by which approach is used, evidence that is credible should be determined by the question, the context, the methodological assumptions, the theory of evaluation and the resources available.
Hot Resource! Check out evaluator Jane Davidson’s blog post on why there is no best tool to measure effectiveness!
The question that arises now is how to add value to peacebuilding evaluation methodologies in the midst of the framer-circler debate. Despite the differences between the two, there are common goals that can be built upon. Both groups desire to transform conflicts, promote peace, and respond to community needs. With this in mind, Neufeldt suggests that there are two emerging models that combine both circler and framer approaches.
First, Neufeldt mentions theory of change as an approach in which both circlers and framers can apply their pre-existing worldviews. Framers can develop potential cause-effect relationships within theories of change while circlers are not limited to linear approaches and can still use more nuanced language. More specific to evaluation, Neufeldt states “Frameworkers can look to test theories of change, while circlers can look to reflect on practice and better understand what is going on and what they are learning in the process about how to impact change.”
Hot Resource! Peacebuilding with Impact: Defining Theories of Change by CARE
Second, the Most Significant Change approach is identified as a bridge in the circler-framer tension. Through “systematic storytelling,” MSC allows enough rigidity to appease framers but still allows fluidity for circlers; for framers it is a system, and for circlers it is storytelling. This fluid rigidity allows for a group of stakeholders to systematically review and collect stories to determine what is the most significant change or impact of the intervention. MSC also works to solve the differing “value-free” framer perspective and the “value-bound” circler perspective. Because MSC must determine what exactly is “significant” it must choose between competing value systems, but this is done through a decision-making process that is equitable to donors, implementers, and community members.
Hot Resource! The Most Significant Change: A Guide to its Use by Rick Davies and Jess Dart
Despite differing and often opposing worldviews between circlers and framers, there is space in which collaboration is possible. These creative approaches that attempt to incorporate both worldviews have a potential to add value and rigor to peacebuilding impact evaluation.
Randomized Control Trials for the Impact Evaluation of Development ... by Carlos Barahona
"Frameworkers” and “Circlers” – Exploring Issues in Impact Assessment by Reina C. Neufeldt
The Most Significant Change: A Guide to its Use by Rick Davies and Jess Dart
 Reina Neufeldt “Frameworkers” and “Circlers”—Exploring Assumptions in Impact Assessment, Berghof Handbook for Conflict Transformation, 2011, p. 491, accessed 6 June 2012, http://dmeforpeace.org/learn/frameworkers-and-circlers-exploring-as....
 Carlos Barahona, Randomised Control Trials for the Impact Evaluation of Development Initiatives: A Statistician’s Point of View, ILAC Working Paper 13, 2010, pg. 10, accessed 6 June 2012, http://www.cgiar-ilac.org/files/publications/working_papers/ILAC_Wo...
 Barahona, Randomised Control Trials, 11.
 S. Donaldson and C. Christie, What Counts as Credible Evidence in Applied Research and Evaluation Practice?, Sage Publications, 2008 quoted in Barahona, Randomized Control Trials, 11.
 Neufeldt,”Farmeworkers” and “Circlers,” 502.
 Ibid, 495.
 Ibid, 496.
 Ibid, 499.
 Ibid, 499.