The Ceja Andina project is working to ensure the sustainability of agriculture and forest biodiversity in the Andes of northern Ecuador. Funded by the International Development Research Centre (IDRC) and implemented by Corporación EcoPar, an Ecuadorian research NGO, the project works with a diverse range of stakeholders, from farmer research groups to local municipalities. Together, they aim to develop knowledge, capacities and social learning to support community-based natural resource management.

Before commencing, the Ceja Andina project team members and stakeholders came together to discuss the monitoring and evaluation objectives. All participants in the project were conscious of the need to engage in some form of accountability exercise, but also wanted to place high priority on collective and individual learning. We began by examining the purpose of doing M&E through open and facilitated discussions involving the project team and its ‘boundary partners’ – individuals, groups and/or organisations with whom the project works in order to influence and support changes in behaviour.

The purpose of M&E

Was M&E necessary to ensure accountability? If so, to whom? Or was it intended to encourage learning processes, at the group, organisational and/or individual levels? It was agreed that the objective of M&E was not to produce voluminous reports recording all the things the project had done to ‘achieve impact’, but rather to identify and analyse the changes taking place in groups and individuals involved in the project, what worked and what didn’t, and how the project could continually improve.

The project defined success not just as the achievement of material goals such as an increase in the number of trees planted, watershed management plans developed, improved water quality, or reduced soil erosion. Even more important were the complex and progressive changes in the behaviour of the actors involved, on whom such material changes depend. Collective learning came through an analysis of how those changes were shaped, and what changes each individual or group (including the project team) has the power to influence.

What the project wanted to examine was in fact the changes – especially behavioural changes – that were taking place, and not simply to make a claim to them. As a group, we required a methodology and a set of tools that would allow us to tell a story demonstrating the richness of the changes going on behind the scenes, and not just those at centre stage. Outcome Mapping (OM) proved to be such a methodology.

Best used as a facilitated participatory and inclusive process, OM helps a programme, or project team, to identify those actors with whom it wants to work, what changes it expects to see (outcomes), and the strategies it can use to support those changes. An essential aspect of OM is that ‘outcomes’ are defined as changes in the behaviour, relationships, activities or actions of the people, groups and organisations that are within the project’s ability to influence.

Organising ourselves

In order to learn from change, and to learn together by celebrating successes and examining failures, the project team, together with our boundary partners, decided at the outset that the M&E process should be a participatory one. We hoped that the process would provide a space for social learning and creative dialogue, bringing together the different perceptions, experiences and actions of boundary partners and other stakeholders in the development of the region. We also envisaged that this common space would work with this diversity to encourage discussion and negotiation, resulting in a common vision and shared priorities, as well as plans for concerted action based on the recognition of the interdependence of all the actors involved.

In our efforts to keep with this goal, we developed participatory planning sessions, where the project facilitated negotiation and concerted action among boundary and strategic partners. We also organised and facilitated two-day monitoring workshops with our boundary partners, held every six months, that allowed us to explore ‘what happened’ and who had contributed to it. Not only did these sessions allow us to monitor and assess the different contributions. They also helped us to plan modifications and improvements with a focus on ‘sharing the workload’ among the boundary and strategic partners. That we had reached consensus on a common vision at the start of the project helped us to maintain this complementarity, as this became a reference point to guide strategy formulation and actions, and against which we were then able to track performance.

Discoveries about OM

Outcome Mapping did not answer all of our M&E queries. In fact, in its application it created even more questions, which in turn resulted in fruitful reflection and dialogue on monitoring and evaluation in general. Some of the main points that emerged from that dialogue are sketched out in the following.

OM is not a panacea. It does not replace the log frame approach (LFA). However, by discussing the usefulness of OM and LFA, and identifying the gaps in the two approaches, we were able to determine when it was best to use one or the other. We also began to explore other complementary tools and methodologies. For example, the log frame approach can be useful in outlining the broad scope or reach of the project, and the ‘change in state’ that the project is aiming to achieve. By complementing this approach with OM, a project can put its plans ‘under the microscope’ and look at what really causes such changes in state, or changes in the behaviour of different actors. The project then has the opportunity to plan, examine, analyse and adjust its strategies to support those changes in behaviour, before trying to account for changes in state. Besides the log frame approach, there are many other complementary tools and methodologies that can help to strengthen the OM process. For example, when using the monitoring journals in OM, some practitioners have found it useful to bring in the ‘most significant change’ methodology.

Outcome Mapping is only as good as the facilitation that is used to apply it. Indeed, effective facilitation is imperative for its success. An external facilitator is not always necessary. The Ceja Andina project implemented OM with internal facilitation, although other projects and programmes have found external facilitators useful for OM training, intentional design workshops, and to provide assistance during the monitoring and evaluation. OM, especially when it aims to create collective learning processes, must be guided by someone who can dedicate time to it, and who also has a solid background in facilitation. Adequate resources must be planned for and dedicated to this process.

OM can be replicated in a diverse range of settings and contexts, but an understanding of OM’s central concepts – behavioural change, contribution and not attribution, finding the balance between accountability and learning – is vital. With these key concepts guiding the process, OM can still be flexible enough to meet the monitoring and evaluation needs of a programme or project. It inspires and encourages creativity, so that the likelihood of endogenous change processes being created is high.

At a broader level, there must be further exploration of how OM and the learning that emerges from it can contribute to policy processes, civil society strengthening, advocacy and good governance.

Accountability and learning

If a programme, project or organisation is truly interested in achieving a balance between accountability and learning, this will require judicious choices about what needs be monitored and evaluated. For the Ceja Andina project, the richness of the information produced during the OM process presented us with the challenge of deciding what information it was ‘necessary’ to collect for accountability purposes, and what data needed to be analysed to contribute to learning.

Developing a detailed and well thought out monitoring and evaluation plan (with corresponding resources, such as budgets and human resources), as well as planning, organising and facilitating the monitoring workshops, helped to streamline the M&E processes using OM, and to make them more effective learning opportunities.

As our writing and reporting skills improved, many practical tasks became simpler and more transparent. In our reporting to IDRC, we were able to honestly ‘tell the story’ of what had happened, including whether or not the strategies we had developed had contributed to outcomes, and what other strategies (set by other stakeholders, including our boundary partners) had contributed as well. The process was iterative but successful, judging from the positive feedback received from our partners and stakeholders, as well as from IDRC.

Diverse perspectives

For all the stakeholders in the Ceja Andina project, taking on and implementing OM was a valuable and creative experience. It helped to develop the capacities not only of the project team, but of its partners as well. Through participatory planning, monitoring and evaluation, these partners were able to exercise and therefore increase their capacities for critical analysis, social learning, team building and negotiation, and constructing consensus. The project team, through the use of OM, incremented their capacity to facilitate the process. It also pushed the team to look at this particular initiative from diverse perspectives, particularly those related to behavioural change.

The results of using this approach, and the capacities produced because of the high level of engagement of all parties involved in the process, has led the Ceja Andina project to conclude that Outcome Mapping is a methodology that is tremendously conducive to achieving learning with and accountability to multiple stakeholders.

Links

Corporación EcoPar

International Development Research Centre (IDRC) Outcome Mapping

International Association for Public Participation (IAP2)

International Institute for Facilitation and Consensus (IIFAC)

Latin American Center for Outcome Mapping (LACOM)

Outcome Mapping Learning Community

Resource Centres for Participatory Learning and Action (RCPLA)

Further reading

S. Earl et al. (2001) Outcome Mapping: Building Learning and Reflection into Development Programs, IDRC.

B.M. Kibel, Focusing on spirit: The journey mapping approach.

S. Montague et al. (undated) Using Circles to Tell the Performance Story.

T. Smutylo (2005) Outcome Mapping: A Method for Tracking Behavioural Changes in Development Programs, ILAC Brief 7, CGIAR.

Outcome Mapping Users’ Workshop, Lima, Peru, October 2004.