Bringing Monitoring, Evaluation and, Learning to animal advocates: 6 months of lessons learned

Introduction and Summary

Why focus on Monitoring, Evaluation, and Learning in the animal cause area?

When we started our international interventions in September 2023, we were quite certain that MEL could increase the cost-effectiveness and impact of interventions in the animal cause area, and avoid doing harm. See e.g. this post about why Anima International suspended the campaign to end live fish sales in Poland (Anima International 2022).

Tools and insights from MEL can help organizations design potentially more (cost)-effective interventions from the start, know if their interventions are on track, and adapt their implementation when necessary.

We also believe MEL can contribute to increasing the evidence base for interventions in the animal cause area. Neill Buddy Shah, a co-founder of IDInsight, observed, “The animal welfare research infrastructure and ecosystem is incredibly immature compared to what has developed over decades in social policy, medicine, and public health.” (EAG San Francisco 2019). Since 2019, research and the number of animal-cause area-specific research databases have increased (Navigating Research Databases for Animal Advocacy, 2024). However, the amount of research available still pales compared to other cause areas.

Uncertainties and findings

We were less certain about the willingness and ability of Animal and Vegan Advocacy organizations to engage with MEL. We also didn’t know if MEL tools used in other cause areas such as Global Health and Development would be applicable and useful in the animal cause area.

Overall, MEL is still a neglected topic in the animal community. EA-aligned organizations generally use MEL tools but many others don’t, and so far we have only verified a handful of organizations that have complete MEL systems in place that do not require additional support.

Specialized support for MEL is still very limited. If you are interested in supporting animal organizations with MEL, please consider working with us as MEL Associate or communications volunteer.

Below you will find 11 key lessons learned from our pilot intervention to train and support animal and vegan advocacy organizations in Monitoring, Evaluation, and Learning. 

We hope this post will be particularly relevant for charities and funders in the animal cause area and will lead to more organizations engaging with MEL and sharing best practices.

In this post, we will share what we’ve done so far, 11 key lessons we’ve learned, how they have influenced our strategy, and what you can do to help advance MEL and the overall evidence base in the animal cause area.

I. What have we done so far?

The Mission Motor’s current interventions were shaped during AIM’s (Charity Entrepreneurship) Incubation Program 2023. Between September 2023 and April 2024, we

• trained and supported eight animal and vegan advocacy charities to develop and implement MEL systems
• provided ad hoc support to another seven animal organizations
• are building a community of MEL peers and practitioners through Slack and monthly meetings

Our interventions are in the ideation phase with a focus on learning if, and why or why not, they work as intended. Although we are still in the first half of our pilot program, we see some early successes. The training and coaching sessions helped staff members

• increase their knowledge of MEL
• design theories of change
• identify key assumptions and risks
• unify the collective understanding of their programs.

As a result, several organizations identified key activities they wish to add to their program, decided to focus their efforts on specific target groups, or otherwise change their program design.

To mention one example: when designing the Theory of Change for their running cage-free campaign, an organization realized eggs from caged chickens and cage-free chickens were not distinguishable for consumers. Consumers were one of their target groups. If consumers cannot know where their eggs come from, they cannot decide to buy cage-free eggs and the organization’s Theory of Change doesn’t hold. We are now working on an adapted Theory of Change that has stronger links from activities to intended impact.

II. Key Lessons Learned

Here’s the short list of lessons learned, and how that influenced our strategy, followed by more detail below. We support and interact with EA-aligned and non-EA-aligned organizations. We did not separate our learnings for different types of organizations.

Lessons about the current state of MEL in the animal movement

1. Interest in and a perceived need for MEL exists in the animal cause area.
   => We have deprioritized motivational outreach activities.
2. MEL is often perceived as complex.
   => We are prioritizing offering specific, targeted tools versus complete MEL systems
3. Specialized MEL support is limited. Longer-term support has some unique advantages.
   => We continue to prioritize longer-term support
4. Most animal and vegan advocacy charities lack the capacity to fully engage with MEL.
   => We no longer strive to design and implement complete MEL systems for all participants. For some organizations, taking incremental steps, only using the most useful MEL tools, and gathering key data is much more beneficial given their capacity.
5. MEL tools from other cause areas are useful for the animal cause area. However, there are cause-area-specific challenges.
   => Because of low sample sizes, we place more emphasis on using qualitative methods.
   => We put more effort into detecting and sharing relevant case studies.
6. Funders can be a driving force for increased evidence-based work in the animal cause area. Factors such as funding MEL, requiring suitable MEL and a learning attitude can contribute to becoming that driving force.
   => We will explore if we can support funders in developing best practices. This could be in the form of an independent intervention of The Mission Motor, or in a supporting role to an organization like Charity Entrepreneurship which already designed a funder program concerning effective altruïsm.

Lessons for our own operations

1. Building rapport, involving the right staff, and understanding interventions are necessary ingredients for delivering useful MEL training and support.
   => We are building these elements into our standard operating procedures
2. Our cohort model does not generate sufficient value.
   => We let go of the original cohort model and tailor our support to the needs of each organization.
3. We still grapple with the question of how to narrow down our target organizations e.g. by geography or size.
   => We will continue to work with a variety in size and age of the organizations, and continue to explore if and how we can best focus on certain types of organizations.
4. Working with organizations on MEL primarily requires tailor made support. This makes our own cost-effectivess challenging.
   => We will prioritize supporting bandwagon interventions, defined as “an intervention that has been done a lot or is spreading rapidly to other organizations or contexts”. We expect the use of MEL for these interventions can be most impactful, if the insights influence others as well.
   => We will explore if and how we can contribute to the exchange of data and information between research organizations and implementing organizations, hoping to create a positive feedback loop toward more evidence-based working.
5. Using the right MEL tools at the right time is imperative to successful MEL.
   => Depending on the stage of the intervention, and size and development of the organization, we aim to offer specific, targeted tools versus complete MEL systems.

Lessons learned in more detail

1. Interest in and a perceived need for MEL exists in the animal cause area.

One of our key learning questions was if stakeholders in the animal cause area would be interested in learning and using MEL.

The response to our activities does indicate a perceived need exists: 

• 30 organizations applied to our one-year MEL support program
• over 150 animal advocates joined the MEL Slack subchannel
• our monthly online MEL meeting averages roughly 25 participants
• so far we have received 15 additional ad hoc support requests from other organizations

The organizations we support in our twelve-month program are direct implementation charities. We have provided ad-hoc support to direct implementation charities, coalitions, capacity-building organizations, and funders. 

We also have spoken about MEL with 6 grantmaking foundations. All of them think MEL in some form is important for their beneficiaries, and most agree animal charities can improve their MEL systems and processes.

Surprisingly, we haven't met a lot of resistance or skepticism. Where we met skepticism we often could take part of it away by focusing on using MEL tools that align with the type of intervention.

Although we realize we naturally attract organizations that are interested in MEL, we believe that the level of response indicates a fair amount of interest in the topic.

2. MEL is often perceived as complex.

Many organizations we spoke with, perceive MEL as complex or need help knowing where to start. During one of our monthly MEL peer support meetings, participants dove into reasons why MEL fails. Reasons listed included 

• MEL being complex and overwhelming
• using the wrong tools
• lack of training
• some types of interventions such as lobbying, are hard to measure
• it’s often difficult to quantify the impact

3. Specialized MEL support is limited.

Specialized MEL support in the animal cause area is limited. It mainly consists of advice, some consultancy and evaluation services, and introductory workshops by incubation and capacity-building organizations.

The Mission Motor primarily provides longer-term MEL support, which is a rare commodity. Through longer-term support, we can ensure that rigorous and suitable MEL systems are implemented, increase the likelihood appropriate data is collected, and heavily encourage organizations to act upon their findings. Currently, we receive more requests for support than we can address.

4. Most organizations lack the capacity to fully engage with MEL

Learning how to use MEL tools, setting up MEL systems, and implementing MEL all take time. Not many animal and vegan charities have dedicated staff hours to engage with MEL, and just a handful of the larger organizations employ full-time MEL staff. This makes it difficult to maintain a useful level of focus on MEL.

KPMG International, a large consultancy firm, surveyed 35 large organizations in the global development sector (M&E in the Development Sector 2014). They found their respondents spent between < 1 and 10% of their program budget on MEL. The majority of the respondents indicated a lack of resources to be the main challenge in improving evaluation systems.

Experts indicate spending between 3 and 20% of a program budget on MEL is deemed adequate for ensuring accountability, assessing effectiveness, and providing opportunities for learning and continuous improvement. The animal cause area is a long way from meeting this proportion.

5. MEL tools from other cause areas are useful for the animal cause area. However, there are cause-area-specific challenges.

When we started our interventions in September 2023 we were uncertain if MEL tools used in other cause areas were useful and applicable in the animal cause area. So far, we found they generally are.

There are cause-area-specific differences and challenges:

• Because of the scale of the interventions and often small sample sizes, quantitative MEL methods are not always suitable. We rely more heavily on qualitative methods but assume when the cause area grows, the balance will change.
• Compared to other sectors, the level of evidence in the animal cause area, whether through research or gained via practicing MEL, is still limited. This makes building robust theories of change with high-confidence assumptions difficult.
• The lack of case studies and role models hinders the use of and sometimes the willingness of animal-focused organizations to learn MEL.
• Similar to other cause areas, there are varied ideas of what roads lead to impact such as improvement of animal welfare, improvement of animal rights, increasing veganism, and changing ethical beliefs. However, where in the Global Health and Development space generally accepted metrics exist such as QALYs and DALYs, the currently used metrics in the animal cause area such as animal lives saved or averted, and welfare improvements are not clearly defined or accepted. This makes it more difficult to speak a common language and complicates impact measurement.

6. Funders can be a driving force for increased evidence-based work in the animal cause area. Factors such as funding MEL, requiring suitable MEL and a learning attitude can contribute to becoming that driving force.

When starting our activities, we assumed funders could serve as a driving force for increased evidence-based work in the animal cause area. We haven’t found any indicators to the contrary. We did gain insights into what factors could help to fulfill this role well.

Next to the perceived complexity of MEL, many organizations stated a lack of capacity as a reason to not or only in a limited way engage with MEL.

We also learned that some organizations do not believe the interventions they are implementing will have the greatest possible effect in their context, but sometimes choose to implement them anyway because there is funding available. Some other charities struggle with funders who are focused on continuously increasing outputs such as “number of people reached”, whereas the charities instead would like to gather data on early outcomes such as behavioral change.

We think the role of funders when it comes to MEL includes: 

• Funding increased capacity for MEL implementation, particularly because capacity is mentioned as one of the major bottlenecks.
• Only requiring suitable MEL from their beneficiaries.
• Embracing a learning attitude. This includes a focus on the “why” when positive or negative results come in, and providing ample space for pivots based on data and insights.
• Establishing good and preferably longer-term relations with their beneficiaries to encourage charities to also share less favorable results
   

7. Building rapport, involving the right staff, and understanding interventions are necessary ingredients for delivering useful MEL training and support. 

Over the past few months, we learned that a certain level of knowledge about the interventions helps to identify red flags, such as the risk of substitution or shifting bad animal welfare practices to other regions. Knowing about the organization and context helps us to ask the right questions.

We also noticed that building rapport with the people in a given organization helps to boost engagement and involve all relevant people from the organization.

We underestimated the importance of having the right staff members involved at the right time. Even though one of our selection criteria was the mandatory involvement of the approver of the interventions we support, we did not react adequately when in practice the approver did not attend. Not having the approver involved with the MEL process, jeopardized the ability for staff to make changes. We course-corrected after realizing this.

The organization also needs to be willing to involve other relevant staff members at certain points in the MEL design process. This can be particularly challenging for organizations that work with specialized departments or staff members involved in multiple interventions.

8. Our cohort model does not generate sufficient value.

For our twelve-month support program, we selected ten organizations based on criteria such as a learning attitude, size, and perceived capacity to reallocate resources if results showed some of their interventions were more or less effective. Eight organizations started the online program simultaneously.

Although working with a cohort generated some efficiency, it didn’t deliver the peer support between the participating organizations we aimed for. The cohort approach did not prove to be suitable for organizations with different structures, interventions, and contexts. We will therefore not repeat the cohort model in its original form. We will only consider a cohort model under specific circumstances such as involving different teams from one organization at once, or multiple organizations working on similar interventions in very similar contexts.

9. We still grapple with the question of how to narrow down our target organizations.
 

Geography
The contexts in which the participating organizations operate were more different than we anticipated, even between countries within the same region. This makes it difficult to assess the potential for impact. We currently think that selecting for a learning attitude and perceived capacity to reallocate resources, if needed, generates a bigger chance of impactful pivots than a specific geographical focus.

Size of the organization

Small changes in a large organization can potentially be more impactful than big changes in a small organization. On the other hand, changes are generally easier to make within small organizations, making MEL interventions more tractable.

We currently work with organizations, or branches of organizations, that have between four and fifteen staff members. Despite limited capacity, these relatively small organizations are advancing in the use of MEL. They designed theories of change, identified key assumptions and risks, and unified the collective understanding of their programs. As a result, several organizations identified key activities they wish to add to their program or have decided to focus their efforts on specific target groups.

We also interacted with several larger organizations that participate in our Community of MEL Practitioners or approached us with questions. Based on these interactions, we think larger organizations could also be suitable for the type of MEL support we provide. We will therefore explore the possibility of working with one large organization if it meets our selection criteria.
 

10. Working with organizations on MEL primarily requires tailor-made support. This makes our own cost-effectiveness challenging.

We think we can be most impactful if we work with “bandwagon” farmed animal welfare and vegan advocacy interventions. We define a bandwagon intervention as “an intervention that has been done a lot or is spreading rapidly to other organizations or contexts”. Examples of bandwagon interventions are cage-free campaigns, vegan challenges, and plant-based school- and university catering. We think it is particularly important to use MEL for bandwagon interventions as the spread of the interventions most likely means that resources spent on these interventions are relatively large or growing. 

We assume insights from one organization will spread to funders and other organizations. If they also act upon the insights, we amplify our effect. A prerequisite is that the interventions are not locked in by e.g. funding or expertise, and pivots within or between interventions are possible.
 

11. Using the right MEL tools at the right time is imperative to successful MEL.

Similar to MEL in other cause areas, we believe that MEL tools need to be tailored for different phases and types of interventions, and the size and stage of the organization. Using the wrong tools at the wrong time makes obtaining relevant data harder, leading to wasted time and faulty insights. For example:

• charities should focus more on basic monitoring and quick evaluations in the early days, and only in a later stage on impact evaluations
• conducting needs assessments and prototyping is more suitable for the intervention’s ideation phase, while conducting impact evaluations for seemingly successful interventions, is a suitable tool before scaling up

Stakeholders generally do not yet look at MEL as something that needs to be tailored to different phases. We have been approached by several organizations that would like to do impact evaluations while their interventions are not yet ready for this.

If you would like to know more about The Mission Motor’s work, please read our strategy and annual plan.

How you can advance the use of research and MEL in the animal cause area

If you are excited about Monitoring, Evaluation, and Learning, and about increasing the evidence base in the animal cause area, there are various things you can do.

• Look into MEL possibilities for your charity.
  • Join the MEL Slack sub-channel of Impactful Advocacy Advocacy (Hive), where we share best practices and learning opportunities. Here you can also ask for input and support from animal advocacy MEL practitioners.
  • Please also join when you are skeptical about MEL! We can benefit from hearing your reasoning and experience, and possibly help you to find the right tools for your interventions.
• Donate, volunteer, or apply with a research organization in the animal cause area.
  See this post about the benefits and limitations of animal advocacy research, and how you can get involved (ACE, 2023). See the Animal Advocacy Career job board for vacancies.
• Work with The Mission Motor
  We are looking for an entry-level MEL Associate. A generalist with good facilitation skills, who is eager to become a MEL specialist. We aim to fill this paid position as soon as possible.
• Volunteer with The Mission Motor
  We are looking for communication volunteers (2-4 hours per week). We aim to build a small group of volunteers who support our work on LinkedIn, Slack, the website, and the newsletter.

Thank you!

My co-director Blake and I want to express a huge thank you to all our supporters:

• Our advisors and partners: George Bridgwater (Animal Ask), Constance Li and Sofia Balderson (Impactful Animal Advocacy/Hive), Caryn Ginsberg (Priority Visions), and the entire AIM (Charity Entrepreneurship) community.
• Everyone in the EA and animal community who donated to The Mission Motor: AIM (Charity Entrepreneurship) seed funding network, Craigslist Charitable Foundation, EA Animal Welfare Fund, Effectiv Spenden, and several private donors.
• The Mission Motor’s supervisory board: Michelle Vellinga, Lauren Mee, Katrina Sill, Lauri van Oosterom, and Jesse van Elzelingen.

It takes a village :-)