Yeah this would be a big investment (though I’d actually push against “tens of millions” for a single idea. My naïve expectations currently sit more between $200k and $1m for per idea reviewed if we optimise our processes well).
I think the thing the movement needs to understand is that the EA global development model is just not available to us. Groups like Evidence Action or the organisations recommended by Give Well are predicated on decades of research and piloting that have honed their understanding of what works both in terms of ask and approach. Think of Against Malaria Foundation - that works because someone figured out how to make nets, then someone figured out how to treat them with insecticide, and then many people have spent many years figuring out how to deliver things to people in a way that they will use them.
We don’t have any of that. So if we want evidenced programs, we need to invest in building a body of evidence. We just unfortunately can’t have our cake and eat it too on this one.
Hey Jacob! Great thoughts. Adaption is actually something that IPA’s Staged Based Learning systems tackles in it’s own separate section. They link to some useful resources at the end of their section. In general, this is a conundrum a lot of development experts have dealt with and built systems for.
To answer your direct question: You’re right that we cannot conclude from a model farms working in one region that it’ll work elsewhere. There is a separate process of evaluating an intervention’s fit in a new context, and how different the context is informs how in-depth you need to do that evaluation.
Though I would say that, at this stage in the animal movement’s lifecycle, I am more concerned about finding programs that work in any context. I think of it kinda like: A prerequisite for an idea to scale well across contexts is that it works in at least one context. This doesn’t necessarily mean that we shouldn’t consider translatability in our research processes and idea prioritisations. But I’d say we’ve got a lot of shit to figure out as it is, and I think it’s okay for adaptability to take a back seat for now.
Yeah, I think there is a thin line sometimes between not overwhelming newcomers and not misleading them.
Honestly, if you'd told me how hard Fish Welfare Initiative would be, I'm not sure that past Tom would have chosen to found it (though I am extremely glad I did!). Now I struggle with how to communicate with the new founders.
Though I would say it is definitely a mistake for bought-in advocates like yourself not to be clear on the realistic timelines/likelihoods of success. And I would also say that it is possible for people to get excited about longer-term pushes where milestones are smaller but realistic.
And also, there is a place for optimism and pushing for great things that others don't think are possible. So there's a balance to all of this.
Yes! I think we agree here, but we're framing things differently. We would say that mass investment, even in speculative gains, is not necessarily a bad thing (though we're also not saying it's necessarily a good thing).
However, we would say that such a mass investment should be done with our eyes open. We define a bubble specifically as "a situation where expected impact in a given timeframe substantially exceeds realistic impact". It is the mismatch of expectations (in both the upswing and the downswing) that we mean to point to as causing harm to the movement.
1. "Isn't most of the focus [for cultivated meat] already on development?" Yes, it is. However, our experience was that during the peak hype period, clean meat was communicated as almost ready to be on store shelves worldwide. So we are arguing that there is pressure for ideas to be presented as close to last-mile delivery, even when this is not the case.
2. "I don't see why [leafletting] was named as a 'bubble'" For us, a bubble just refers to "a situation where expected impact in a given timeframe substantially exceeds realistic impact", so it does not necessitate that the downward adjustment is wrong. Though maybe your point is more "why include it if the downward adjustment was justified?". To that, we would say there is damage caused on the upswing and the downswing. In this instance, the damage was in the detraction from other strategies and the unnecessary disillusionment. Also, we meant for the historical examples to be more illustrative of the pattern, and definitely don't mean to be disparaging to those efforts or decisions.
3. "I think one needs to make an actual argument at the object level for why there could be an overcorrection towards welfare tech" Interesting point for clarification! In part, we mean for this post to be a flag, giving a name to a pattern and signalling that welfare tech shares many of the hallmarks. Similarly, we don't mean to say that any given effort is necessarily a mistake. That said, we do think that there is a significant amount of resources currently being directed towards welfare tech (for example here, here, here) in ways that are oriented towards a belief that welfare tech is either ready for implementation and scaling or close to it. That could totally be right in some instances! But it could also be wrong. Even the fact that we are talking about welfare tech as a monolith is, for us, a signal that it is being simplified in the way that we describe as part of the cycle of a bubble.
Thanks for posting :) I’m coming at this as someone who spent a lot of time running on-farm research at Fish Welfare Initiative (and planning to do more through my new charity). I broadly agree, but I’d add a few caveats:
The core issue with farms as “welfare labs” is heterogeneity (variability). Especially in LMICs, farms are messy, uncontrolled environments where confounding variables easily creep in. That creates a lot of statistical noise. If you’re aiming for high certainty, farms can make that difficult.
Relatedly, on-farm research makes it harder to isolate specific effects. You mention the benefit of insights collected “under real commercial conditions”, and I agree that ultimately effectiveness in the real world is what matters. But there’s also strong value in isolating variables to understand mechanisms. If we’re testing whether pigs prefer straw or wood shavings, we may not want to simultaneously capture differences in how farmers manage those materials. Otherwise, when results don’t support our hypothesis, we don’t know whether we’re observing animal preference or management differences. That’s why the difference between efficacy (does it work in controlled conditions?) and effectiveness (does it work in real-world conditions?) is useful.
So in sum: on-farm research is a valuable tool, but it can’t replace controlled research, and I’d hesitate to frame it as more valuable.
I also think two ideas may be getting conflated: monitoring existing farm conditions vs running experiments on farms. Both can be useful, but both need a clear use-case for the data. I agree, though, that there’s strong potential on both fronts.
Strong agree. And I understand why this is a problem. It can be hard to independently create contacts in these spaces from scratch, and there is an aspect of not knowing what you don't know at play. I'm almost certain I am committing the same mistake in multiple places in my work.
Would be interested to think about solutions here. Like perhaps a group such as Consultants For Impact could take on a role of knowledge dispersal, doing things like getting project management experts to give a talks at EAGs?
I would say that EAs are missing large parts of M&E, including: - The formal setting of key questions / assumptions that form the basis of what you will focus on trying to answer - Creating formal monitoring frameworks (e.g. a log frame) that takes these questions / assumptions and identifies practical indicators and a method of measuring them - I think EAs don't use the full diversity of M&E tools. In my experience we tend to over-index on surveys (vs., say, interviews, focus group discussion, or observational data) - I think considering the frequency of use of surveys, we could generally up-skill in high-quality survey design - Using a diverse set of evaluation types (EAs generally know about RCTs, but these are such a narrow slice of the available evaluation types)
In general I think we care about M&E but lack experience in the formal processes of it, especially monitoring. So application is patchy and not generally in line with best practices.
I should perhaps clarify that I am mostly talking about the non-global development side of EA. I think their norms for M&E are significantly better.
Intrac's M&E universe is one place to see an overview of what M&E entails. I think also The Mission Motor intends to create more resources on these topics in the future :)
Yeah this would be a big investment (though I’d actually push against “tens of millions” for a single idea. My naïve expectations currently sit more between $200k and $1m for per idea reviewed if we optimise our processes well).
I think the thing the movement needs to understand is that the EA global development model is just not available to us. Groups like Evidence Action or the organisations recommended by Give Well are predicated on decades of research and piloting that have honed their understanding of what works both in terms of ask and approach. Think of Against Malaria Foundation - that works because someone figured out how to make nets, then someone figured out how to treat them with insecticide, and then many people have spent many years figuring out how to deliver things to people in a way that they will use them.
We don’t have any of that. So if we want evidenced programs, we need to invest in building a body of evidence. We just unfortunately can’t have our cake and eat it too on this one.
I like your specific suggestion!
Hey Jacob!
Great thoughts. Adaption is actually something that IPA’s Staged Based Learning systems tackles in it’s own separate section. They link to some useful resources at the end of their section. In general, this is a conundrum a lot of development experts have dealt with and built systems for.
To answer your direct question: You’re right that we cannot conclude from a model farms working in one region that it’ll work elsewhere. There is a separate process of evaluating an intervention’s fit in a new context, and how different the context is informs how in-depth you need to do that evaluation.
Though I would say that, at this stage in the animal movement’s lifecycle, I am more concerned about finding programs that work in any context. I think of it kinda like: A prerequisite for an idea to scale well across contexts is that it works in at least one context. This doesn’t necessarily mean that we shouldn’t consider translatability in our research processes and idea prioritisations. But I’d say we’ve got a lot of shit to figure out as it is, and I think it’s okay for adaptability to take a back seat for now.
Yeah, I think there is a thin line sometimes between not overwhelming newcomers and not misleading them.
Honestly, if you'd told me how hard Fish Welfare Initiative would be, I'm not sure that past Tom would have chosen to found it (though I am extremely glad I did!). Now I struggle with how to communicate with the new founders.
Though I would say it is definitely a mistake for bought-in advocates like yourself not to be clear on the realistic timelines/likelihoods of success. And I would also say that it is possible for people to get excited about longer-term pushes where milestones are smaller but realistic.
And also, there is a place for optimism and pushing for great things that others don't think are possible. So there's a balance to all of this.
Hey Aidan :)
Yes! I think we agree here, but we're framing things differently.
We would say that mass investment, even in speculative gains, is not necessarily a bad thing (though we're also not saying it's necessarily a good thing).
However, we would say that such a mass investment should be done with our eyes open. We define a bubble specifically as "a situation where expected impact in a given timeframe substantially exceeds realistic impact". It is the mismatch of expectations (in both the upswing and the downswing) that we mean to point to as causing harm to the movement.
Thanks for the clarification!
I added the word "potentially" to the relevant section to make it more accurate.
Hey Caleb!
Going through each point in turn:
1. "Isn't most of the focus [for cultivated meat] already on development?"
Yes, it is. However, our experience was that during the peak hype period, clean meat was communicated as almost ready to be on store shelves worldwide.
So we are arguing that there is pressure for ideas to be presented as close to last-mile delivery, even when this is not the case.
2. "I don't see why [leafletting] was named as a 'bubble'"
For us, a bubble just refers to "a situation where expected impact in a given timeframe substantially exceeds realistic impact", so it does not necessitate that the downward adjustment is wrong.
Though maybe your point is more "why include it if the downward adjustment was justified?".
To that, we would say there is damage caused on the upswing and the downswing. In this instance, the damage was in the detraction from other strategies and the unnecessary disillusionment.
Also, we meant for the historical examples to be more illustrative of the pattern, and definitely don't mean to be disparaging to those efforts or decisions.
3. "I think one needs to make an actual argument at the object level for why there could be an overcorrection towards welfare tech"
Interesting point for clarification!
In part, we mean for this post to be a flag, giving a name to a pattern and signalling that welfare tech shares many of the hallmarks. Similarly, we don't mean to say that any given effort is necessarily a mistake.
That said, we do think that there is a significant amount of resources currently being directed towards welfare tech (for example here, here, here) in ways that are oriented towards a belief that welfare tech is either ready for implementation and scaling or close to it. That could totally be right in some instances! But it could also be wrong. Even the fact that we are talking about welfare tech as a monolith is, for us, a signal that it is being simplified in the way that we describe as part of the cycle of a bubble.
Aaron!
Thanks for posting :) I’m coming at this as someone who spent a lot of time running on-farm research at Fish Welfare Initiative (and planning to do more through my new charity). I broadly agree, but I’d add a few caveats:
The core issue with farms as “welfare labs” is heterogeneity (variability). Especially in LMICs, farms are messy, uncontrolled environments where confounding variables easily creep in. That creates a lot of statistical noise. If you’re aiming for high certainty, farms can make that difficult.
Relatedly, on-farm research makes it harder to isolate specific effects. You mention the benefit of insights collected “under real commercial conditions”, and I agree that ultimately effectiveness in the real world is what matters. But there’s also strong value in isolating variables to understand mechanisms. If we’re testing whether pigs prefer straw or wood shavings, we may not want to simultaneously capture differences in how farmers manage those materials. Otherwise, when results don’t support our hypothesis, we don’t know whether we’re observing animal preference or management differences. That’s why the difference between efficacy (does it work in controlled conditions?) and effectiveness (does it work in real-world conditions?) is useful.
So in sum: on-farm research is a valuable tool, but it can’t replace controlled research, and I’d hesitate to frame it as more valuable.
I also think two ideas may be getting conflated: monitoring existing farm conditions vs running experiments on farms. Both can be useful, but both need a clear use-case for the data. I agree, though, that there’s strong potential on both fronts.
Cool! Would be keen to sign on to a mailing list :)
Strong agree.
And I understand why this is a problem. It can be hard to independently create contacts in these spaces from scratch, and there is an aspect of not knowing what you don't know at play. I'm almost certain I am committing the same mistake in multiple places in my work.
Would be interested to think about solutions here. Like perhaps a group such as Consultants For Impact could take on a role of knowledge dispersal, doing things like getting project management experts to give a talks at EAGs?
For sure!
I would say that EAs are missing large parts of M&E, including:
- The formal setting of key questions / assumptions that form the basis of what you will focus on trying to answer
- Creating formal monitoring frameworks (e.g. a log frame) that takes these questions / assumptions and identifies practical indicators and a method of measuring them
- I think EAs don't use the full diversity of M&E tools. In my experience we tend to over-index on surveys (vs., say, interviews, focus group discussion, or observational data)
- I think considering the frequency of use of surveys, we could generally up-skill in high-quality survey design
- Using a diverse set of evaluation types (EAs generally know about RCTs, but these are such a narrow slice of the available evaluation types)
In general I think we care about M&E but lack experience in the formal processes of it, especially monitoring. So application is patchy and not generally in line with best practices.
I should perhaps clarify that I am mostly talking about the non-global development side of EA. I think their norms for M&E are significantly better.
Intrac's M&E universe is one place to see an overview of what M&E entails. I think also The Mission Motor intends to create more resources on these topics in the future :)