Thanks for putting this together, it’s a really nice intro. (Particularly like the attribution waterfall in 9.4!). I think it's worth being really clear that much of evaluation in practice is judgement, and judgement takes experience. E.g., each multiplier in 9.4 (deadweight, credit share, implementation rate) is a call that experienced evaluators make by drawing on comparable cases, stakeholder knowledge and a feel for where might data mislead. I wouldn’t actually expect someone new to the field to be able to “estimate counterfactual impact and full, marginal cost” etc from a guide alone. Pointing readers to in-depth resources and training could help.

A couple of other suggestions from running MEL for an advocacy org:

1. Ch 9.5 could say more about what qualitative approaches give you beyond a credit share. Contribution analysis, or related methods like process tracing or the General Elimination Methodology, don’t just test whether you plausibly contributed to an outcome. They also show why and how it happened: which mechanisms worked, which didn’t, and what else was going on. For advocacy that’s often the most decision-relevant part, because it tells you what to do differently next time, not just how much to claim.
2. The cost of measurement to implementers. Ch 24 covers what to measure but not the reporting burden it creates. For small advocacy orgs that cost is often the binding constraint, so a “proportionate MEL” note would help.

Agree with a lot here! Before ProVeg I spent years in academic behaviour change research, across conservation, climate communication and plant-based advocacy. Some thoughts:

On behavioural data: ugh, this was the thing I chafed at most. It could be surprisingly hard to persuade colleagues that we should be measuring what people actually do, rather than settling for attitudes and intentions. Real behaviour is harder to get at and often means using proxies, but designing a good proxy can be one of the more creative and enjoyable parts of a study! There are also well-established tools that get you much closer to behaviour without a full field trial, e.g. multiple price lists from experimental economics, where participants make a series of choices with real money at stake.

On practitioner relevance: I think one of the most useful things a PhD student or early career researcher can do is work closely with practitioners, whether through consultancies, placements or collaborations. Writing for a practitioner audience is a very different skill from writing for reviewers, and it rarely gets taught.

On publishing: agreed. In my experience, the more practical the research, the less prestigious the journal it ends up in (field dependent of course).

On connecting with movements: it’s environmental rather than animal activism, but getting involved with Extinction Rebellion was rewarding in several ways. It was personally uplifting, I could see the research I did with them having a real impact, and it opened up career opportunities I wouldn’t otherwise have had. I’d encourage more researchers to get involved with the movements they study.

Final note, when I was in conservation there was real resistance to me working on plant-based advocacy at all. On a personal level, I was often the only vegan in the department, which probably didn’t help.

Ah no, we didn’t, though results from a perceived-probability question may have been interesting. Whatever participants believed about their chances should be the same across arms, so I'm confident in the relative ranking of modules even if the absolute $10 figure may be an upper limit.

No, they were told 20 people would be selected to receive their choice for real, but not how many people were in the study overall.

Amazing to see! Publishing your full costs, preregistering the next round and committing to track non-attendees is fantastic, and I hope other talent programmes follow suit.

Fair point on the “massively oversubscribed” condition. Historically you didn’t have a big pool of near misses, but with a ~15% acceptance rate in 2026, it's likely you do now? A few low-cost steps taken at the point of selection would make future comparisons much stronger:

• Record applicant scores. If reviewers give each applicant a numeric score (even a rough one), you can later compare people just above and just below the cutoff. That gets you close to a regression discontinuity design without changing who you admit. It would also show, as OscarD says, whether your selection judgements predict career outcomes.
• Measure both groups the same way. Like you said, the response rate for the comparison group will probably be lower. Still worth doing, but I’d make sure to code current role from public profiles for everyone (both alumni and comparison group), ideally by someone who doesn’t know which group each person is in.
• Check the dropouts are comparable before relying on them. Illness is plausibly random. Visa problems probably aren’t: they will cluster by nationality, and therefore by region, and you’ve shown that region strongly affects outcomes. Comparing visa dropouts with alumni from the same bootcamp, or at least the same region, would deal with most of that.

The comparison group also gives you a way to test the self-reported counterfactual. Your respondents imply that about half of their AI safety work is down to ML4Good (0.53). If waitlisted applicants turn out to be working in AI safety at roughly half the rate of alumni, the self-reports hold up well! If waitlisted applicants are nearly as likely to be in AI safety, that you can calculate how much to discount the self-report. (Useful for every programme in the field.)

Really glad to see this. Happy to look over the comparison design if useful.

Good to see this open. I’ve put some thoughts in a quick take on what my previous evaluation of Blackfish suggests about when films lead to change. If any grantees want help working out how to evaluate their film’s impact, I’d be glad to talk it through.

With Who Let the Docs Out now taking applications, here are a couple of quick thoughts from an evaluation of Blackfish.

Blackfish is the film everyone points to. A few years ago I looked at whether it actually caused SeaWorld’s troubles, using stock market data and interviews with 26 people from marine parks, zoo associations, animal welfare NGOs, the film crew, marine science and narrative research. (Blackfish paper; plain-English write-up in The Conversation). We used General Elimination Methodology to test the competing explanations. Some didn’t hold up. In 2013 SeaWorld’s Orlando and California parks lost visitors while Universal and Disney gained them, so tighter household budgets can’t explain the drop.

Blackfish did contribute to the fall in attendance and share price, the redesigned orca show and the end of the breeding programme. Interviewees put its impact down to four things:

• Distribution. CNN aired it roughly 40 to 50 times, and the CNN branding made it read as news to a lot of viewers.
• Content. Empathy for Tilikum, plus anger at a company that had not been straight with the public.
• Timing. It landed on decades of campaigning, along with Free Willy, The Cove and the book Death at SeaWorld, and campaigners were ready to use it.
• SeaWorld’s own response, which was slow and defensive.

Some advice for farmed animal documentary filmmakers:

1. Ask who will pick the film up and use it. Blackfish had a single company to target and campaigns ready to go.
2. Budget for distribution as seriously as production.
3. Views and social engagement tell you about reach. Plan from the start how you’d find out whether anything changed, even if the answer is qualitative.

Happy to talk to any filmmakers or funders thinking about evaluation.

Nice to see the causal chain out laid out!

The counterfactual impact of (1) is that individuals "do good" they otherwise wouldn't have, but not necessarily that more good is done in the world overall. That relies on the pool being talent-constrained, so jobs would either sit empty, or be filled by demonstrably less effective candidates.

I would lean to the employer-side evaluation. Might be the costly option, but it's more informative, and could potentially be done via asking hiring managers at the point of hire, who else was in the final set and what they would have done if this candidate had not applied. That's close to a one-question survey. 

Also worth surveying hiring managers on what are the biggest blockers - e.g., contextual knowledge? Specific skill sets? Could be useful for lighter touch evaluations of the programme down the road (i.e., are participants actually leaving with better contextual knowledge?)
 

I'm less sure about putting a money value on months saved early on - quite a few assumptions would be involved. 

I would definitely recommend pre-registering the outcome definitions and the analysis before anyone sees the data, timestamped somewhere public. Ideally the outcome coding would also be done by someone blind whether participants were part of the cohort or not.

I think the wartime framing actually argues for the retrospective design - you wouldn't need to randomise anything, ask a mentor to take a fellow they did not choose, or make a single cohort worse.