1st year PhD student in Agricultural and Resource Economics at Berkeley. Likes animal welfare, development economics and impact evaluation. Past lives at World Bank, IMF, and doing software engineering.
geoffrey
Agricultural and Resource Economics PhD @ Berkeley
Hey! Wanna reiterate something I said in the other question you asked.
You need to talk to someone in real life with richer knowledge about the universities you’re considering and ideally someone’s who has gone down the same path. Internet advice is going to reflect people's intuitions about CS and medical school in the US and UK.
Rankings are immensely contextual. In some contexts, top 5 versus top 10 has no meaning. In others, it’s a big jump. In rare cases (though I think these are exaggerated), it can be better to get mentorship from the lower ranked place.
And the ranking likely reflects things that aren't what you're optimizing towards. For example, have CS professors mentored people outside their university? How is the AI safety community of grad students and bachelors? How much of as structure is there for networking? Is either school known for connections to Bay Area? These things can be very idiosyncratic
My hunch is it doesn't matter. Master programs, especially in policy, tend to be cash cows. That's not to say they're worthless. But you are paying money for the network, credential, and access to a cool career. So admission standards are lower. It wouldn't surprise me if some internationally recognized exam like GRE took outsized weight in admissions.
When it does matter, it's not obvious to me that (1) is worst. The more prestigious Master's programs (typically those with full funding) do try to train their admissions staff to recognize differences in preparation across schools. And I wouldn't discount peer effects or building general cognitive endurance through a challenging curriculum.
Finally, I really don't know what middle-tier and top-tier mean here. And I think you should try to find a few people who took some Taiwan -> US master's transition to get their sense of where the ranking falls / how much it matters. I'm familiar with "tiers" in a few contexts: (1) NYC high schools, (2) US college admissions, and (3) Econ PhD admissions. There's no portable rule that works for all cases. Sometimes, the quality difference drops really sharply at some rank. Other times, the quality difference is flat through the distribution. Other times, the real quality distinction is within-a-school rather than across schools (e.g. the prestige at a school comes from a specific program than only half the applicants get accepted into)
A quick comment jumping off the $>20M of evidence collection work...
I've always been curious how much money has been spent to-date on global health research. If we make an (unrealistic) assumption that we could repeat GiveWell for animals, then we'd also have to fund the research that came before. Getting some kind of number, even an ultra-lower bound, would help anchor discussions on evidence-generation in animal welfare.
A starting point could be working backwards from malaria nets. What are the studies that are cited in GiveWell's spreadsheet model for malaria nets? How much did each of those studies cost? How much did it cost to train the researchers who conducted those studies? What's the bottom-line number we get from adding up all this up?
Would love to hear if this exists / see someone research it
Until recently, I always had the impression that there was a glut of animal activists and there'd be little point in me participating. It's not something I ever bothered to check.
For one thing, I heard plenty of stories of how hard it is to get a job at an animal organization. So I figured that would be same for animal activism and that each campaign was saturated with volunteers.
And for another thing, I usually don't hear about pressure campaigns unless they're successful or have tons of people. Understandably nobody wants to promote the mediocre attempts where there's only 3 people pressuring a target and failing. Activist campaigns want to seem like a cool group on the cusp of victory while newspapers have better stories to run.
It wasn't until I said all this to an organizer a few weeks ago that I learned all the struggles with recruiting and retaining activists. And then I made a mental note to join the next action I could.
I don't know how many people would benefit from hearing, "Hey our capacity is actually very low and we'd benefit from any non-crazy person with a pulse", but I certaintly did. And now I'm among the irregular participants who show up at least once or twice a year.
For what it's worth, I read that abstract as saying something like, "within the class of interventions studied so far, the literature has yet to settle onto any intervention that can reliably reduce animal product consumption by a meaningful amount, where meaningful amount might be a 1% reduction at Costco scale or long-term 10% reduction at a single cafeteria. The class of interventions being studied tends to be informational and nudge-style interventions like advertising, menu design, and media pamphlets. When effect sizes differ for a given type of intervention, the literature has not offered a convincing reason why a menu-design choice works in one setting versus another."
Okay, now that I've typed that up, I can see why "unsolved problem" is unclear.
And I'm probably taking a lot of leaps of faith in interpretation here
Seth, for what it's worth, I found your hourly estimates (provided in these forum comments but not something I saw in the evaluator response) on how long the extensions would take to be illuminating. Very rough numbers like this meta-analysis taking 1000 hours for you or a robustness check taking dozens / hundreds of hours more to do properly helps contextualize how reasonable the critiques are.
It's easy for me (even now while pursuing research, but especially before when I was merely consuming it) to think these changes would take a few days.
It's also gives me insight into the research production process. How long does it take to do a meta-analysis? How much does rigor cost? How much insight does rigor buy? What insight is possible given current studies? Questions like that help me figure out whether a project is worth pursuing and whether it's compatible with career incentives or more of a non-promotable task
Would you recommend that I share any such posts with both the authors and the evaluators before making them?
Yes. But zooming back out, I don't know if these EA Forum posts are necessary.
A practice I saw i4replication (or some other replication lab) is that the editors didn't provide any "value-added" commentary on any given paper. At least, I didn't see these in any tweets they did. They link to the evaluation reports + a response from the author and then leave it at that.
Once in a while, there will be a retrospective on how the replications are going as a whole. But I think they refrain from commenting on any paper.
If I had to rationalize why they did that, my guess is that replications are already an opt-in thing with lots of downside. And psychologically, editor commentary has a lot more potential for unpleasantness. Peer review tends to be anonymous so it doesn't feel as personal because the critics are kept secret. But editor commentary isn't secret...actually feels personal, and editors tend to have more clout.
Basically, I think the bar for an editor commentary post like this should be even higher than the usual process. And the usual evaluation process already allows for author review and response. So I think a "value-added" post like this should pass a higher bar of diplomacy and insight.
Chiming in here with my outsider impressions on how fair the process seems
@david_reinstein If I were to rank the evaluator reports, evaluation summary, and the EA Forum post in which ones seemed the most fair, I would have ranked the Forum post last. It wasn't until I clicked through to the evaluation reports that I felt the process wasn't so cutting.
Let me focus on one very specific framing in the Forum post, since it feels representative. One heading includes the phrase "this meta-analysis is not rigorous enough". This has a few connotations that you probably didn't mean. One, this meta-analysis is much worse than others. Two, the claims are questionable. Three, there's a universally correct level of quality that meta-analyses should reach and anything that falls short of that is inadmissible as evidence.
In reality, it seems this meta-analysis is par for the course in terms of quality. And it was probably more difficult to do so given the heterogeneity in the literature. And the central claim of the meta-analysis doesn't seem like something either evaluator disputed (though one evaluator was hesitant).
Again, I know that's not what you meant and there are many caveats throughout the post. But it's one of a few editorial choices that make the Forum post seem much more critical than the evaluation reports, which is a bit unusual since the Evaluators are the ones who are actually critiquing the paper.
Finally, one piece of context that felt odd not to mention was the fundamental difficulty of finding an expert in both food consumption and meta-analysis. That limits the ability of any reviewer to make a fair evaluation. This is acknowledged at the bottom of the Evaluation Summary. Elsewhere, I'm not sure where it's said. Without that mentioned, I think it's easy for a casual reader to leave thinking the two Evaluators are the "most correct".
Really enjoyed this. Not much public debate in this space as far as I can see. To two of your cruxes:
Is meta-analysis even useful in these contexts, with heterogeneous interventions, outcomes, and analytical approaches?
Will anyone actually do/fund/reward rigorous continued work?
I've sometimes wondered if it'd be worth funding a "mega study" like Milkman et al. (2021). They tested 54 different interventions to boost exercise among 61,000 members. Something similar for meat reduction could allow for some clean apples-to-apples comparisons.
I've seen the number $2.6 million floating around for how much this intervention costs. Granted, that's probably on top of convincing the mega-team of researchers to work on the project, which might only happen through the prestige of an academic lab. But it's also not an astronomical cost. And there'd be still some learning value from a smaller set of interventions and a smaller sample.
This might be a better use of resources than striving for the "ideal" meta-analysis, since that sounds expensive too.
Hey! Wanna reiterate something I said in the other question you asked.
You need to talk to someone in real life with richer knowledge about the universities you’re considering and ideally someone’s who has gone down the same path. Internet advice is going to reflect people's intuitions about CS and medical school in the US and UK.
Rankings are immensely contextual. In some contexts, top 5 versus top 10 has no meaning. In others, it’s a big jump. In rare cases (though I think these are exaggerated), it can be better to get mentorship from the lower ranked place.
And the ranking likely reflects things that aren't what you're optimizing towards. For example, have CS professors mentored people outside their university? How is the AI safety community of grad students and bachelors? How much of as structure is there for networking? Is either school known for connections to Bay Area? These things can be very idiosyncratic
My hunch is it doesn't matter. Master programs, especially in policy, tend to be cash cows. That's not to say they're worthless. But you are paying money for the network, credential, and access to a cool career. So admission standards are lower. It wouldn't surprise me if some internationally recognized exam like GRE took outsized weight in admissions.
When it does matter, it's not obvious to me that (1) is worst. The more prestigious Master's programs (typically those with full funding) do try to train their admissions staff to recognize differences in preparation across schools. And I wouldn't discount peer effects or building general cognitive endurance through a challenging curriculum.
Finally, I really don't know what middle-tier and top-tier mean here. And I think you should try to find a few people who took some Taiwan -> US master's transition to get their sense of where the ranking falls / how much it matters. I'm familiar with "tiers" in a few contexts: (1) NYC high schools, (2) US college admissions, and (3) Econ PhD admissions. There's no portable rule that works for all cases. Sometimes, the quality difference drops really sharply at some rank. Other times, the quality difference is flat through the distribution. Other times, the real quality distinction is within-a-school rather than across schools (e.g. the prestige at a school comes from a specific program than only half the applicants get accepted into)
A quick comment jumping off the $>20M of evidence collection work...
I've always been curious how much money has been spent to-date on global health research. If we make an (unrealistic) assumption that we could repeat GiveWell for animals, then we'd also have to fund the research that came before. Getting some kind of number, even an ultra-lower bound, would help anchor discussions on evidence-generation in animal welfare.
A starting point could be working backwards from malaria nets. What are the studies that are cited in GiveWell's spreadsheet model for malaria nets? How much did each of those studies cost? How much did it cost to train the researchers who conducted those studies? What's the bottom-line number we get from adding up all this up?
Would love to hear if this exists / see someone research it
Until recently, I always had the impression that there was a glut of animal activists and there'd be little point in me participating. It's not something I ever bothered to check.
For one thing, I heard plenty of stories of how hard it is to get a job at an animal organization. So I figured that would be same for animal activism and that each campaign was saturated with volunteers.
And for another thing, I usually don't hear about pressure campaigns unless they're successful or have tons of people. Understandably nobody wants to promote the mediocre attempts where there's only 3 people pressuring a target and failing. Activist campaigns want to seem like a cool group on the cusp of victory while newspapers have better stories to run.
It wasn't until I said all this to an organizer a few weeks ago that I learned all the struggles with recruiting and retaining activists. And then I made a mental note to join the next action I could.
I don't know how many people would benefit from hearing, "Hey our capacity is actually very low and we'd benefit from any non-crazy person with a pulse", but I certaintly did. And now I'm among the irregular participants who show up at least once or twice a year.
Don’t have answers but just wanted to say I really appreciate this mini-compilation of what’s already been done
For what it's worth, I read that abstract as saying something like, "within the class of interventions studied so far, the literature has yet to settle onto any intervention that can reliably reduce animal product consumption by a meaningful amount, where meaningful amount might be a 1% reduction at Costco scale or long-term 10% reduction at a single cafeteria. The class of interventions being studied tends to be informational and nudge-style interventions like advertising, menu design, and media pamphlets. When effect sizes differ for a given type of intervention, the literature has not offered a convincing reason why a menu-design choice works in one setting versus another."
Okay, now that I've typed that up, I can see why "unsolved problem" is unclear.
And I'm probably taking a lot of leaps of faith in interpretation here
Seth, for what it's worth, I found your hourly estimates (provided in these forum comments but not something I saw in the evaluator response) on how long the extensions would take to be illuminating. Very rough numbers like this meta-analysis taking 1000 hours for you or a robustness check taking dozens / hundreds of hours more to do properly helps contextualize how reasonable the critiques are.
It's easy for me (even now while pursuing research, but especially before when I was merely consuming it) to think these changes would take a few days.
It's also gives me insight into the research production process. How long does it take to do a meta-analysis? How much does rigor cost? How much insight does rigor buy? What insight is possible given current studies? Questions like that help me figure out whether a project is worth pursuing and whether it's compatible with career incentives or more of a non-promotable task
Yes. But zooming back out, I don't know if these EA Forum posts are necessary.
A practice I saw i4replication (or some other replication lab) is that the editors didn't provide any "value-added" commentary on any given paper. At least, I didn't see these in any tweets they did. They link to the evaluation reports + a response from the author and then leave it at that.
Once in a while, there will be a retrospective on how the replications are going as a whole. But I think they refrain from commenting on any paper.
If I had to rationalize why they did that, my guess is that replications are already an opt-in thing with lots of downside. And psychologically, editor commentary has a lot more potential for unpleasantness. Peer review tends to be anonymous so it doesn't feel as personal because the critics are kept secret. But editor commentary isn't secret...actually feels personal, and editors tend to have more clout.
Basically, I think the bar for an editor commentary post like this should be even higher than the usual process. And the usual evaluation process already allows for author review and response. So I think a "value-added" post like this should pass a higher bar of diplomacy and insight.
Chiming in here with my outsider impressions on how fair the process seems
@david_reinstein If I were to rank the evaluator reports, evaluation summary, and the EA Forum post in which ones seemed the most fair, I would have ranked the Forum post last. It wasn't until I clicked through to the evaluation reports that I felt the process wasn't so cutting.
Let me focus on one very specific framing in the Forum post, since it feels representative. One heading includes the phrase "this meta-analysis is not rigorous enough". This has a few connotations that you probably didn't mean. One, this meta-analysis is much worse than others. Two, the claims are questionable. Three, there's a universally correct level of quality that meta-analyses should reach and anything that falls short of that is inadmissible as evidence.
In reality, it seems this meta-analysis is par for the course in terms of quality. And it was probably more difficult to do so given the heterogeneity in the literature. And the central claim of the meta-analysis doesn't seem like something either evaluator disputed (though one evaluator was hesitant).
Again, I know that's not what you meant and there are many caveats throughout the post. But it's one of a few editorial choices that make the Forum post seem much more critical than the evaluation reports, which is a bit unusual since the Evaluators are the ones who are actually critiquing the paper.
Finally, one piece of context that felt odd not to mention was the fundamental difficulty of finding an expert in both food consumption and meta-analysis. That limits the ability of any reviewer to make a fair evaluation. This is acknowledged at the bottom of the Evaluation Summary. Elsewhere, I'm not sure where it's said. Without that mentioned, I think it's easy for a casual reader to leave thinking the two Evaluators are the "most correct".
Really enjoyed this. Not much public debate in this space as far as I can see. To two of your cruxes:
I've sometimes wondered if it'd be worth funding a "mega study" like Milkman et al. (2021). They tested 54 different interventions to boost exercise among 61,000 members. Something similar for meat reduction could allow for some clean apples-to-apples comparisons.
I've seen the number $2.6 million floating around for how much this intervention costs. Granted, that's probably on top of convincing the mega-team of researchers to work on the project, which might only happen through the prestige of an academic lab. But it's also not an astronomical cost. And there'd be still some learning value from a smaller set of interventions and a smaller sample.
This might be a better use of resources than striving for the "ideal" meta-analysis, since that sounds expensive too.