Love these questions, and love talking nitty gritty of meta-analysis 😃 

1. IMHO, the "math hard" parts of meta-analysis are figuring out what questions you want to ask, what are sensible inclusion criteria, and what statistical models are appropriate. Asking how much time this takes is the same as asking, where do ideas come from?
2. The "bodybuilding hard" part of meta-analysis is finding literature. The evaluators didn't care for our search strategy, which you could charitably call "bespoke" and uncharitably call "ad hoc and fundamentally unreplicable." But either way, I read about 1000 papers closely enough to see if they qualified for inclusion, and then, partly to make sure I didn't duplicate my own efforts, I recorded notes on every study that looked appropriate but wasn't. I also read, or at least read the bibliographies of, about 160 previous reviews. Maybe you're a faster reader than I am, but ballpark I think I've already detailed >500 hours of work here.
3. Making everything computationally reproducible, e.g. writing the functions, checking my own work, setting things up to be generalizable -- a week of work in total? I'm not sure.
4. The paper went through many internal revisions and changed shape a lot from its initial draft when we pivoted in how we treated red and processed meat.  That's hundreds of hours. Peer review was probably another 40 hour workweek.
5. As I reread reviewer 2's comments today, it occurred to me that some of their ideas might be interesting test cases for what Claude Code is and is not capable of doing. I'm thinking particularly of trying to formally incorporate my subjective notes about uncertainty (e.g. the many places where I admit that the effect size estimates involved a lot of guesswork) into some kind of...supplementary regression term about how much weight an estimate should get in meta-analysis? Like maybe I'd use Wasserstein-2 distance, as my advisor Don recently proposed? Or Bayesian meta-analysis? This is an important problem. As my co-authors Don and Betsy & co. comment in a separate paper on which I was an RA:
   > Too often, research syntheses focus solely on estimating effect sizes, regardless of whether the treatments are realistic, the outcomes are assessed unobtrusively, and the key features of the experiment are presented in a transparent manner. Here we focus on what we term landmark studies, which are studies that are exceptionally well-designed and executed (regardless of what they discover). These studies provide a glimpse of what a meta-analysis would reveal if we could weight studies by quality as well as quantity.
6. It's possible that some of the proposed changes would take less time than that. Maybe risk of bias assessment could be knocked out in a week?. But it's been about a year since the relevant studies were in my working memory, which means I'd probably have to re-read them all, and across our main and supplementary dataset, that's dozens of papers. How long does it take you to read dozens of papers? I'd say I can read about 3-4 papers a day closely if I'm really, really cranking. So in all likelihood, yes, weeks of work, and that's weeks where I wouldn't be working on a project about building empathy for chickens. Which admittedly I'm procrastinating on by writing this 500+ word comment 😃 

David, there are two separate questions here, which is whether these analyses should be done or whether I should have done them in response to the evaluations.  If you think these analyses are worth doing, by all means, go ahead!

A final reflective note: David, I want to encourage you to think about the optics/politics of this exchange from the point of view of prospective Unjornal participants/authors. There are no incentives to participate. I did it because I thought it would be fun and I was wondering if anyone would have ideas or extensions that improved the paper. Instead, I got some rather harsh criticisms implying we should have written a totally different paper. Then I got this essay, which was unexpected/unannounced and used, again, rather harsh language to which I objected. Do you think this exchange looks like an appealing experience to others? I'd say the answer is probably not. 

A potential alternative: I took a grad school seminar where we replicated and extended other people's papers. Typically the assignment was to do the robustness checks in R or whatever, and then the author would come in and we'd discuss. It was a great setup. It worked because the grad students actually did the work, which provided an incentive to participate for authors. The co-teachers also pre-selected papers that they thought were reasonably high-quality, and I bet that if they got a student response like Matthew's, they would have counseled them to be much more conciliatory,  to remember that participation is voluntary, to think through the risks of making enemies (as I counseled in my original response), etc. I wonder if something like that would work here too. Like, the expectation is that reviewers will computationally reproduce the paper, conduct extensions and robustness checks, ask questions if they have them, work collaboratively with authors, and then publish a review summarizing the exchange. That would be enticing! Instead what I got here was like a second set of peer reviewers, and unusually harsh ones at that, and nobody likes peer review.

It might be the case that meta-analyses aren't good candidates for this kind of work, because the extensions/robustness checks would probably also have taken Matthew and the other responder weeks, e.g. a fine end of semester project for class credit but not a very enticing hobby.

Just a thought.  

For what it's worth, I thought David's characterization of the evaluations was totally fair, even a bit toned down. E.g. this is the headline finding of one of them:

major methodological issues undermine the study's validity. These include improper missing data handling, unnecessary exclusion of small studies, extensive guessing in effect size coding, lacking a serious risk-of-bias assessment, and excluding all-but-one outcome per study.

David characterizes these as "constructive and actionable insights and suggestions". I would say they are tantamount to asking for a new paper, especially the excluding of small studies, which was core to our design and would require a whole new search, which would take months. To me, it was obvious  that I was not going to do that (the paper had already been accepted for publication at that point). The remaining suggestions also implied dozens ( hundreds?) of hours of work. Spending weeks satisfying two critics didn't pass a cost-benefit test.^[1] It wasn't a close call.

1. ^{^}

    really need to follow my own advice now and go actually do other projects 😃

@geoffrey We'd love to run a megastudy! My lab put in a grant proposal with collaborators at a different Stanford lab to do just that but we ultimately went a different direction. Today, however, I generally believe that we don't even know what is the right question to be asking -- though if I had to choose one it would be, what ballot intiative does the most for animal welfare while also getting the highest levels of public support, e.g. is there some other low-hanging fruit equivalent to "cage free" like "no mutilation" that would be equally popular. But in general I think we're back to the drawing board in terms of figuring out what is the study we want to run and getting a version of it off the ground, before we start thinking about scaling up to tens of thousands of people. 

@david_reinstein, I suppose any press is good press so I should be happy that you are continuing to mull on the lessons of our paper 😃 but I am disappointed to see that the core point of my responses is not getting through. I'll frame it explicitly here: when we did one check and not another, or one one search protocol and not another, the reason, every single time, is opportunity costs.  When I say "we thought it made more sense to focus on the risks of bias that seemed most specific to this literature," I am using the word 'focus' deliberately, in the sense of "focus means saying no." In other words, because of opportunity costs, we are always triaging. At every juncture, navigating the explore/exploit dilemma requires judgment calls. You don't have to like that I said no to you, but it's not a false dichotomy, and I do not care for that characterization.

To the second question of whether anyone will do the kind of extension work, I personally see this as a great exercise for grad students. I did all kinds of replication and extension work in grad school. A deep dive into a subset of contact hypothesis literature I did in a political psychology class in 2014 , which started with a replication attempt, eventually morphed into The Contact Hypothesis Re-evaluated. If a grad student wanted to do this kind of project, please be in touch, I'd love to hear from you.

That's interesting, but not what I'm suggesting. I'm suggesting something that would, e.g., explain why you tell people to "ignore the signs of my estimates for the total welfare" when you share posts with them. That is a particular style and it says something about whether one should take your work in a literal spirit or not, which falls under the meta category of why you write the way you write; and to my earlier point, you're sharing this suggestion here with me in a comment rather than in the post itself 😃 Finally, the fact that there's a lot of uncertainty about whether wild animals have positive or negative lives is exactly the point I raised about why I have trouble engaging with your work. The meta post I am suggesting, by contrast, motivate and justify this style of reasoning as a whole, rather than providing a particular example of it. The post you've shared is a link in a broader chain. I'm suggesting you zoom out and explain what you like about this chain and why you're building it.

By all means, show us the way by doing it better 😃 I'd be happy to read more about where you are coming from, I think your work is interesting and if you are right, it has huge implications for all of us. 

Observations:

1. Echoing Richard's comment, EA is a community with communal norms, and a different forum might be a better fit for your style. Substack, for instance, is more likely to reward a confrontational approach. There is no moral valence to this observation, and likewise there is no moral valence to the EA community implicitly shunning you for not following its norms. We're talking about fit.
2. Pointing out "the irony of debating “AI rights” when basic human rights are still contested" is contrary to EA communal norms in several ways, e.g. it's not intended to persuade but rather to end/substantially redirect a conversation, its philosophical underpinnings have extremely broad and (I think to us) self-evidently absurd implications (should we bombard the Game of Thrones subreddit with messages about how people shouldn't be debating fiction when people are starving?), its tone was probably out of step with how we talk, etc. Downvoting a comment like that amounts to “this is not to my tastes and I want to talk about something else.”
3. "I started to notice a pattern — sameness in tone, sameness in structure, even sameness in thought. Ideas endlessly repackaged, reframed, and recycled. A sort of intellectual monoculture." This is a fairly standard EA criticism. Being an EA critic is a popular position. But I think you can trust that we've heard it before, responded before, etc. I am sympathetic to folks not wanting to do it again.

(Vasco asked me to take a look at this post and I am responding here.) 

Hi Vasco,

I've been taking a minute to reflect on what I want to say about this kind of project. A few different thoughts, at a few different levels of abstraction.

1. In the realm of politics, I'm glad the ACLU and FIRE exist, even if I don't agree with them on everything, because I think they're useful poles in the ecosystem. I feel similarly about your work: I think this kind of detailed cost-benefit work on non-standard issues, or on standard issues but that leads to non-standard conclusions, is a healthy contribution to EA, separately from whether I agree with or even understand it.
2. The main barrier to my engaging deeply with your work is that your analyses hinge on strong assumptions that I have no idea how to verify even in theory. The claim that nematodes live net-negative lives, for instance, which you believe with 55% confidence: I have no clue if this is true. I'm not even sure how many hours I would need to devote to it to form any belief on this whatsoever. (Hundreds?) In general, I have about 2-3 hours of good thinking per day.
3. I notice that the the top comment on this post seems to express the "EA consensus" about your style of analysis; I believe that because it has gotten more upvotes and such than the post itself. One lesson we might draw from this is that perhaps there is some persuasion work to be done get folks onboard with some of your assumptions, stylistic choices, and modes of analysis. Perhaps a post along the lines of "why I write the way I write" -- Nietzsche did this -- or "The moral philosophical assumptions underpinning my style of analysis" would go some of the way to bridging that gap.
   1. I get the sense that you are building an elaborate intellectual edifice whose many moving parts are distributed in many posts, comments, and external philosophical texts. That's well and good, I also have a "headcanon" about my work and ideas that I haven't fully systematized, e.g. I write almost exclusively about the results of randomized controlled trials without getting into the intellectual foundations of why I do that. But I think your intellectual foundations are more abstruse and counterintuitive. I'd I think folks might benefit from a meta post about them: a "start here to understand Vasco Grilo's writing" primer. Just an idea.
4. I am generally on board with using the EA forum as an extended job interview, e.g. establishing a reputation as someone who can reason clearly about an arbitrary subject. I think you're doing a fine job of that. On the other hand, the interaction with Kevin Xia about whether this work is appropriate for Hive, the downvotes that post received, and the fact that you are the only contributor to the soil animals topic here are face value evidence that writing about this topic as much as you do is not career-optimal. Perhaps it deserves its own forum: soilanimalsmatter.substack.com or something like that? And then you you can actually build up the whole intellectual edifice from foundations upwards. I do this (https://regressiontothemeat.substack.com/) and it is working for me. Just a thought.

I am amenable to this argument and generally skeptical of longtermism on practical grounds. (I have a lot of trouble thinking of someone 300-500 years ago plausibly doing anything with my interests in mind that actually makes a difference. Possible exceptions include folks associated with the Gloriois Revolution.)

I think the best counterargument is that it’s easier to set things on a good course than to course correct. Analogy: easier to found Google, capitalizing on advertisers’ complacency, than to fix advertising from within; easier to create Zoom than to get Microsoft to make Skype good. 

Im not saying this is right but I think that is how I would try to motivate working on longtermism if I did (work on longtermism).