Anyone know any Earn-To-Givers who might be interested in participating in an AMA during Giving Season? If a few are interested, it might be fun to experiment with an AMA panel, where Forum users ask questions, and any of the AMA co-authors can respond/ co-authors can disagree. Why? Giving Season is, in my opinion, a really great time to highlight the earn-to-give work which is ongoing all year, but is generally under-celebrated by the EA community. + Earn-to-givers might have good insights on how to pick donation targets during the donation election, and Giving Season more generally.   

I would advise being careful with RP's Cross-cause effectiveness tool as it currently stands, especially with regards to the chicken campaign. There appears to be a very clear conversion error which I've detailed in the edit to my comment here. I was also unable to replicate their default values from their source data, but I may be missing something. 

Re "pivotal questions"... Some thoughts on what The Unjournal (unjournal.org) can offer, cf existing EA-aligned research orgs (naturally, there are pros and cons) ... both in terms of defining and assessing the 'pivotal questions/claims', and in evaluating specific research findings that most inform these. 1. Non-EA-aligned expertise and engagement: We can offer mainstream (not-EA aligned) feedback and evaluation, consulting experts who might not normally come into this orbit. We can help engage non-EA academics in the priorities and considerations relevant to EAs and EA-adjacent orgs. This can leverage the tremendous academic/government infrastructure to increase the relevant research base. Our processes can provide 'outside the EA bubble' feedback and perhaps measure/build the credibility of EA-aligned work. 2. Depth and focus on specific research and research findings: Many EA ~research orgs focus on shallow research and comms. Some build models of value and cost-effectiveness targeted to EA priorities and 'axiology'. In contrast, Unjournal expert evaluations can dig deeply into the credibility of specific findings/claims that may be pivotal to these models. 3. Publicity, fostering public feedback and communication: The Unjournal is building systems for publishing and promoting our evaluations. We work to link these to the scholarly/bibliometric tools and measures people are familiar with. We hope this generates further feedback, public discussion, research, and application of this research.

I think that data ethics and drug ethics together or as separate social functions can save humanity at large.

I was surprised to find that I felt slightly uncomfortable positioning myself on the 'animal welfare' side of the debate week scale. I guess I generally think of myself as more of a 'global health & development' person, and might have subconscious concerns about this as an implicit affiliational exercise (even though I very much like and respect a lot of AW folks, I guess I probably feel more "at home" with GHD)? Obviously those kinds of personal factors shouldn't influence our judgments about an objective question like the debate week question is asking. But I guess they inevitably do. I don't know if this observation is even worth sharing, but there it is, fwiw. I guess I'd just like to encourage folks to be aware of their personal biases and try to bracket them as best they can. (I'd like to think of all EAs as ultimately "on the same side" even when we disagree about particular questions of cause prioritization, so I feel kind of bad that I evidently have separate mental categories of "GHD folks" and "AW folks" as though it were some kind of political/coalitional competition.)

I want to once again congratulate the forum team on this voting tool. I think by doing this, the EA forum is at the forefront of internal community discussions. No communities do this well and it's surprising how powerful it is. 

The 80,000 Hours team just published that "We now rank factory farming among the top problems in the world." I wonder if this is a coincidence or if this planned to coincide with the EA Forum's debate week? Combined with the current debate week's votes on where an extra $100 should be spent, these seem like nice data points to show to anyone that claims EA doesn't care about animals.  

I'm not really focused on animal rights nor do I spend much time thinking about it, so take this comment with a grain of salt. However, if I wanted to make the future go well for animals I'd be offering free vegan meals in the Bay Area or running a conference on how to ensure that the transition to advanced AI systems goes well for animals in the Bay Area. Reality check: Sorry for being harsh, but you're not going to end factory farming before the transition to advanced AI technologies. Max 1-2% chance of that happening. So the best thing to do is to ensure that this goes well for animals and not just humans. Anyway, that concludes my hot-take.

The real danger isn't just from AI getting better- it's from it getting good enough that humans start over-relying on it and offloading tasks to it.  Remember that Petrov had automatic detection systems, too; he just independently came to the conclusion not to fire nukes back.

EA needs more communications projects. Unfortunately, the EA Communications Fellowship and the EA Blog prize shut down[1]. Any new project needs to be adapted to the new funding environment. If someone wanted to start something in this vein, what I'd suggest would be something along the lines of AI Safety Camp. People would apply with a project to be project leads and then folk could apply to these projects. Projects would likely run over a few months, part-time remote[2]. Something like this would be relatively cheap as it would be possible for someone to run this on a volunteer basis, but it might also make sense for there to be a paid organiser at a certain point. 1. ^ Likely due to the collapse of FTX 2. ^ Despite the name, AI Safety Camp is now remote.

Does anyone know of a low-hassle way to charge invoices for services but it's a third-party charity that gets paid? It could well be an EA charity if that makes it easy. I'm hoping for something slightly more structured than "I'm not receiving any pay for my services but I'm trusting you to donate X amount to this charity instead".

Thinking of trying to re-host innovationsinfundraising.org, which I stopped hosting maybe a year ago. Not sure I have the bandwidth to keep it updated as a ~living literature review, but the content might be helpful to people. You can see some of the key content on the wayback machine, e.g., the table of evidence/consideration of potential tools .  Any thoughts/interest in using this or collaborating on a revival (focused on the effective giving part)? This, along with the barriers to effective giving might (or might not) also be a candidate for Open Phil's living literature project. (The latter is still hosted, some overlaps with @Lucius Caviola  and @Stefan_Schubert's book).

I was going through Animal Charity Evaluators' reasoning behind which countries to prioritize (https://animalcharityevaluators.org/charity-review/the-humane-league/#prioritizing-countries) and I notice they judge countries with a higher GNI per capita as more tractable. This goes against my intuition, because my guess is your money goes further in countries that are poorer. And also because I've heard animal rights work in Latin America and Asia is more cost-effective nowadays. Does anyone have any hypotheses/arguments? This quick take isn't meant as criticism, I'm just informing myself as I'm trying to choose an animal welfare org to fundraise for this week (small, low stakes). When I have more time I'd be happy to do more research and contact ACE myself with these questions, but right now I'm just looking for some quick thoughts.

From Reuters: I sincerely hope OpenPhil (or Effective Ventures, or both - I don't know the minutia here) sues over this. Read the reasoning for and details of the $30M grant here.  The case for a legal challenge seems hugely overdetermined to me: * Stop/delay/complicate the restructuring, and otherwise make life appropriately hard for Sam Altman * Settle for a large huge amount of money that can be used to do a huge amount of good * Signal that you can't just blatantly take advantage of OpenPhil/EV/EA as you please without appropriate challenge I know OpenPhil has a pretty hands-off ethos and vibe; this shouldn't stop them from acting with integrity when hands-on legal action is clearly warranted

The Debate Week banner is under construction... get your takes ready for Monday morning! In the mean time, you can brush up on the reading list in the announcement post, or respond to this post with ideas for posts you'd like to see next week. 

The CEA Online Team (which runs this Forum) has finalized our OKRs for this new half-quarter, and I've updated the public doc, so I'm reposting the link to the doc here.

In America, dining services influence far more meals than vegans' personal consumption Aramark, Compass Group, etc. each serve billions of meals annually, and even the largest individual correctional facilities, hospital campuses, school districts, public universities, baseball stadiums, etc. each serve millions of meals annually to largely captive audiences

A note on how I think about criticism (This was initially meant as part of this post,[1] but while editing I thought it didn't make a lot of sense there, so I pulled it out.) I came to CEA with a very pro-criticism attitude. My experience there reinforced those views in some ways,[2] but it also left me more attuned to the costs of criticism (or of some pro-criticism attitudes). (For instance, I used to see engaging with all criticism as virtuous, and have changed my mind on that.) My overall takes now aren’t very crisp or easily summarizable, but I figured I'd try to share some notes. ... It’s generally good for a community’s culture to encourage criticism, but this is more complicated than I used to think. Here’s a list of things that I believe about criticism: 1. Criticism or critical information can be extremely valuable. It can be hard for people to surface criticism (e.g. because they fear repercussions), which means criticism tends to be undersupplied.[3] Requiring critics to present their criticisms in specific ways will likely stifle at least some valuable criticism. It can be hard to get yourself to engage with criticism of your work or things you care about. It’s easy to dismiss true and important criticism without noticing that you’re doing it.  1. → Making sure that your community’s culture appreciates criticism (and earnest engagement with it), tries to avoid dismissing critical content based on stylistic or other non-fundamental qualities, encourages people to engage with it, and disincentivizes attempts to suppress it can be a good way to counteract these issues.  2. At the same time, trying to actually do anything is really hard.[4] Appreciation for doers is often undersupplied. Being in leadership positions or engaging in public discussions is a valuable service, but opens you up to a lot of (often stressful) criticism, which acts as a disincentive for being public. Psychological safety is important in teams (and communities), so it’s u

A note on mistakes and how we relate to them (This was initially meant as part of this post[1], but I thought it didn't make a lot of sense there, so I pulled it out.) “Slow-rolling mistakes” are usually much more important to identify than “point-in-time blunders,”[2] but the latter tend to be more obvious. When we think about “mistakes”, we usually imagine replying-all when we meant to reply only to the sender, using the wrong input in an analysis, including broken hyperlinks in a piece of media, missing a deadline, etc. I tend to feel pretty horrible when I notice that I've made a mistake like this. I now think that basically none of my mistakes of this kind — I’ll call them “Point-in-time blunders” — mattered nearly as much as other "mistakes" I've made by doing things like planning my time poorly, delaying for too long on something, setting up poor systems, or focusing on the wrong things.  This second kind of mistake — let’s use the phrase “slow-rolling mistakes” — is harder to catch; I think sometimes I'd identify them by noticing a nagging worry, or by having multiple conversations with someone who disagreed with me (and slowly changing my mind), or by seriously reflecting on my work or on feedback I'd received.  ... This is not a novel insight, but I think it was an important thing for me to realize. Working at CEA helped move me in this direction. A big factor in this, I think, was the support and reassurance I got from people I worked with.  This was over two years ago, but I still remember my stomach dropping when I realized that instead of using “EA Forum Digest #84” as the subject line for the 84th Digest, I had used “...#85.” Then I did it AGAIN a few weeks later (instead of #89). I’ve screenshotted Ben’s (my manager’s) reaction. ... I discussed some related topics in a short EAG talk I gave last year, and also touched on these topics in my post about “invisible impact loss”.  An image from that talk. 1. ^ It was there because

I've been looking at the numbers with regards to how many GPUs it would take to train a model with as many parameters as the human brain has synapses. The human brain has 100 trillion synapses, and they are sparse and very efficiently connected. A regular AI model fully connects every neuron in a given layer to every neuron in the previous layer, so that would be less efficient. The average H100 has 80 GB of VRAM, so assuming that each parameter is 32 bits, then you have about 20 billion per GPU. So, you'd need 10,000 GPUs to fit a single instance of a human brain in RAM, maybe. If you assume inefficiencies and need to have data in memory as well you could ballpark another order of magnitude so 100,000 might be needed. For comparison, it's widely believed that OpenAI trained GPT4 on about 10,000 A100s that Microsoft let them use from their Azure supercomputer, most likely the one listed as third most powerful in the world by the Top500 list. Recently though, Microsoft and Meta have both moved to acquire more GPUs that put them in the 100,000 range, and Elon Musk's X.ai recently managed to get a 100,000 H100 GPU supercomputer online in Memphis. So, in theory at least, we are nearly at the point where they can train a human brain sized model in terms of memory. However, keep in mind that training such a model would take a ton of compute time. I haven't done to calculations yet for FLOPS so I don't know if it's feasible yet. Just some quick back of the envelope analysis.

When EA Lund does tabling at student association fairs, one thing that's gotten a laugh out of some people is having two plates of cookies people can take from. One of them gets a sticky saying "this cookie saves one (1) life", and the other gets a sticky saying "this cookie saves 100 lives!" 

I'd love to see the EA forum add a section titled "Get Involved" or something similar. There is the groups directory, but it's one of only many ways that folks can get more involved, from EAGx Conferences, to Virtual Programs, 80,000 Hours content/courses to donating.

I quickly wrote up some rough project ideas for ARENA and LASR participants, so I figured I'd share them here as well. I am happy to discuss these ideas and potentially collaborate on some of them. Alignment Project Ideas (Oct 2, 2024) 1. Improving "A Multimodal Automated Interpretability Agent" (MAIA) Overview MAIA (Multimodal Automated Interpretability Agent) is a system designed to help users understand AI models by combining human-like experimentation flexibility with automated scalability. It answers user queries about AI system components by iteratively generating hypotheses, designing and running experiments, observing outcomes, and updating hypotheses. MAIA uses a vision-language model (GPT-4V, at the time) backbone equipped with an API of interpretability experiment tools. This modular system can address both "macroscopic" questions (e.g., identifying systematic biases in model predictions) and "microscopic" questions (e.g., describing individual features) with simple query modifications. This project aims to improve MAIA's ability to either answer macroscopic questions or microscopic questions on vision models. 2. Making "A Multimodal Automated Interpretability Agent" (MAIA) work with LLMs MAIA is focused on vision models, so this project aims to create a MAIA-like setup, but for the interpretability of LLMs. Given that this would require creating a new setup for language models, it would make sense to come up with simple interpretability benchmark examples to test MAIA-LLM. The easiest way to do this would be to either look for existing LLM interpretability benchmarks or create one based on interpretability results we've already verified (would be ideal to have a ground truth). Ideally, the examples in the benchmark would be simple, but new enough that the LLM has not seen them in its training data. 3. Testing the robustness of Critique-out-Loud Reward (CLoud) Models Critique-out-Loud reward models are reward models that can reason explici

It's important to think about the policy space from a meta-level incentives/factors that might get in the way of having an impact, such as making AI safer. One I heard today was that policy people thrive in moments of regulatory uncertainty, while this is bad for companies.

Some quick responses to Nuño’s article about EA Forum stewardship I work on the CEA Online Team, which runs the Forum, but I am only speaking for myself. Others on my team may disagree with me. I wrote this relatively quickly so I wouldn’t be surprised if I changed my mind on things upon reflection. Overall, I really appreciated Nuño’s article, and did not find it to be harsh, overly confrontational, or unpleasant to read. I appreciated the nice messages that he included to me, a person working on the Forum, at the start and end of the piece. On the design change and addition of features: People have different aesthetic preferences, and I personally think the current Forum design looks nicer than the 2018 version, plus I think it has better usability in various ways. I like minimalism in some contexts, but I care more about doing good than about making the Forum visually pleasing to me. To that end, I think it is correct for the Forum to have more than just a simple frontpage list of posts plus a “recent discussion” feed (which seems to be the entirety of the 2018 version). For example, I think adding the “quick takes” and “popular comments” sections to the home page have been really successful. By making quick takes more salient, we’ve encouraged additional discussions on the site (since quick takes are intended for less polished posts). “Popular comments” helps to highlight ongoing discussions in posts that may not be visible elsewhere. I take the fact that LessWrong borrowed these sections for their site as further evidence of their value, and in fact LessWrong has had some impactful discussions happen in their quick takes section. As another example, features like the “Groups directory” and the “People directory” are not available anywhere else online, and I view them as more like “essential infrastructure for the EA community”. I think it’s reasonable for those to live on the EA Forum, where people already gather to talk about EA things and look for some E

Around discussions of AI & Forecasting, there seems to be some assumption like: 1. Right now, humans are better than AIs at judgemental forecasting. 2. When humans are better than AIs at forecasting, AIs are useless. 3. At some point, AIs will be better than humans at forecasting. 4. At that point, when it comes to forecasting, humans will be useless. This comes from a lot of discussion and some research comparing "humans" to "AIs" in forecasting tournaments. As you might expect, I think this model is incredibly naive. To me, it's asking questions like, "Are AIs better than humans at writing code?" "Are AIs better than humans at trading stocks?" "Are AIs better than humans at doing operations work?" I think it should be very clear that there's a huge period, in each cluster, where it makes sense for humans and AIs to overlap. "Forecasting" is not one homogeneous and singular activity, and neither is programming, stock trading, or doing ops. There's no clear line for automating "forecasting" - there are instead a very long list of different skills one could automate, with a long tail of tasks that would get increasingly expensive to automate.  Autonomous driving is another similar example. There's a very long road between "helping drivers with driver-assist features" and "complete level-5 automation, to the extent that almost no human are no longer driving for work purposes." A much better model is a more nuanced one. Break things down into smaller chunks, and figure out where and how AIs could best augment or replace humans at each of those. Or just spend a lot of time working with human forecasting teams to augment parts of their workflows.

I'm exploring the possibility of building an alignment research organization focused on augmenting alignment researchers and progressively automating alignment research (yes, I have thought deeply about differential progress and other concerns). I intend to seek funding in the next few months, and I'd like to chat with people interested in this kind of work, especially great research engineers and full-stack engineers who might want to cofound such an organization. If you or anyone you know might want to chat, let me know! Send me a DM, and I can send you some initial details about the organization's vision. Here are some things I'm looking for in potential co-founders: Need * Strong software engineering skills Nice-to-have * Experience in designing LLM agent pipelines with tool-use * Experience in full-stack development * Experience in scalable alignment research approaches (automated interpretability/evals/red-teaming)

Thoughts on the applicant feedback problem. Could job application / grant application / whatever application feedback be outsourced? Lack of feedback maybe causes a lack of improvement in applications (for jobs, grants, whatever) and keeps the entire pool of applicants (and therefore effective altruists) less competitive. I wonder under what conditions people could benefit from one another's feedback being visible. For example if I applied for a job and was knocked back (e.g. an EA job) and not given any feedback, how could my information gap be filled in novel ways? 1. Maybe outsource feedback to different people such as a curated community of HR people incentivised to contribute to a feedback pool so that future job applicants are more prepared  2. Maybe outsource feedback to an AI using specific training and specific templates so that questions are designed to surface blind spots 3. Maybe outsource feedback to structured industry specific mentorship / networking organisations  4. Maybe the person who didn't hire me could visibly publish one detailed constructive criticism publicly, clearly deidentified or depersonalised Considerations: 1. Maybe organisations have strong incentives to not show constructive criticism 1. Perhaps those incentives are more important places to intervene or problem solve? 2. Maybe the cost of publishing high quality constructive criticism, done effectively with the right parameters, would offer enough return to be worth it 1. E.g. a grants organisation publishing 'top 10 reasons we didn't go with x proposal' 1. Is this too controversial? Could the balance be struck between informative and helpful without being condescending or offensive?  If more people had more access to more quality feedback, would that likely improve 'things'?