calebp's Quick takes

calebp

calebp

2w

12

Is there a way for me to filter posts below say 15 karma from my frontage? I couldn’t easily find it on mobile.

NunoSempere

1w

2

https://forum.nunosempere.com/frontpage?minScore=15

Toby Tremlett🔹

2w

2

I don't think so. Best bet is to use the all posts page, weekly, sort by karma. Then close your eyes when you scroll down to posts below 15 karma.

What was the use case here?

calebp

2w

2

When I posted, the front page felt "spammy", with lots of generic, low-quality posts, presumably from first-time posters. I now rarely click on anything below 15 karma, but that was >50% of my frontpage.

Toby Tremlett🔹

2w*

2

Thanks! That's great to hear because it's a problem I'm thinking about right now.
Obviously we can't just not show low karma posts to people because every post was once low karma (and every author too).

One solution, which I think I mentioned to you before, is to have a more curated and streamlined frontpage, for those who want to only see the best of the (recent) Forum. Below is a mock-up:

Would something like this solve your problem do you think? Posts which appeared on this page would be hand chosen, with a similar bar to the Digest, but updated daily.

PS- I know it is bad product discovery to just ask lol. I still think you get some evidence from a strong no or a strong yes.

calebp

2w*

9

It doesn't appeal to me because I want higher info density, and this reduces it - I could see others liking it .

It's not obvious to me that you can't just hide low karma posts as an option - some options:
1. All first-time posts are reviewed by the forum team, who can upvote things and put them above the threshold
2. Make the low karma/low user karma post hiding disabled by default
3. Try out an option for a bit and remove it if it's not working well

Toby Tremlett🔹

2w

2

We'll definitely be doing (3), hopefully in the next few weeks.

"It doesn't appeal to me because I want higher info density" - fair!

How about a normal list of posts but still on a 'featured' page. In other words, these would be the best posts, chosen by the Forum team.

Sort of like the Digest, but all the time, and without the organisation updates and most of community, and with the addition of good recent discussion.

calebp

2w

2

I could see that being good. I think I'd still prefer the filter-out-under-n karma option because I understand what's actually going on under the hood. I expect I'd prefer it to the current front page.

abrahamrowe

2w

2

I'd be in favor of something like ">15 karma + many curated new posts by the moderation team" as Caleb suggested below as a filtering option to try to resolve the issue of people just filtering out 100% of new posts and new posts never rising. E.g. I imagine you all could often whitelist certain users or just glance at a post and tell that it is worth pushing through without reading it (and generally I feel pro-heavier moderation on the Forum especially if it is focused on maintaining quality).

Toby Tremlett🔹

2w

2

I imagine you all could often whitelist certain users or just glance at a post and tell that it is worth pushing through without reading it

Yep this is true. Though it's perhaps not reassuring to new authors lol.

">15 karma + many curated new posts by the moderation team"

My current plan is >100 karma, and loads of 'featured' posts by the moderation team for the featured page, and then keeping the frontpage as is. Once I've built this thing there will be a lot of room for experimentation with the algorithm though.

Toby Tremlett🔹

2w

2

Update- looks like this seemed like a particular issue this week because some part of the refactor broke the algo. Should be fixed now.

calebp

9mo*

48

(Weakly) Against 1:1 Fests

I just returned from EAG NYC, which exceeded my expectations - it might have been the most useful and enjoyable EAG for me so far.

Ofc, it wouldn’t be an EAG without inexperienced event organisers complaining about features of the conference (without mentioning it in the feedback form), so to continue that long tradition here is an anti-1:1s take.

EAGs are focused on 1:1s to a pretty extreme degree. It’s common for my friends to have 10-15 30 minute 1:1s per day, at other conferences I’ve been to it’s generally more like 0-5. I would prefer a culture of closer to 5 1:1s per day, with half of them organised after the conference starts.

Some upsides of my imaginary system relative to the current system:

Far less tiring for attendees
Far more opportunity to use earlier conversations to inform later ones (e.g. you could have a new project idea, then talk to a collaborator about it, then secure funding all at the conference)
More opportunities for small group conversations, which are extremely hard to organise in Swapcard and, in my opinion, are much more valuable than 1:1s.
Less planning overhead, where you need to start booking meetings very early so that people still have time in their calendars (and regular pre-con visits to swapcard to see who has recently joined the platform)

Some concrete recommendations to try at the next EAG could be:
Figure out how to make group conversations easier to organise (maybe ditch swapcard and online makes their own platform??)
Block out every 2nd or 3rd session by default.
Create nice zones for spontaneous conversations (not sure how to do this well) or set up the space with more nooks for organic conversations (or maybe have high effort after parties with more of this vibe)
Encourage attendees to keep at least half the schedule free till after the first day.

I’m not sure what actions I plan to take at an individual level, it feels like it’s hard for me to realise something like the above vision just for myself. Some options that I feel pretty good about trying include:

budget 3x more time for scheduling and make more small group conversations happen
block out lots of time in swapcard (though it’s not super useful if others don’t do this too)
think of lower downside interventions and lobby the EAG team to try them out - one issue is that I’m not sure I have any interventions that result in better feedback form scores in the short-term - even if the change is better in the long term.

P.S. Thanks again to the EAG team for another excellent conference!

Jan_Kulveit

9mo

15

My impression is EAGx Prague 22 managed to balance 1:1s with other content simply by not offering SwapCard 1:1s slots part of the time, having a lot of spaces for small group conversations, and suggesting to attendees they should aim for something like balanced diet. (Turning off SwapCard slots does not prevent people from scheduling 1:1, just adds a little friction; empirically it seems enough to prevent the mode where people just fill their time by 1:1s).

As far as I understand this will most likely not happen, because weight given to / goodharting on metrics like people reporting 1:1s is the most valuable use of time, metrics tracking "connections formed" and weird psychological effect of 1:1 fests. (People feel stimulated, connected, energized,... Part of the effect is superficial). Also the counterfactual value lost from lack of conversational energy at scales ~3 to 12ppl is not visible and likely not tracked in feedback (I think this has predictable effects on what types of collaborations do start and which do not, and the effect is on the margin bad.) The whole is downstream of problems like Don't Over-Optimize Things / We can do better than argmax.

Btw I think you are too apologetic / self-deprecating ("inexperienced event organisers complaining about features of the conference"). I have decent experience running events and all what you wrote is spot on.

Jordan Pieters 🔸

8mo*

1

Thanks Jan, I appreciate this comment. I'm on the EAG team, but responding with my personal thoughts.

While it's true that we weight 1:1s heavily in assessing EAG, I don't think we're doing 'argmax prioritisation'—we still run talks, workshops, meetups, and ~1/4 of our team time goes to this. My read of your argument is that we're scoring things wrong and should give more consideration to the impact of group conversation. You're right that we don't currently explicitly track the impact of group conversations, which could mean we're missing significant value.

I do plan to think more about how we can make these group conversations happen and measure their success. I haven't yet heard a suggestion (in this thread or elsewhere) that I believe would sufficiently move the needle, but maybe this is because we're over-optimising for better feedback survey scores in the short term (e.g., we'll upset some attendees if we turn off specific 1:1 slots).

Saul Munn

9mo

14

(written v quickly, sorry for informal tone/etc)

i think that a happy medium is getting small-group conversations (that are useful, effective, etc) of size 3–4 people. this includes 1-1s, but the vibe of a Formal, Thirty Minute One on One is a very different vibe from floating through 10–15, 3–4-person conversations in a day, each that last varying amounts of time.

much more information can flow with 3-4 ppl than with just 2 ppl
people can dip in and out of small conversations more than they can with 1-1s
more-organic time blocks means that particularly unhelpful conversations can end after 5-10m, and particularly helpful ones can last the duration that would be good for them to last (even many hours!)
3-4 person conversations naturally select for a good 1-1. once 1-2 people have left a 3-4 person conversation, the conversation is then just a 1-1 of the two people who've engaged in the conversation longest — which seems like some evidence of their being a good match for a 1-1.

however, i think that this is operationally much harder to do for organizers than just 1-1s. my understanding is that this is much of the reason EAGs (& other conferences) do 1-1s, instead of small group conversations.

i think Writehaven did a mediocre job of this at LessOnline this past year (but, tbc, it did vastly better than any other piece of software i've encountered).
i think Lighthaven as a venue forces this sort of thing to happen, since there are so so so many nooks for 2-4 people to sit and chat, and the space is set up to make 10+ person conversations less likely to happen.

i know that The Curve (from @Rachel Weinberg) created some "Curated Conversations:" they manually selected people to have predetermined conversations for some set amount of time. iirc this was typically 3-6 people for ~1h, but i could be wrong on the details. rachel: how did these end up going, relative to the cost of putting them together?

Chris Leong

9mo*

5

Create nice zones for spontaneous conversations (not sure how to do this well)

I've tried pushing for this without much success unfortunately.

It really is a lot more effort to have spontaneous conversations when almost all pairs are a one-on-one and almost all people by themselves are waiting for a one-on-one.

I've seen attempts to declare a space an area that's not for one-on-ones, but people have one-on-ones there anyway. Then again, organisers normally put up one or two small signs.

Honestly, the only way to stop people having one-on-ones in the area for spontaneous conversation might be to have an absurd number of big and obvious signs.

calebp

9mo*

5

Yeah I also think hanging out in a no 1:1s area is weirdly low status/unexciting. I’d be a bit more excited about cause or interest specific areas like “talk about ambitious project ideas”.

claerer

9mo

4

I completely agree and have tried so set up some small group meetings - you can have up to 24 people in one 1:1 meeting on swapcard. This works especially well if you create a 1:1 with one person and then add others rather than immediately creating one with 24 people because the latter does not show you whether invitees are actually free at the planned time while the former does.

On the organiser side, it might be cool to move the cause area/work specific meetups earlier in the conference.

Mick

9mo*

3

I've "only" been to 2 EAGs and 4 EAGx's so take this with that as context

For previous EAGs I always booked my schedule full of 1-1's to ask people about their experience, resolve uncertainties, and just generally network with people in similar roles. This EAG (NYC 2025) I didn't find as many people on Swapcard that I wanted to talk to and received much less requests for 1-1s, so I also ended up having just 7 1-1s in total. This was a fun experiment. I found it much more relaxed, and I enjoyed being able to have spontaneous conversations with people I ran into, but I think overall I got less value out of this EAG than if I had booked more meetings: I have less actionable insights and met less people than during other EAG(x) conferences I have attended. However, I'm definitely in favour of less 1-1 cramming.

I do think if this was one of my first EAGs and I didn't know anyone, I would've been quite lost without the structure of the 1-1's and the explicit encouragement that it is normal to book a lot. I also feel weird about just joining a conversation in case it was people having a private 1-1. Having an improved spontaneous conversations area with bigger signs/cause area specific areas (or time slots?) sounds like a great solution for both of these problems.

Tangentially, my favourite meetups are also those where you just stand and mingle, ideally with specific areas in specific corners, rather than do forced speed meets or roundtable discussions. This makes it much easier to leave if you don't like a conversation and move on to a different one until you find one you like.

calebp

1y*

49

(edit: I was mostly thinking of public criticism of EA projects - particularly projects with <10 FTE. This isn't clear from the post. )

I think it’s wild how pro-criticism of projects the EA forum is when:

Most people agree that there are a lack of good projects, and public critique clearly creates barriers to starting new projects
Almost all EA projects have low downside risk in absolute terms
There are almost no examples of criticism clearly mattering (e.g. getting someone to significantly improve their project)
Criticism obviously driving away valuable people from the forum - like (at least in part) the largest ever EA donor
(Less important, but on priors, I’m not sure you should expect high-quality criticism of EA projects because they are often neglected, and most useful criticism comes from people who have operated in a similar area before. )

Edit: I'm curious about counterexamples or points against any of the bullets. There are lots of disagree reacts, and presumably some of those people have seen critques that were actually useful.

Rohin Shah

1y*

78

I'm not especially pro-criticism but this seems way overstated.

Almost all EA projects have low downside risk in absolute terms

I might agree with this on a technicality, in that depending on your bar or standard, I could imagine agreeing that almost all EA projects (at least for more speculative causes) have negligible impact in absolute terms.

But presumably you mean that almost all EA projects are such that their plausible good outcomes are way bigger in magnitude than their plausible bad outcomes, or something like that. This seems false, e.g.

FTX
Any kind of political action can backfire if a different political party gains power
AI safety research could be used as a form of safety washing
AI evaluations could primarily end up as a mechanism to speed up timelines (not saying that's necessarily bad, but certainly under some models it's very bad)
Movement building can kill the movement by making it too diffuse and regressing to the mean, and by creating opponents to the movement
Vegan advocacy could polarize people, such that factory farming lasts longer than it would be default (e.g. if cheap and tasty substitutes would have caused people to switch over if they weren't polarized)

There are almost no examples of criticism clearly mattering

ChatGPT can talk, but OpenAI employees sure can’t
Habryka on Anthropic non-disparagements
FrontierMath was funded by OpenAI
Concerns with Intentional Insights
It's hard to tell, but I'd guess Critiques of Prominent AI Safety Labs changed who applied to the critiqued organizations
Gossip-based criticism of Leverage clearly mattered and imo it would have been better if it was more public
Sharing Information About Nonlinear clearly mattered in the sense of having some impact, though the sign is unclear
Same deal for Why did CEA buy Wytham Abbey?
Back in the era when EA discussions happened mainly on Facebook there were all sorts of critiques and flame wars between protest-tactics and incremental-change-tactics for animal advocacy, I don't think this particularly changed what any given organization tried to do, but it surely changed views of individual people

I'd be happy to endorse something like "public criticism rarely causes an organization to choose to do something different in a major org-defining way" (but note that's primarily because people in a good position to change an organization through criticism will just do so privately, not because criticism is totally ineffective).

calebp

1y*

9

on

Almost all EA projects have low downside risk in absolute terms

I agree with some of the points on point 1, though other than FTX, I don't think the downside risk of any of those examples is very large. I'd walk back my claim to the downside risk to most EA projects seems low (but there are ofc exceptions).

on

There are almost no examples of criticism clearly mattering

Agree that criticisms of AI companies can be good, I don't really consider them EA projects but it wasn't clear that was what I was referring to in my post - my bad. Responding quickly to some of the other ones.

Concerns with Intentional Insights
- This seems good, though it was a long time ago.
It's hard to tell, but I'd guess Critiques of Prominent AI Safety Labs changed who applied to the critiqued organizations
- Idk if these are "EA" projects. I think I'm much more pessimistic than you are that these posts made better things happen in the world. I'd guess that people overupdated on these somewhat. That said, I quite like these posts and the discussion in the commentts.
Gossip-based criticism of Leverage clearly mattered and imo it would have been better if it was more public
- This also seems good, though it was a long time ago and I wasn't around when leverage was a thing.
Sharing Information About Nonlinear clearly mattered in the sense of having some impact, though the sign is unclear
- Sign seems pretty negative to me.
Same deal for Why did CEA buy Wytham Abbey?
- Sign seems pretty negative to me. Like even the title is misleading and this generated a lot of drama.
Back in the era when EA discussions happened mainly on Facebook there were all sorts of critiques and flame wars between protest-tactics and incremental-change-tactics for animal advocacy, I don't think this particularly changed what any given organization tried to do, but it surely changed views of individual people
- Not familiar but maybe this is useful? Idk.
- Open Phil and RP both had pieces that were pretty critical of clean meat work iirc that were large updates for me. I don't think they were org-level critiques, but I could imagine a version of them being critiques of GFI.

So overall, I think I stand by the claim that there aren't many criticisms that clearly mattered, but this was a positive update for me. Maybe I should have said that a very small fraction of critical EA forum posts have clear positive effects or give people useful information.

This was a great comment - thanks for writing it.

Rohin Shah

1y*

14

I agree with some of the points on point 1, though other than FTX, I don't think the downside risk of any of those examples is very large

Fwiw I find it pretty plausible that lots of political action and movement building for the sake of movement building has indeed had a large negative impact, such that I feel uncertain about whether I should shut it all down if I had the option to do so (if I set aside concerns like unilateralism). I also feel similarly about particular examples of AI safety research but definitely not for the field as a whole.

Agree that criticisms of AI companies can be good, I don't really consider them EA projects but it wasn't clear that was what I was referring to in my post

Fair enough for the first two, but I was thinking of the FrontierMath thing as mostly a critique of Epoch, not of OpenAI, tbc, and that's the sense in which it mattered -- Epoch made changes, afaik OpenAI did not. Epoch is at least an EA-adjacent project.

Sign seems pretty negative to me.

I agree that if I had to guess I'd say that the sign seems negative for both of the things you say it is negative for, but I am uncertain about it, particularly because of people standing behind a version of the critique (e.g. Habryka for the Nonlinear one, Alexander Berger for the Wytham Abbey one, though certainly in the latter case it's a very different critique than what the original post said).

I think I stand by the claim that there aren't many criticisms that clearly mattered, but this was a positive update for me.

Fwiw, I think there are probably several other criticisms that I alone could find given some more time, let alone impactful criticisms that I never even read. I didn't even start looking for the genre of "critique of individual part of GiveWell cost-effectiveness analysis, which GiveWell then fixes", I think there's been at least one and maybe multiple such public criticisms in the past.

I also remember there being a StrongMinds critique and a Happier Lives Institute critique that very plausibly caused changes? But I don't know the details and didn't follow it

Ben_West🔸

1y*

26

I think I agree with your overall point but some counterexamples:

EA Criticism and Red Teaming Contest winners. E.g. GiveWell said "We believe HLI’s feedback is likely to change some of our funding recommendations, at least marginally, and perhaps more importantly improve our decision-making across multiple interventions"
GiveWell said of their Change Our Mind contest "To give a general sense of the magnitude of the changes we currently anticipate, our best guess is that Matthew Romer and Paul Romer Present's entry will change our estimate of the cost-effectiveness of Dispensers for Safe Water by very roughly 5 to 10% and that Noah Haber's entry may lead to an overall shift in how we account for uncertainty (but it's too early to say how it would impact any given intervention)."
HLI discussed some meaningful ways they changed as the result of criticism here.

calebp

1y*

6

Those are great examples.

Larks

1y

16

I feel like you are missing some important causal avenues through which plentiful criticism can be good:

If the expectation of harsh future criticism is a major deterrent from engagement, presumably it disproportionately deters the type of projects that expect to be especially criticized.
Criticism is educational for third parties reading and can help improve their future projects.

calebp

1y

13

Disproportionately deterring bad projects is a crux. I think if people are running a “minimise criticism policy” they aren’t going to end up doing very useful things (or they’ll do everything in secret or far from EA). I currently don’t think nearly enough people are trying to start projects and many project explorations seem good to me on net, so the discrimination power needs to be pretty strong for the benefits to pencil.

I think there are positives about criticism which I didn’t focus on, but yeah if I were to write a more comprehensive post I think the points you raised are good to include.

I’m not convinced that criticism is very counterfactually educational for 3rd parties. Particularly when imo lots of criticism is bad. Feels like it could go either way. If more criticism came from people who had run substantial projects or operated in the same field or whatever I think I’d trust their takes more. Many of the examples raised in this thread are good imo and have this property.

Jason

1y

16

There are almost no examples of criticism clearly mattering (e.g. getting someone to significantly improve their project)

I don't know what "clearly mattering" means, but I think this characterization unduly tips the scales. People who don't like being criticized are often going to be open about that fact, which makes it easier to build an anti-criticism case under a "clearly" standard.

Also, "criticism" covers a lot of ground -- you may have a somewhat narrower definition in mind, but (even after limiting to EA projects with <10 FTEs) people are understandably reacting to a pretty broad definition.

The most obvious use of criticism is probably to deter and respond to inappropriate conduct. Setting aside whether the allegations were sustained, I think that was a major intended mechanism of action in several critical pieces. I can't prove that having a somewhat pro-criticism culture furthers this goal, but I think it's appropriate to give it some weight. It does seem plausible on the margin that (e.g.) orgs will be less likely to exaggerate their claims and cost-effectiveness analyses given the risk of someone posting criticism with receipts.

A softer version of this purpose could be phrased as follows: criticism is a means by which the community expresses how it expects others to act (and hopefully influences future actions by third parties even if not by the criticized organization). In your model, "public critique clearly creates barriers to starting new projects," so one would expect public critique (or the fear thereof) to influence decisions by existing orgs as well. Then we have to decide whether that critique is on the whole good or not.

Criticism can help direct resources away from certain orgs to more productive uses. The StrongMinds-related criticisms of 2023 come to mind here. The resources could include not only funding but also mindshare (e.g., how much do I want to defer to this org?) and decisions by talent. This kind of criticism doesn't generally pay in financial terms, so it's reasonable to be generous in granting social credit to compensate for that. These outcomes could be measured, but doing so will often be resource-intensive and so they may not make the cut under a "clearly" standard either.

Criticism can also serve the function of market research. The usual response to people who aren't happy about how orgs are doing their work is to go start their own org. That's a costly response -- for both the unhappy person and for the ecosystem! Suppose someone isn't happy about CEA and EA Funds spinning off together and is thinking about trying to stand up an independent grantmaker. First off, they need to test their ideas against people who have different perspectives. They would also need to know whether a critical mass of people would move their donations over to an independent grantmaker for this or other reasons. (I think it would also be fair for someone not in a position to lead a new org to signal support for the idea, hoping that it might inspire someone else.)

It's probably better for the market-research function to happen in public rather than in back channels. Among other things, it gives the org a chance to defend its position, and gives it a chance to adjust course if too many relevant stakeholders agree with the critic. The counterargument to this one is that little criticism actually makes it into a new organization. But I'm not sure what success rate we should expect given considerable incumbency advantage in some domains.

Howie_Lempel

1y

16

"People who don't like being criticized are often going to be open about that fact"

[Just responding to this narrow point and not the comment as a whole, which contains plenty of things I agree with.]

Fwiw, I don't think this is true in this community. Disliking criticism is a bad look and seeming responsive to criticism is highly valued. I've seen lots of situations up close where it would have been very aversive/costly for someone to say "I totally disagree with this criticism and think it wasn't useful" and very tempting for someone to express lots of gratitude for criticism and change in response to it whether or not it was right. I think it's not uncommon for the former to take more bravery than the latter and I personally feel unsure whether I've felt more bias towards agreeing with criticism that was wrong or disagreeing with criticism that was right.

titotal

1y

11

Do you have an alternate suggestion for how flaws and mistakes made by projects in the EA sphere can be discovered?

As a scientist, one of the reasons people trust our work is the expectation that the work we publish has been vetted and checked by other experts in the field (and even with peer review, sloppy work gets published all the time). Isn't one of the goals of the EA forum to crowdsource at least some of this valuable scrutiny?

calebp

1y

6

I agree that this is one of the upsides of criticism on the forum. I don’t think it outweighs the costs in many cases.

Jonas_

1y

7

I think this is excellent criticism!

calebp

1y

8

Damnit!

Will Howard🔹

1y

6

I don't think it's obvious that less chance of criticism implies a higher chance of starting a project. There are many things in the world that are prestigious precisely because they have a high quality bar.

calebp

1y*

8

I’m a huge fan of having high standards. Posts that are like “we reproduced this published output and think they made these concrete errors” are often great. But I notice much more “these people did a bad job or spent too much money” takes often from people who afaict haven’t done a bunch of stuff themselves so aren’t very calibrated, and don’t seem very scope sensitive. If people saw their projects being critiqued and were then motivated to go and do more things more quickly I’d think that was great (or were encouraged to do more things more quickly from “fear” of critiques) I think we’d be in a better equilibrium.

For example people often point out that LW and the forum are somewhat expensive per user as evidence they are being mismanaged and imo this is a bad take which is rarely made by people who have built or maintained popular software projects/forums or used the internet enough to know that discussion of the kind in these venues is really quite rare and special.

To be clear, I think the “but have they actually done stuff” critique should also be levelled at grantmakers. I’m sympathetic to grantmakers who are like “the world is burning and I just need to do a leveraged thing right now” but my guess is that if more grantmakers had run projects in the reference class of things they want to fund (or founded any complicated or unusual and ambitious projects) we’d be in a better position. I think this general take is very common in YC/VC spaces, which perform a similar function to grantmaking for their ecosystem.

Many examples of criticism in replies, are high quality posts that I think improve standards. I may spend an hour going through the criticism tag and sorting them into posts I think are useful/anti-useful to check.

Will Howard🔹

1y*

6

I'm not quite as convinced of the much greater cost of "bad criticism" over "good criticism". I'm optimistic that discussions on the forum tend to come to a reflective equilibrium that agrees with valid criticism and disregards invalid criticism. I'll give some examples (but pre-committing to not rehashing these too much):

I think HLI is a good example of long-discussion-that-ends-up-agreeing-with-valid-criticism, and as discussed by other people in this thread this probably led to capital + mind share being allocated more efficiently.
I think the recent back and forth between VettedCauses and Sinergia is a good example of the other side. Setting aside the remaining points of contention, I think commenters on the original post did a good job of clocking the fact that there was room for the reported flaws to have a harmless explanation. And then Carolina from Sinergia did a good job of providing a concrete explanation of most of the supposed issues^[1].

It's possible that HLI and Sinergia came away equally discouraged, but if so I think that would be a misapprehension on Sinergia's part. Personally I went from having no preconceptions about them to having mildly positive sentiment towards them.

Perhaps we could do some work to promote the meme that "reasonably-successfully defending yourself against criticism is generally good for your reputation not bad".

(Stopped writing here to post something rather than nothing, I may respond to some other points later)

^{^}
You could also argue that not everyone has time to read through the details of these discussions, and so people go away with a negative impression. I don't think that's right because on a quick skim you can sort of pick up the sentiment of the comment section, and most things like this don't escape the confines of the forum.

Ozzie Gooen

1y*

6

As I'm sure many would imagine, I think I disagree.

There are almost no examples of criticism clearly mattering (e.g. getting someone to significantly improve their project)

There's a lot here I take issue with:
1. I'm not sure where the line is between "criticism" and "critique" or "feedback." Would any judgements about a project that aren't positive be considered "criticism"? We don't have specific examples, so I don't know what you refer to.
2. This jumps from "criticism matters" to "criticism clearly matters" (which is more easily defensible, but less important), to "criticism clearly mattering (e.g. getting someone to significantly improve their project)", which is one of several ways that criticism could matter, clearly or otherwise. The latter seems like an incredibly specific claim that misses much of the discussion/benefits of criticism/critique/feedback.

I'd rate this post decently high on the "provocative to clarity" measure, as in it's fairly provocative while also being short. This isn't something I take issue with, but I just wouldn't spend too much attention/effort on it, given this. But I would be a bit curious what a much longer and detailed version of this post would be like.

calebp

1y

6

Rohin and Ben provided some examples that updated me upwards a little on critique posts being useful.

I think most of my points are fairly robust to the different definitions you gave so the line isn't super important to me. This feels a bit nitpicky.

I don't think that "criticism clearly mattering (e.g. getting someone to significantly improve their project)" is a very specific claim. I think that one of the main responses people would like to see to criticism of a specific project is for that project to change in line with the criticism. Unlike many of the other proposed benefits of criticism, it is a very empirical claim.

It suspect you think that this post should have been closer to "here are some points for and against criticism" on the EA Forum, but I don't think posts need to be balanced or well-rounded like that, especially because, from my perspective, the forum is too pro-criticism but yeah, seems fine for you not to engage with this kind of content - I definitely don't think you're obliged to.

mlsbt

1y*

3

"public critique clearly created barriers to starting new projects" In what sense? People read criticism of other projects and decide that starting their own isn't worth it? People with new active projects discouraged by critique?

calebp

1y

4

Mostly, people with active projects are discouraged by critiques and starting new public ambitious projects is much less fun if there are a bunch of people on a forum who are out to get you.

mlsbt

1y

7

If I have an active project I want it to be as good as possible. Certainly there's been mean-spirited, low-quality criticism on the EA Forum before, but not a high proportion. If relatively valid criticism bothers the founder that much, their project is just probably not going to make it. Or they don't really believe in their project (maybe for good reason, as pointed out by the critique).

calebp

1y

6

Have you run a public EA project before or spent time talking to founders of similar projects? This seems extremely off to me.

mlsbt

1y

3

I have run non-EA projects that have been criticized internally and externally. Why do you think it's off? Criticism is just feedback + things that don't matter, when you believe in what you're doing. The EA world is rational enough to adjust its opinions properly in the fullness of time.

Ozzie Gooen

1y*

4

"starting new public ambitious projects is much less fun if there are a bunch of people on a forum who are out to get you"

To be clear, I assume that the phrase "are out to get you" is just you referring to people giving regular EA Forum critique?

The phrase sounds to me like this is an intentional, long-term effort from some actors to take one down, and they just so happen to use critique as a way of doing that.

calebp

8mo*

27

Quick, non-exhaustive list of places where a few strategic, dedicated, and ambitious altruists could make a significant dent within a year (because, rn, EA is significantly dropping the ball).

Improving the media, China stuff, increasing altruism, moral circle expansion, AI mass movement stuff, frontier AI lab insider coordination (within and among labs), politics in and outside the US, building up compute infrastructure outside the US, security stuff, EA/longtermist/School for Moral Ambition/other field building, getting more HNW people into EA, etc.

(List originally shared with me by a friend)

kuhanj

8mo

4

Will's list from his recent post has good candidates too:

AI character^[5]
AI welfare / digital minds
the economic and political rights of AIs
AI-driven persuasion and epistemic disruption
AI for better reasoning, decision-making and coordination
the risk of (AI-enabled) human coups
democracy preservation
gradual disempowerment
biorisk
space governance
s-risks
macrostrategy
meta

Peter

8mo

3

What do you think is causing the ball to be dropped?

calebp

8mo*

17

A lack of sufficiently strategic, dedicated, and ambitious altruists. Deferrence to authority figures in the EA community when people should be thinking more independently. Suboptimal status and funding allocation etc.

Peter

8mo

2

Would be curious to hear more. I'm interested in doing more independent projects in the near future but am not sure how they'd be feasible.

Bella

8mo

2

What's "SMA" in this context?

calebp

8mo

2

sorry - school for moral ambition

calebp

3y

81

Some quick thoughts on AI consciousness work, I may write up something more rigorous later.

Normally when people have criticisms of the EA movement they talk about its culture or point at community health concerns.

I think aspects of EA that make me more sad is that there seems to be a few extremely important issues on an impartial welfarist view that don’t seem to get much attention at all, despite having been identified at some point by some EAs. I do think that ea has done a decent job of pointing at the most important issues relative to basically every other social movement that I’m aware of but I’m going to complain about one of it’s shortcomings anyway.

It looks to me like we could build advanced ai systems in the next few years and in most worlds we have little idea of what’s actually going on inside them. The systems may tell us they are conscious, or say that they don’t like the tasks we tell them to do but right now we can’t really trust their self reports. There’ll be a clear economic incentive to ignore self reports that create a moral obligation to using the systems in less useful/efficient ways. I expect the number of deployed systems to be very large and that it’ll be plausible that we lock in the suffering of these systems in a similar way to factory farming. I think there are stronger arguments for the topic’s importance that I won’t dive into right now but the simplest case is just the “big if true-ness” of this area seems very high.

My impression is that our wider society and community is not orienting in a sane way to this topic. I don’t remember ever coming across a junior EA seriously considering directing their career to work in this area. 80k has a podcast with Rob Long and a very brief problem profile (that seems kind of reasonable), ai consciousness (iirc) doesn’t feature in ea virtual programs or any intro fellowship that I’m aware of, there haven’t been many (or any?) talks about it at eag in the last year. I do think that most organisations could turn around and ask “well what concrete action do you actually want our audience to take” and my answers are kind of vague and unsatisfying right now - I think we were at a similar point with alignment a few years ago and my impression is that it had to be on the communities mind for a while before we were able to pour substantial resources into it (though the field of alignment feels pretty sub-optimal to me and I’m interested in working out how to do a better job this time round).

I get that there aren’t shovel ready directions to push people to work on right now, but in so far as our community and its organisations brand themselves substantially as the groups identifying and prioritising the worlds most pressing problems it sure does feel to me like more people should have this topic on their minds.

There are some people I know of dedicating some of their resources to making progress in this area, and I am pretty optimistic about the people involved - the ones that I know of seem especially smart and thoughtful.

I don’t want all of the EA to jump into this rn, and I’m optimistic about having a research agenda in this space that I’m excited about and maybe even a vague plan about what one might do about all this by the end of this year - after which I think we’ll be better positioned to do field building. I am excited about people who feel especially well placed moving into this area - in particular people with some familiarity with both mainstream theories of consciousness and ml research (particularly designing and running empirical experiments). Feel free to reach out to me or apply for funding at the ltff.

Thomas Larsen

3y

23

(quick thoughts, may be missing something obvious)

Relative the scale of the long term future, the number of AIs deployed in the near term is very small, so to me it seems like there's pretty limited upside to improving that. In the long term, it seems like we have AIs to figure out the nature of consciousness for us.

Maybe I'm missing the case that lock-in is plausible, it currently seems pretty unlikely to me because the singularity seems like it will transform the ways the AIs are running. So in my mind it mostly matters what happens after the singularity.

I'm also not sure about the tractability, but the scale is my major crux.

I do think understanding AI consciousness might be valuable for alignment, I'm just arguing against work on nearterm AI suffering.

calebp

3y

2

Edit: I have a lot of sympathy for the take above but I tried to write up my response around why I think lock-ins are pretty plausible.

I’m not sure rn whether the majority of downside comes from lock-in but I think that’s what I’m most immediately concerned about.

I assume by singularity you mean an intelligence explosion or extremely rapid economic growth. I think my default story for how this happens in the current paradigm involves people using AIs in existing institutions (or institutions that look pretty similar today’s one’s) in markets that looks pretty similar to current markets which (on my view) are unlikely to care about the moral patienthood of AIs in a pretty similar ways to current market failures.

On the “markets still exist and we do things in kind of like how we do now view” - I agree that in principle we’d be better positioned to make progress on problems generally if we had something like PASTA but I feel like you need to tell a reasonable story for one of

how governance works post TAI so that you can easily enact improvements like eliminating ai suffering
why current markets do allow for things like factory farming and slavery but wouldn’t allow for violation of AI preferences

I’m guessing your view is that progress will be highly discontinuous and society will look extremely different post singularity to how it does now (kind of like going from pre-agricultural revolution to now whereas my view is more like preindustrial revolution to now).

I’m not really sure where the cruxes are on this view or how to reason about it well but my high level argument is that the “god like AGI which has significant responsibility but still checks in with its operators” will still need to make some trade offs across various factors and unless it’s doing some cev type thing, outcomes will be fairly dependent on the goals that you give it and it’s not clear to me that the median world leader or ceo gives the agi goals that concern the ai’s wellbeing (or its subsystems wellbeing) - even if it’s relatively cheap to evaluate it. I am more optimistic about agi controlled by person sampled from a culture that has already set up norms around how to orient to the moral patienthood of ai systems than one that needs to figure it out on the fly. I do feel much better about worlds where some kind of reflection process is overdetermined.

My views here are pretty fuzzy and are often influenced substantially by thought experiments like “If random tech ceo could effectively control all the worlds scientists and have them run at 10x speed and had 100 trillion dollars does factory farming still exist?” which isn’t a very high epistemic bar to beat. (I also don’t think I’ve articulated my models very well and I may take another stab at this later on).

I have some tractability concerns but my understanding is that few people are actually trying to solve the problem right now and when few people are trying it’s pretty hard for me to actually get a sense of how tractable a thing is, so my priors on similarly shaped problems are doing most of the work (which leaves me feeling quite confused).

Will Aldred

2y

2

I agree with your “no lock-in” view in the case of alignment going well: in that world, we’d surely use the aligned superintelligence to help us with things like understanding AI sentience and making sure that sentient AIs aren’t suffering.

In the case of misalignment and humanity losing control of the future, I don’t think I understand the view that there wouldn’t be lock-in. I may well be missing something, but I can't see why there wouldn’t be lock-in of things related to suffering risk—for example, whether or not the ASI creates sentient subroutines which help it achieve its goals but which incidentally suffer—that could in theory be steered away from even if we fail at alignment, given that the ASI’s future actions (even if they're very hard to exactly predict) are decided by how we build it, and which we could likely steer away from more effectively if we better understood AI sentience (because then we’d know more about things like what kinds of subroutines can suffer).

Shakeel Hashim

3y

19

I'm really glad you wrote this; I've been worried about the same thing. I'm particularly worried at how few people are working on it given the potential scale and urgency of the problem. It also seems like an area where the EA ecosystem has a strong comparative advantage — it deals with issues many in this field are familiar with, requires a blend of technical and philosophical skills, and is still too weird and nascent for the wider world to touch (for now). I'd be very excited to see more research and work done here, ideally quite soon.

rileyharris

2y

7

I hadn't seen this until now, but it's good to see that you've come to the same conclusion I have. I've just started my DPhil in Philosophy and plan on working on AI mental states and welfare.

zdgroff

3y

9

Very strong +1 to all this. I honestly think it's the most neglected area relative to its importance right now. It seems plausible that the vast majority of future beings will be digital, so it would be surprising if longtermism does not imply much more attention to the issue.

calebp

1y

34

In my opinion, one of the main things that EA / rationality / AI safety communities have going for them is that they’re extremely non-elitist about ideas. If you have a “good idea” and you write about it on one of the many public forums, it’s extremely likely to be read by someone very influential. And insofar as it’s actually a good idea, I think it’s quite likely to be taken up and implemented, without all the usual status games that might get in the way in other fields.

Stefan_Schubert

1y*

38

I might call that "meritocratic about ideas".

calebp

1y

14

Why would I listen to you? You don't even have an English degree.

Joseph_Chu

1y*

14

While I agree that it's not "elitist" in the sense that anyone can put forward ideas and be considered by significant people in the community (which I think is great!), I would say there's still some expectations that need to be met in that the "good idea" generally must accept several commonly agreed up premises that represent what I'd call the "orthodoxy" of EA / rationality / AI safety.

For instance, I noticed way back when I first joined Less Wrong that the Orthogonality Thesis and Instrumental Convergence are more or less doctrines, and challenging them tends to receive a negative or at best lukewarm reception, or at least, demand a very strong case to be made.

In that sense, there is still a bit of elitism in the sense that some of the ideas of the sorta co-founders of the movements, like Eliezer Yudkowsky, Nick Bostrom, Will MacAskill and such, are likely to be treated with notably more deference. It used to be the case, for instance, that on Less Wrong, posts that challenged these views would lead to somewhat dismissive "Read The Sequences" responses, although I admit I haven't seen that particular refrain in quite a while, so the community might have improved in this regard.

And admittedly, I do think compared to say, most political ideological movements and religions, the EA / rationality / AI safety communities are MUCH more open to dissenting views and good productive debates. This is one thing I do like about us.

MichaelDickens

1y

17

In that sense, there is still a bit of elitism in the sense that some of the ideas of the sorta co-founders of the movements, like Eliezer Yudkowsky, Nick Bostrom, Will MacAskill and such, are likely to be treated with notably more deference.

I think I disagree with this.

It has been joked that disagreeing with Eliezer is the favorite activity of LessWrongers.
Will MacAskill has written some very well-reasoned and well-received EAF posts that nonetheless were met with strong disagreement, for example Are we living at the most influential time in history? I think this is a sign of good epistemics: people recognized that post as good and interesting, but they didn't defer much—they mostly disagreed with its conclusion.

I think the main reason people are more likely to talk about those people's ideas is because their ideas are genuinely really good.

There will inevitably be some people who have good ideas, and lots of people are persuaded, and from the outside this looks like deference.

Joseph_Chu

1y*

1

I mean, from the inside it would look like what you said, good ideas that are persuasive. And I'm not saying they aren't good ideas that are persuasive (I agree they are). I'm more just pointing out some examples of ideas that form core elements of our belief ecosystem, that have their source in the works of particular elites, in this case, being named and popularized by Bostrom's book Superintelligence, and then further popularized by Yudkowsky in the Sequences. In the sense that this is elitism, it's very mild, and I don't mean to imply it's a problem or anything. It's natural for the more elite among us to be in a better position to come up with the good ideas. I think also that to the extent there is deference here, it is usually well deserved, and also very tame compared to other intellectual communities.

Neel Nanda

1y

7

I think this is reasonable as a way for the community to reflexively react to things, to be honest. The question I'm trying to answer when I see someone making a post with an argument that seems worth engaging with is: what's the probability that I'll learn something new or change my mind as a result of engaging with this?

When there's a foundational assumption disagreement, it's quite difficult to have productive conversations. The conversation kind of needs to be about the disagreement about that assumption, which is a fairly specific kind of discussion. Eg if someone hasn't really thought about AI alignment much, thinks it's not an issue, but isn't familiar with the reasons I believe it matters, then I put a much lower (though still non-zero) probability that I'll make useful updates from talking to them. Because I have a bunch of standard arguments for the most obvious objections people sometimes raise, and don't learn much from stating them. And I think there's a lot of value to having high-context discussion spaces where people broadly agree on these foundational claims.

These foundational claims are pretty difficult to establish consensus on if people have different priors, and discussing them doesn't really tend to move people either way. I get a lot of value from discussing technical details of what working on AI safety is like with people, much more so than I get from the average "does AI safety matter at all?" conversation.

Obviously, if someone could convince me that AI safety doesn't matter, that would be a big deal. But I'd guess it's only really worth the effort if I'm reasonably sure the person understands why I believe it does matter and disagrees anyway, in a way that isn't stemming from some intractable foundational disagreements in worldviews

Joseph_Chu

1y*

3

I want to clarify that I don't think ideas like the Orthogonality Thesis or Instrumental Convergence are wrong. They're strong predictive hypotheses that follow logically from very reasonable assumptions, and even the possibility that they could be correct is more than enough justification for AI safety work to be critical.

I was more just pointing out some examples of ideas that are very strongly held by the community, that happen to have been named and popularized by people like Bostrom and Yudkowsky, both of whom might be considered elites among us.

P.S. I'm always a bit surprised that the Neel Nanda of Google DeepMind has the time and desire to post so much on the EA Forums (and also Less Wrong). That probably says very good things about us, and also gives me some more hope that the folks at Google are actually serious about alignment. I really like your work, so it's an honour to be able to engage with you here (hope I'm not fanboying too much).

Chris Leong

1y*

5

If your complaint is that the default assumption is that these are more or less true, well, my claim is that even though normies tend to see this as a negative signal, it's actually a positive signal for those with good epistemics.

That said, it's important to keep in mind that these aren't directly talking about actual systems (the Orthogonality Thesis is about possible systems and Instrumental Convergence is about incentives).

Joseph_Chu

1y

3

I more or less agree. It's not really a complaint from me. I probably was too provocative in my choice of wording earlier.

NickLaing

1y

11

This is true, and I've appreciated it personally. I've been pleasantly surprised how people have responded to a couple of things I've written, even when they didn't know me from a bar of soap. I think this was unlikely to happen in academia or in the high brow public health world where status games often prevail like you said.

There is still though an element of being "known" which helps your ideas get traction. This does make sense as if someone has written something decent in the past, there's a higher chance that other things they write may also be decent, so we are more likely to give their next idea a read. This is why we follow certain people on Substack and Twitter and don't just read things at random (to state the obvious).

So I think there can still a hill to climb a the beginning to get good ideas seen, especially for very new people.

geoffrey

1y

7

Agreed. I’d extend the claim past ideas and say that EA is very non-elitist (or at least better than most professional fields) at any point of evaluation.

Maybe because of that, it seems more elitist elsewhere.

calebp

1y*

35

I'm interested in figuring out "what skill profiles are most leveraged for altruistic work after we get the first competent AI agents"?

I think this might be one of the most important questions for field-building orgs to work on (in any cause area but particularly AI safety). I think 80k and other AIS-motivated careers groups should try to think hard about this question and what it means for their strategy going forward.

I'm optimistic about productivity on the question of "what skill profiles are most leveraged after we get the first competent AI-agents" because:

few people have actually tried to think about what managing an AI workforce would look like
this doesn't feel conceptually abstract (e.g. some people have already adopted LMs in their day-to-day workflows)
something like drop-in replacements for human remote workers are more likely than AIs less analogous to human workers if timelines are short (and if timelines are short, this work is more urgent than if timelines are long)

One important consideration is whether human managers are going to be good proxies for AI-agent managers. It's plausible to me that the majority of object-level AI safety work that has ever be done will be done by AI agents. It could be that the current status quo persists, where people's leverage routes through things like leadership and management skills, or it could be that ~everyone will become more leveraged proportional to their access to AI models (roughly their budget) due to AI agents having different affordances than humans, or something else.

For example, suppose that AI agents are much more tolerant, hardworking, and understanding of user intent for short tasks than human workers. I expect that having good strategic takes would become much more leveraged and having good internal stakeholder management skills would become much less leveraged (relative to today). If 80k (for example) thought that was likely, then maybe they should double down on finding people with great strategy takes and care less about typical management skills.

Maybe "junior" roles will be automated before "senior" roles almost uniformly across the economy. In that case, it's even more valuable than it is right now to focus on getting "senior" people.

Or maybe coding will be automated before everything else, and for some reason people's relative leverage inside the company stays pretty fixed, or it's too hard to say which kinds of jobs will be most leveraged at software companies. Then "coding" still becomes much more leveraged per dollar than today, and it's probably useful to find more value-aligned coders.

It would be surprising to me if the price of certain kinds of intellectual labour decreased by 100x, and this had little impact on people's relative leverage.

(disclaimer: I'm not sure whether the above stories actually go through, they are meant to just be illustrative of the kind of thinking that seems undersupplied)

calebp

1y

19

Matching campaigns get a bad rep in EA circles* but it’s totally reasonable for a donor to be concerned that if they put in lots of money into an area other people won’t donate, and matching campaigns preserve the incentive for others to donate, crowding in funding.

* I agree that campaigns claiming you’ll have twice the impact as your donation will be matched are misleading.

Ian Turner

1y

4

Have you read Holden's classic on this topic? It sounds like you are describing what he calls "Influence matching".

Jason

1y

4

It's understandable for a donor to have that concern. However, I submit that this goes both ways -- it's also reasonable for smaller donors to be concerned that the big donors will adjust their own funding levels to account for smaller donations, reducing the big donor's incentives to donate. It's not obvious to me which of these concerns predominates, although my starting assumption is that the big donors are more capable of adjusting than a large number of smaller donors.

Much electronic ink has been spilled over the need for more diversification of funding control. Given that, I'd be hesitant to endorse anything that gives even more influence over funding levels to the entities that already have a lot of it. Unless paired with something else, I worry that embracing matching campaigns would worsen the problem of funding influence being too concentrated.

calebp

1y*

2

Seems plausible that EA Funds should explore offering matches to larger projects that it wants to fund to help increase the project’s funding diversity.

calebp

2y

36

Effective giving

I feel a bit confused about how much I should be donating.

On the one hand there’s just a straight forward case that donating could help many sentient beings to a greater degree than it helps me. On the other hand, donating 10% for me feels like it’s coming from a place of fitting in with the EA consensus, gaining a certain kind of status and feeling good rather than believing it’s the best thing for me to do.
I’m also confused about whether I’m already donating a substantial fraction of my income.
- I’m pretty confident that I’m taking at least a 10% pay-cut in my current role. If nothing else my salary right now is not adjusted for inflation which was ~8% last year so it feels like I’m at least underpaid by that amount (though it’s possible they were overpaying me before). Many of my friends earn more than twice as much as I do and I think if I negotiated hard for a 100% salary increase the board would likely comply.
- So how much of my lost salary should I consider to be a donation? I think numbers between 0% and 100% are plausible. -50% also isn’t insane to me as my salary does funge with other peoples donations to charities.
- One solution is that I should just negotiate for my salary from a non-altruistic perspective, and then decide how much I want to donate back to my organisation after that. This seems a bit inefficient though and I think we should be able to do better.
One reason I don’t donate ~50% of my salary is that I genuinely believe it’s more cost-effective for me to build runway than donate right now. I quite like the idea of discussing this with someone who strongly disagrees with me and I admire and see if they come round to my position. It feels a bit too easy to find reasons not to give, and I’m very aware of my own selfishness in many parts of my life.

Joel Tan🔸

2y

17

A couple of considerations I've thought about, at least for myself

(1) Fundamentally, giving helps save/improve lives, and that's a very strong consideration that we need equally strong philosophical or practical reasons to overcome.

(2) I think value drift is a significant concern. For less engaged EAs, the risk is about becoming non-EA altogether; for more engaged EAs, it's more about becoming someone less focused on doing good and more concerned with other considerations (e.g. status); this doesn't have to be an explicit thing, but rather biases the way we reason and decide in a way that means we end up rationalizing choices that helps ourselves over the greater good. Giving (e.g. at the standard 10%) helps anchor against that.

(3) From a grantmaking/donor advisory perspective, I think it's hard to have moral credibility, which can be absolutely necessary (e.g. advising grantees to put up modest salaries in their project proposals, not just to increase project runway but also the chances that our donor partners approve the funding request). And this is both psychologically and practically hard to do this if you're not just earning more but earning far more and not giving to charity! Why would they listen to you? The LMIC grantees especially may be turned off - disillusioned, by the fact that they have to accept peanuts while those of us with power over them draw fat stacks of cash! The least we can do is donate! Relatedly, I think part of Charity Entrepreneurship's success is absolutely down to Joey and co leading by example and taking low salaries.

(4) Runway is a legitimate consideration, especially since there are a lot of potentially impactful things one can do but which won't be funded upfront (so you need to do it on savings, prove viability and then get it funded). However, I don't think this is sufficient to outweigh points 1-3.

(5) In general, I think it's not useful at all to compare with how much others are earning - that only leads to resentment, unhappiness, and less impactful choices. For myself, the vast majority of my friends are non-EAs; we have similar backgrounds (elite education, worked for the Singapore government as policy officers/scholars at one point or another) and yet since leaving government I've had a riskier career, earn far less, have fewer savings, and am forced to delay having a family/kids because of all those reasons. All of this is downstream of choices I've made an EA, particularly in avoiding job offers that paid very well but which didn't have impact (or in fact, had negative impact). Is the conclusion I'm supposed to draw that I've made a mistake with my life? I don't think so, because statistically speaking, some random African kid out there is alive as a result of my donations (and hopefully, my work), and that's good enough for me.

calebp

2y

6

Thanks for these thoughts. It's nice to get such detailed engagement. I'm going to try to respond point by point.

(2) - I'm not particularly worried about value drift, and I think there are more effective ways to guard against this than earning to give (e.g. living with people who share your values, talking about EA stuff regularly with people you care about). I think I have quite a lot of evidence in favour of me being pretty resilient to value drift (though I often change my mind about what is important intentionally).

(3) I think this is interesting, though I don't think that I share this view re being taken seriously. I think that I, and many people I know, have taken actions that they found much harder than donating (e.g. I live in a different country than my partner and in a pretty suboptimal timezone because I think I can do my work better from my current location, I work a lot of hours, I spend a lot of time doing tasks that I find emotionally challenging, I've been in situations that I found extremely stressful for ~0 credit). To be clear, I don't think that I am particularly worthy of praise - but I do think that I score reasonably well on "moral credibility". Also, I have concerns about this kind of signalling and think it often leads to concerning dynamics - I don't want EA Funds grantees to feel pressured into taking shoestring salaries. When I was at CE, I remember there being a lot of pressure to take extremely low salaries despite many successful charity founders thinking this was a bad idea. It also led to weird epistemic effects (though I hear things have improved substantially).

(4) I don't think that runway and grants from EA funders are as fungible as you do. I can talk a bit more about this if that's useful. I guess that this general point (3) is where we have substantive disagreement. It seems likely to me that I can have much more impact through my career than through my donations - and that having more runway could substantially increase the value of my career. If it doesn't increase the value of my career and I am wrong, then I can donate later (which I don't think incurs much in the way of losses from a NTist perspective, but it's more confusing from a LTist one). To be clear, I think that I'd like to build up 12-24 months of runway, and right now, I have substantially less than that - I am not talking about being able to retire in 10 years or anything.

(5) I think for me, the comparison stuff doesn't really lead to resentment/unhappiness. It wasn't clear from my post, but one of the reasons that I made this comparison was because many of my friends do very altruistically valuable work and earn substantially more than I do. They are extremely talented and hard-working (and lucky), and whilst this doesn't mean that I could get a highly-paying job that generated a lot of altruistic value, I think talking to them regularly has given me an understanding of the kind of work that they do and what it might take to enter a similar role, and it feels doable for me to enter similar roles in a relatively short amount of time (on my inside view). I also have friends that I think are similarly smart/hardworking etc., who earn a lot more money than me in purely for-profit roles. Again, I don't resent any of these people, and the comparison seems pretty useful to me.

MathiasKB🔸

2y

3

For what it's worth, I think saving up runway is a no brainer.

During my one year as a tech consultant, I put aside half each month and donated another 10%. The runway I built made the decision for me to quit my job and pursue direct work much easier.

In the downtime between two career moves, it allowed me to spend my time pursuing whatever I wanted without worrying about how to pay the bills. This gave me time to research and write about snakebites, ultimately leading to Open Phil recommending a $500k investment into a company working on snakebite diagnostics.

I later came upon great donation opportunity to a fish welfare charity, which I gave a large part of my runway to and wouldn't have been able to support if I had given all my money away two years prior.

Had I given more away sooner I think it would be clearer to myself and others that I was in fact altruistically motivated. I also think my impact would have been lower. Impact over image.

EDIT: Actually it's probably a some-brainer a lot of the time, seeing as I currently have little runway and am taking a shoestring salary. The reason I take a shoestring salary is to increase my organization's runway, which is valuable for the same reasons that increasing one's personal runway is. You don't have to spend as much time worrying about how your org is going to pay the bills and you can instead focus on impact.

Linch

2y

2

(I work with Caleb. Opinions are my own.)

Thank you for your comment.

I'm probably misunderstanding you, but I'm confused by (3) and (5). They seem like they somewhat contradict each other. Remove the emotive language and (3) is saying that people in positions of power should donate and/or have lower salaries because donors or grantees might be upset in comparison, and (5) is saying that we shouldn't compare our own earnings and donations to others.

These claims contradict each other in the following ways:

If we take (5) as a given, (3) no longer makes sense. If it truly is the case that comparing earnings is never useful, we should not expect (or want) grantees or donors to compare earnings.
1. Hypothetically, maybe your position might be more like "oh it's clearly bad to compare earnings, but we live in a flawed world with flawed people." But if that were the case, then acceding to people's comparisons is essentially enabling a harmful activity, and maybe we should have a higher bar for enabling others' negative proclivities.
If we take (3) as the primary constraint (Donors/grantees respect us less if we don't visibly make sacrifices for the Good), then it seems like (5) is very relevant. Pointing out ways in which we sacrificed earnings to take on EA jobs just seems like a really good reply to concerns that we are being overpaid in absolute terms, or are only doing EA jobs for the money. At least in my case, I don't recall any of our large donors complaining about my salary, but if I did, "I took a >>70% pay cut originally to do EA work" ^[1] seems like a reasonable response that I predict to mollify most donors.

^{^}
Though I think it's closer to ~40-50% now at my current salary, adjusting for inflation? On the other hand, if I stayed and/or switched jobs in tech I'd probably have had salary increases substantially above inflation as well, so it's kind of confusing what my actual counterfactual is^[2]. But I'm also not sure how much I should adjust for liking EA work and being much more motivated at it, which seems like substantial non-monetary compensation. But EA work is also more stressful and in some ways depressing, so hazard pay is reasonable, so...¯\_(ツ)_/¯.
^{^}
In part because I think if I wasn't doing EA work the most obvious alternative I'd be aiming for high-variance earning-to-give, which means high equity value in expectation but ~0 payout in the median case.

GraceAdams🔸

2y

13

Hey Caleb!

(I'm writing this in my personal capacity, though I work at GWWC)

On 1: While I think that giving 10% is a great norm for us to have in the community (and to inspire people worldwide who are able to do the same), I don't think there should be pressure for people to take a pledge or donate who don't feel inspired to do so - I'd like to see a community where people can engage in ways that make sense for them and feel welcomed regardless of their donation habits or career choices, as long as they are genuinely engaging with wanting to do good effectively.

On 3: I think it makes sense for people to build up some runway or sense of financial stability, and that they should generally factor this in when considering donating or taking a pledge. I personally only increased my donations to >10% after I felt I had enough financial stability to manage ongoing health issues.

I do think that people should consider how much runway or savings they really need though, and whether small adjustments in lifestyle could increase their savings and allow for more funds to donate - after all, many of us are still in the top few % of global income earners even after taking jobs that are less than we would getting in the private sector.

Brad West🔸

2y

6

It may be that building runway is, in fact, the best way to do good in the long term. And maybe certain levels of personal consumption make you more able to sustainably do good through your work.

But just engage seriously with the cost of that runway. With straightforward Givewell charities, that might mean someone dies annually that you could have saved.

calebp

2y

2

I guess there are two questions it might be helpful to separate.

what is the best thing to do with my money if I am purely optimising for the good?
how much of my money does the good demand?

Looking at the first question (1), I think engaging with the cost of giving (as opposed to the cost of building runway) wrt doing the most good is also helpful. It feels to me like donating $10K to AMF could make me much less able to transition my career to a more impactful path, costing me months, which could mean that several people die that I could have saved via donating to Givewell charities.

It feels like the "cost" applies symmetrically to the runway and donating cases and pushes towards "you should take this seriously" instead of having a high bar for spending money on runway/personal consumption.

Looking at (2) - Again I broadly agree with the overall point, but it doesn't really push me towards a particular percentage to give.

Brad West🔸

2y

2

Yes that's right.

For me, if the answer to #1 is in favor of saving for runway, that disposes of the question. Just need to be careful, as you are aware, of motivated reasoning.

For #2, for me, the good demands all of your money. Of course, you are not going to be the most effective agent if you keep yourself in poverty, so this probably doesn't imply total penury. But insofar as other conscious beings today are capable of positive and negative experiences like you are, it isn't clear why you should privilege your own over those of other conscious beings.

jacquesthibs

2y

4

To share another perspective: As an independent alignment researcher, I also feel really conflicted. I could be making several multiples of my salary if my focus was to get a role on an alignment team at an AGI lab. My other option would be building startups trying to hit it big and providing more funding to what I think is needed.

Like, I could say, "well, I'm already working directly on something and taking a big pay-cut so I shouldn't need to donate close to 10%", but something about that doesn't feel right... But then to counter-balance that, I'm constantly worried that I just won't get funding anymore at some point and would be in need of money to pay for expenses during a transition.

calebp

2y

4

Fwiw my personal take (and this is not in my capacity as a grantmaker) is that building up your runway seems really important, and I personally think that it should be a higher priority than donating 10%. My guess is that GWWC would suggest dropping your commitment to say 2% as a temporary measure while you build up your savings.

GraceAdams🔸

2y

8

Many people see the commitment of the pledge to give 10% as one over their lifetime, so if you needed to drop back to build up runway for a while, with the intention of donating more in the following years once your finances were more secure, I personally think that would be an acceptable way to fulfil the pledge!

There's no strict requirement that donations need to be made each year, but GWWC does encourage regular giving where possible.

(FYI I work at GWWC)

Patrick Gruban 🔸

2y

2

Concerning 2 I think from an organization’s perspective it might be even helpful to have a salary agreed with you that makes it easier to replace you and so calculate a realistic runway for the organization.Then you can still voluntarily reduce the salary and claim it as donation. This is what I’m doing and it helps with my own budgeting and with that of my org. I was inspired by this post: https://forum.effectivealtruism.org/posts/GxRcKACcJuLBEJPmE/consider-earning-less

calebp

2y

28

Companies often hesitate to grant individual raises due to potential ripple effects:

The true cost of a raise may exceed the individual amount if it:

Necessitates adjustments to other salaries via organizational policies
Sets precedents for other employees
Sparks salary discussions among colleagues

An alternative for altruistic employees: Negotiate for charitable donation matches

Generally less desirable for most employees
Similarly attractive to altruists (Assumption: Current donations to eligible charities exceed the proposed raise)

This approach allows altruists to increase their impact without triggering company-wide salary adjustments.

Jeff Kaufman 🔸

2y

12

(About 85% confident, not a tax professional)

At least in the US I'm pretty sure this has very poor tax treatment. The company match portion would be taxable to the employee while also not qualifying for the charitable tax deduction. The idea is they can offer a consistent match as policy, but if they're offering a higher match for a specific employee that's taxable compensation. And the employee can only deduct contributions they make, which this isn't quite.

calebp

1y

14

The flip side of “value drift” is that you might get to dramatically “better” values in a few years time and regret locking yourself into a path where you’re not able to fully capitalise on your improved values.

Lorenzo Buonanno🔸

1y*

13

Every now and then I'm reminded of this comment from a few years ago: "One person's Value Drift is another person's Bayesian Updating"

NickLaing

1y*

11

I probably agree with this idea, but I wouldn't label it it "value drift" myself.

From my perspective I would call what you're describing more keeping a scout mindset around our values, and trying to ever improve.

"Value drift" for me signals the negative process of switching off our moral radar and almost unconscious drifting towards the worlds norms of selfishness, status, blissful ignorance etc. Reversion towards the mean. Hence the the "drift". I'm not sure I've ever seen someone drift their way to better values. Within the church I have seen big downsides of beligerent soldier mindsetting of values, but even then I would say "value drift" towards the mean has been even more harmful.

Somewhat paridoxically looking at my own life, crystallising what I think are my values at certain points in time, has helped me realise when I don't really believe that any more and the change them.

calebp

1y

7

I think from the inside they feel the same. Have you spoken to people who in your view have drifted? If so how did they describe how it felt?

NickLaing

1y

7

People I know who In my opinion have "drifted" (quite a lot of people) are generally unaware of what's happened as it all happens so slowly and "normal" life takes over, or if they are aware they don't really want to talk about it much. My experience though is from community advocacy/social justice circles in my early 20s (I'm now 38), not from EA circles.

calebp

3y

40

Update 2024-Jul-5 as this seems to be getting some attention again: I am not sure whether I endorse the take below anymore - I think 80k made some UI changes that largely address my concerns.

The 80k job board has too much variance.

(Quickly written, will probably edit at some point in future)

Jobs on the main 80k job board can range from (in my estimation) negligible value to among the best opportunities I'm aware of. I have also seen a few jobs that I think are probably actively harmful (e.g., token alignment orgs trying to build AGI where the founders haven't thought carefully about alignment—based on my conversations with them).

I think a helpful orientation towards jobs on the jobs board is, at least one person with EA values who happens to work at 80k thinks it's worth signal boosting. And NOT EA/80k endorses all of these jobs without a lot more thought from potential applicants.

Jobs are also on the board for a few different reasons, e.g., building career cap vs. direct impact vs.... and there isn't lots of info about why the job is there in the first place.

I think 80k does try to give more of this vibe than people get. I don't mean to imply that they are falling short in an obvious way.

I also think that the jobs board is more influential than 80k thinks. Explicit endorsements of organisations from core EA orgs are pretty rare, and I think they'd be surprised how many young EAs overupdate on their suggestions (but only medium confidence about it being pretty influential).

My concrete improvement would be to separate jobs into a few different boards to the degree that they endorse the organisation.

One thing I find slightly frustrating is the response that I have heard from 80k staff to this is that the main reason they don't do this is around managing relationships with the organisations (which could be valid). Idk if it's the right call but I think it's a little sus, I think people are too quick to jump to the nice thing that doesn't make them feel uncomfortable over the impact maximising thing (pin to write more about this in future).

One error that I think I'm making is criticising an org for doing a thing that is probably much better than not doing the thing even if it think it's leaving some value on the table, I think that this is kind of unhealthy and incentives inaction. I'm not sure what to do about this other than flag that I think 80k is great as is most of the stuff they do and I'd rather orgs had a policy of occasionally producing things that I feel moderately about if this helps them do a bunch of cool stuff, than underperform and not get much done (pin to write more about this in future).

Yonatan Cale

3y

7

Agree!

My best idea for solving this is making an alternative view for 80k's job board that has some reasons to obviously prefer it, and to add features to it like "here's a link to the org's AMA post", where I hope the community can comment on things like "this org is trying to build an AGI with little concern for safety", and lots of people can upvote it. No political problems for 80k. Lots of good high quality discussions. Hopefully.

What do you think?

Yonatan Cale

3y

7

Regarding some jobs being there just for building career capital - I only learned about this a few days ago and it kind of worries me. I don't have good ideas on how to solve it

Niel_Bowerman

3y

6

>it kind of worries me
Is that because you think the job board shouldn't list career capital roles, because it wasn't obvious that the roles were career capital-related, or something else?

Yonatan Cale

3y

3

What worries me:

I think lots of people take (and took) a job from 80k's board..

hoping to do something impactful.

in fact doing something neutral or perhaps (we could discuss this point,) actively harmful.

Unaware that this is the situation.

What do you think? (does this seems true? does it seem worrying?)

Niel_Bowerman

3y

3

In case it's helpful, the first thing below the title on the job board says:
>Some of these roles directly address some of the world’s most pressing problems, while others may help you build the career capital you need to have a big impact later.

I'd be interested in any ideas you had for communicating more clearly that a bunch of the roles are there for a mix of career capital and impact reasons. Giving our guess of the extent to which each role is being listed for career capital vs impact reasons isn't feasible for various reasons unfortunately.

Yonatan Cale

3y

10

TL;DR: I think this is very under communicated

You have that line there, but I didn't notice it in years, and I recently talked to other people who didn't notice it and were also very surprised. The only person I think I talked to who maybe knew about it is Caleb, who wrote this shortform.

Everyone (I talked to) thinks 80k is the place to find an impactful job.

Maybe the people I talk to are a very biased sample somehow, it could be, but they do include many people who are trying to have a high impact with their career right now

Yonatan Cale

3y

10

I checked if people know this by opening a poll for the EA Twitter community:

Yonatan Cale

3y

5

Could you say more on why it's not feasible? Maybe it's something we could solve?

Yonatan Cale

3y

4

Just saying, filtering the jobs by org does sound good to me (in almost all situations), in case that's the bottle neck.

"This org - we think it's impactful. That org - just career building"

calebp

3y

4

Oh this is a cool idea! I endorse this on the current margin and think it's cool that you are trying this out.

I think that ideally a high context person/org could do the curation and split this into a bunch of different categories based on their view (ideally this is pretty opinionated/inside viewy).

Yonatan Cale

3y

3

Next idea: Have a job board with open vetting, where anyone can comment or disagree with the impact analysis, including the company itself.

What do you think?

Niel_Bowerman

3y

3

I think linking to organisations' AMAs on the EA Forum is a neat idea! Thanks for sharing. I've added it to our list of feature ideas we might build in the future.

Yonatan Cale

3y

5

Thank you!
I admit I'm a bit worried when I hear "might build in the future" about a feature that seems very small to me (I could add it to my own version), and a part of me is telling me this is your way of saying you actually never want to build it. I'm not sure how to phrase my question exactly.. maybe "if someone else would do the dev work, would you be happy just putting it in, or is there another bottle neck?"
Also excuse me for my difficulty understanding subtext, I am trying

Nathan Young

3y

4

FYI there is a super-linear prize for an automated jobs board. https://www.super-linear.org/prize?recordId=recSFgbnu7VzAHCqY

Yonatan Cale

3y

4

Yeah, have an automation to put the tweets in an Airtable, and something to export the past tweets, just gotta put them together.

Do note that it doesn't solve the problem of high variance

Nathan Young

3y

4

The next feature I want to get is voting, which will work on that problem.

Yonatan Cale

3y

2

Oh, may I please try to convince you not to create your own voting system?

Initial reasons, as an invitation to push back:

Commenting is more important than voting

If, for example, someone thinks a specific org is actively harmful, I think:

Good situation: Someone writes a comment with the main arguments, references, and so on.

Bad situation: Someone needs to get lots of people to downvote the position. (Or people don't notice) (or the org gets lots of people to upvote) (or other similar situations)

Upvoting comments is better than both

And the double "upvote/downvote" + "agree/disagree" is even better, where the best comments float up.

See how conversations like that in the forum/lesswrong look. This is unusually good for the internet, and definitely better than upvoting/downvoting alone.

Is this system perfect? No, but it's better than anything I've seen, definitely better than upvotes alone.

[Reducing friction for people to voice their opinion] is key

+ For platforms like this, the amount of active users matters, there's an importance in having a critical mass.

So:

Adding a new platform is friction.

I vote for using an existing platform. Like the EA Forum.

Maybe a post without the "frontpage" tag
Maybe a comment on a post

These conversations already fit the EA Forum

It's discussing the impact of the org.

(I wouldn't be too surprised if there's a good reason to use something else, but I doubt it would be a good idea to create a NEW platform)

Nathan Young

3y

4

I have tried to convince the forum team of this, using the methods they asked to be convinced via. There has been some move to put jobs on the forum, but no in a searchable way. I think a new site that pushes better norms would be better.

I largely agree with the object level points you make but I don't see why you wouldn't want a new org with better processes.

https://forum.effectivealtruism.org/posts/uxfWrFNH7jSSGhkkS/unofficial-pr-faq-posting-more-jobs-to-the-forum-but-they

Yonatan Cale

3y

2

Any chance you'd share what you don't like?

That they posted like they have the job features even though they don't?

(btw I don't recommend using the forum's FILTERING/SEARCHING, I'd only use their commenting and upvoting. And login)

Nathan Young

3y

4

It's not searchable, filterable, or capable of taking a feed from.

Yonatan Cale

3y

2

I see.

So indeed I wouldn't use the forum for that. I'd only link from [something filterable and so on] to forum comments.

What do you think?

calebp

8mo

9

I suggested the following question for Carl Shulman a few years ago

I'd like to hear his advice for smart undergrads who want to build their own similarly deep models in important areas which haven't been thought about very much e.g. take-off speeds, the influence of pre-AGI systems on the economy, the moral value of insects, preparing for digital minds (ideally including specific exercises/topics/reading/etc.).

I'm particularly interested in how he formed good economic intuitions, as they seem to come up a lot in his thinking/writing.

https://forum.effectivealtruism.org/posts/ytBxJpQsdEEmPAv9F/i-m-interviewing-carl-shulman-what-should-i-ask-him?commentId=vqrxdfNEnYioDEt4N

@Saul Munn recently asked me what my current take on this was and then suggested I share it online - it's very lightly edited.

I think if I were to speak to my undergrad self I'd prioritise:
    •    reading a lot of SOTA background material (e.g. Dwarkesh podcasts on AI stuff)
    •    becoming literate in a bunch of seemingly useful fields (e.g. CS, math, econ, philosophy) with some bias towards things that are fun and motivating
    •    try to contribute to the intellectual frontier as quickly as possible (e.g. via LessWrong) on topics that seem very important
    •    find a community of people that care about similar stuff, ideally in person - consider starting a "better futures", or "what to do about AGI" group.

This is pretty general advice. I might try to focus on Holden's list Important, actionable research questions for the most important century or similar.

4

10

Some AI research projects that (afaik) haven't had much work done on them and would be pretty interesting:

If the US were to co-build secure data centres in allied countries, would that be geopolitically stabilising or destabilising?
What AI safety research agendas could be massively sped up by AI agents? What properties do they have (e.g. easily checkable, engineering > conceptual ...)?
What will the first non-AI R&D uses of powerful and general AI systems be?
Are there ways to leverage cheap (e.g. 100x lower than present-day cost) intelligence or manual labour to massively increase the US's electricity supply?
What kinds of regulation might make it easier to navigate an intelligence explosion (e.g. establishing quick pathways to implement policy informed by AI experts, or establishing zones where compute facilities can be quickly built without navigating a load of red tape)?

jacquesthibs

1y

4

We’re working on providing clarity on:

“What AI safety research agendas could be massively sped up by AI agents? What properties do they have (e.g. easily checkable, engineering > conceptual ...)?”

I’ll strongly consider putting out a post with a detailed breakdown and notes on when we think it’ll be possible. We’re starting to run experiments that will hopefully inform things as well.

calebp

1y

4

Do you have a list of research questions that you think could easily be sped up with AI systems? I suspect that I'm more pessimistic than you are due to concerns around scheming AI agents doing intentional research sabotage and think that the affordances of AI agents might make some currently intractable agendas more tractable.

[anonymous]

1y*

3

Hey Caleb,

For the first take, I'd say the answer is likely geopolitically destabilising. I work in the DC infrastructure industry for a US ally, and we're seeing a deluge of sovereign GPU cloud providers popping up with bare metal, locally housed solutions for defence/gov contracts. There's a market bet (at least in Australia) that compliance demands will favour local AI IaaS providers over co-builds, even if US-operated hyperscalers might provide far greater functionality across the stack. Another valuable reference point would be the political controversy regarding Pine Gap, a joint communications and signals base between the US and Australia.

Re: Point 5. I strongly agree; this is likely to be my research focus during my upcoming MPhil at CSER. While many provider to consumer deployment pathways are currently governed by convention, there are possible regulatory paths to influence key process owners in digital infrastructure to take greater autonomy over their roles and veto risky deployments.

calebp

1y

5

Thank you for replying - it's great that someone within the industry shared their perspective!

I don't really understand why that would make the US building DCs in allied countries destabilising. The short answer for why it might be stabilising is:
* It gives non-US actors more leverage, making deals where benefits are shared more likely.
* It's harder for the US to defect on commitments to develop models safely and not misuse them if it's easy for their allies to spy on them (or they have made commitments for DC use to be monitored)
* It keeps the Western democracies ahead of the CCP.

I think that allied countries themselves building DCs might be comparably stabilising - it gives more leverage to allied countries, at the cost of baking in less coordination and affordances to make deals around how AI is used and developed.

[anonymous]

1y*

3

I didn't articulate myself clearly enough — first-time poster blues! I'd argue these co-builds are a destabilising force for the same reason I mentioned Pine Gap (without explaining myself, whoops).

The benefits allies receive from these facilities are often at the expense of sovereignty over the site or technical oversight by local regulatory bodies.

Now, this tradeoff might be worth it for the intelligence agencies, but the US presence is often conspicuous and jarring to the local population, even in a remote area like Alice Springs, where PG is located. It would be especially so in a major metropolitan area like Sydney, Melbourne, or Canberra, where these DCs would likely be built.

Given the lack of trust foreign populations currently have for the US as a security partner, it would be political anathema for most WEIRD nations, at least within the next four years, to announce these types of co-builds.

There's also a question of what incentives the US would have to create these co-builds if they do provide this spying capability to allies, given the disproportionate amount of local compute to draw upon, particularly in Northern Virginia near the existing Washington security apparatus.
Northern Virginia is world's hyperscale datacenter capital • The Register

Re: your excellent last point, how might this leverage be exercised in a way that leads to greater stability? What type of capital is created? Where would it be expended and to what ends?

In short, I'm very sceptical these co-builds will happen. If they did, they'd be a political football of national relevance in nearly all allies. I'm unsure how allies would use them to cultivate stability, as I'm unsure what criteria we're judging stability by.

Thanks for getting me to explain myself a bit better!

calebp

3y

16

Very half baked

Should recent ai progress change the plans of people working on global health who are focused on economic outcomes?

It seems like many more people are on board with the idea that transformative ai may come soon, let’s say within the next 10 years. This pretty clearly has ramifications for people working on longtermist causes areas but I think it should probably affect some neartermist cause prioritisation as well.

If you think that AI will go pretty well by default (which I think many neartermists do) I think you should expect to see extremely rapid economic growth as more and more of industry is delegated to AI systems.

I’d guess that you should be much less excited about interventions like deworming or other programs that are aimed at improving people’s economic position over a number of decades. Even if you think the economic boosts from deworming and ai will stack, and you won’t have sharply diminishing returns on well-being with wealth I think you should be especially uncertain about your ability to predict the impact of actions in the crazy advanced ai world (which would generally make me more pessimistic about how useful the thing I’m working on is).

I don’t have a great sense of what the neartermists who think AI will go well should do. I’m guessing some could work on accelerating capabilities though I think that’s pretty uncooperative. It’s plausible that saving lives now is more valuable than before if you think they might be uploaded but I’m not sure there is that much of a case for this being super exciting from a consequentialist world view when you can easily duplicate people. I think working on ‘normy’ ai policy is pretty plausible or trying to help governments orient to very rapid economic growth (maybe in a similar way to how various nonprofits helped governments orient to covid).

Jason

3y

4

To significantly change strategy, I think one would need to not only believe "AI will go well" but specifically believe that AI will go well for people of low-to-middle socioeconomic status in developing countries. The economic gains from recent technological explosions (e.g., industrialization, the computing economy) have not lifted all boats equally. There's no guarantee that gaining the technological ability to easily achieve certain humanitarian goals means that we will actually achieve them, and recent history makes me pretty skeptical that it will quickly happen this time.

calebp

3y

6

I’m not an expert but I’d be fairly surprised if the Industrial Revolution didn’t do more to lift people in LMICs out of poverty than any known global health intervention even if you think it increased inequality. Would be open to taking bets on concrete claims here if we can operationalise one well.

Jason

3y

5

I think the Industrial Revolution and other technological explosions very likely did (or will) have an overall anti-poverty impact . . . but I think that impact happened over a considerable amount of time and was not of the magnitude one might have hoped for. In a capitalist system, people who are far removed from the technological improvements often do benefit from them without anyone directing effort at that goal. However, in part because the benefits are indirect, they are often not quick.

So the question isn't "when will transformational AI exist" but "when will transformational AI have enough of an impact on the wellbeing of economic-development-program beneficiaries that it significantly undermines the expected benefits of those programs?" Before updating too much on the next-few-decades impact of AI on these beneficiaries, I'd want to see concrete evidence of social/legal changes that gave me greater confidence that the benefits of an AI explosion would quickly and significantly reach them. And presumably the people involved in this work modeled a fairly high rate of baseline economic growth in the countries they are working in, so massive AI-caused economic improvement for those beneficiaries (say) 30+ years from now may have relatively modest impact in their models anyway.

Tom Barnes🔸

3y

3

Should recent ai progress change the plans of people working on global health who are focused on economic outcomes?

I think so, see here or here for a bit more discussion on this

If you think that AI will go pretty well by default (which I think many neartermists do)

My guess/impression is that this just hasn't been discussed by neartermists very much (which I think is one sad side-effect from bucketing all AI stuff in a "longtermist" worldview)

calebp

3y*

15

If you want there to be more great organisations, don’t lower the bar

I sometimes hear a take along the lines of “we need more founders who can start organisations so that we can utilise EA talent better”. People then propose projects that make it easier to start organisations.

I think this is a bit confused. I think the reason that we don’t have more founders is due to a having few people who have deep models in some high leverage area and a vision for a project. I don’t think many projects aimed at starting new organisations are really tackling this bottleneck at its core and instead lower the bar by helping people access funding, or appear better positioned than they actually are.

I think in general people that want to do ambitious things should focus on building deep domain knowledge, often by working directly with people with deep domain knowledge. The feedback loops are just too poor within most EA cause areas to be able to learn effectively by starting your own thing. This isn’t always true, but I think it’s more often than not true for most new projects that I see.

I don’t think the normal startup advice where running a startup will teach you a lot applies well here. Most startups are trying to build products that their investors can directly evaluate. They often have nice metrics like revenue and daily active users that track their goals reasonably well. Most EA projects lack credible proxies for success.

Some startups also lack credible success proxies such as bio startups. I think bio startups are particularly difficult for investors to evaluate and many experienced vcs avoid the sector entirely unless they have staff with bio PhDs and even then it’s still pretty hard to evaluate the niche area the startup is working in. Anecdotally, moderately successful bio startups seem much more likely to have a BS product than the average tech startup at a similar level of funding/team size.

Of course, I do think there are founders that are above the bar, but I think starting a new project is actually often very hard and a poor learning environment and I would probably prefer the bar was a bit higher and there were fewer nudges for early career people towards starting new things.

calebp

1y

6

I think that "moods" should be a property of the whole discourse, as opposed to specific posts. I find it a bit annoying when commenters say a specific post has a missing mood - most posts don't aim to represent the whole discourse.

calebp

3y

15

A quickly written model of epistemic health in EA groups I sometimes refer to

I think that many well intentioned ea groups do a bad job cultivating good epistemics. By this I roughly mean the culture of the group does not differentially advantage truthseekjng discussions or other behaviours that helps us figure out what is actually true as opposed to what is convenient or feels nice.

I think that one of the main reasons for this is poor gatekeeping of ea spaces. I do think groups do more and more gate keeping, but they are often not selecting on epistemics as hard as I think they should be. I’m going to say why I think this is and then gesture at some things that might improve the situation. I’m not going to talk at this tjme about why I think it’s important - but I do think it’s really really important.

EA group leaders often exercise a decent amount of control over who should be part of their group (which I think is great). Unfortunately, it’s much easier to evaluate what conclusion a person has come to, than how good were their reasoning processes. So "what a person says they think" becomes the filter for who gets to be in the group as opposed to how do they think. Intuitively I expect a positive feedback loop where groups become worse and worse epistemically as people are incentivised to reach a certain conclusion to be part of the group and future group leaders are drawn from a pool of people with bad epistemics and then reinforce this.

If my model is right there are a few practical takeaways: • be really careful about who you make a group leader or get to start a group (you can easily miss a lot of upside that’s hard to undo later) • make it very clear that your EA group is a place for truth seeking discussion potentially at the expense of being welcoming or inclusive • make rationality/epistemics a more core part of what your group values, idk exactly how to do that - I think a lot of this is making it clear that this is what your group is in part about

I’m hoping to have some better takes on this later, I would strongly encourage the CEA groups team to think about this along with EA group leaders. I don’t think many people are working in this area though I’d also be sad if people fill up the space with low quality content so think really hard about it and try to be careful about what you post.

Lorenzo Buonanno🔸

3y

2

I’m not going to talk at this tjme about why I think it’s important - but I do think it’s really really important.

As someone both trying to start a group and to find someone else to run it so I can move to other places, I'm really curious about your perspective on this.
In my model, a lot of the value of a group comes from helping anyone that's vaguely interested in doing good effectively to better achieve their goals, and to introduce them to online resources, opportunities, and communities.

I would guess that even if the leader has poor epistemics, they can still do a good enough job of telling people: "EA exists, here are some resources/opportunities you might find useful, happy to answer your 101 questions".

I have heard a similar take from someone on the CEA groups team, so I would really want to understand this better.

To be stuck inside of Mobile

3y

1

There seems to be anxiety and concern about EA funds right now. One thread is here.

Your profile says you are the head of EA funds.

Can you personally make a statement to acknowledge these concerns, say this is being looked into, or anything else substantive? I think this would be helpful.

calebp

3y

16

Why handing over vision is hard.

I often see projects of the form [come up with some ideas] -> [find people to execute on ideas] -> [hand over the project].

I haven't really seen this work very much in practice. I have two hypotheses for why.

The skills required to come up with great projects are pretty well correlated with the skills required to execute them. If someone wasn't able to come up with the idea in the first place, it's evidence against them having the skills to execute well on it.
Executing well looks less like firing a canon and more like deploying a heat-seeking missile. In reality, most projects are a sequence of decisions that build on each other, and the executors need to have the underlying algorithm to keep the project on track. In general, when someone explains a project, they communicate roughly where the target is and the initial direction to aim in, but it's much harder to hand off the algorithm that keeps the missile on track.

I'm not saying separating out ideas and execution is impossible, just that it's really hard and good executors are rare and very valuable. Good ideas are cheap and easy to come by, but good execution is expensive.

A formula that I see more often works well is [a person has idea] -> [person executes well in their own idea until they are doing something they understand very well "from the inside" or is otherwise hand over-able] -> person hands over the project to a competent executor.

Niel_Bowerman

3y

6

I agree with this and I appreciate you writing this up. I've also been mentioning this idea to folks after Michelle Hutchinson first mentioned it to me.

calebp

3y

13

The importance of “inside view excitement”

Another model I regularly refer to when advising people on projects to pursue. Quickly written - may come back and try to clarify this later.

I think it’s generally really important for people to be inside view excited about their projects. By which I mean, they think the project is good based on their own model of how the project will interact with the world.

I think this is important for a few reasons. The first obvious one is that it’s generally much more motivating to work on things you thing are good.

The second, and more interesting reason, is that if you are not inside view excited I think (generally speaking) you don’t actually understand why your project will succeed. Which makes it hard to execute well on your project. When people aren’t inside view excited about their project I get the sense they either have the model and don’t actually believe the project is good, or they are just deferring to others on how good it is which makes it hard to execute.

calebp

3y

10

One of my criticisms of criticisms

I often see public criticisms of EA orgs claiming poor execution on some object level activity or falling short on some aspect of the activity (e.g. my shortform about the 80k jobs board). I think this is often unproductive.

In general I think we want to give feedback to change the organisations policy (decision making algorithm), and maybe the EA movements policy. When you publicly criticise an org on some activity you should be aware that you are disincentivising the org from generally doing stuff.

Imagine the case where the org was choosing between scrappily running a project to get data and some of the upside value strategically as opposed to carefully planning and failing to execute fully. I think in these cases you should react differently and from the outside it is hard to know which situation the org was in.

If we also criticised orgs for not doing enough stuff I might feel differently, but this is an extremely hard criticism to make unless you are on the inside. I'd only trust a few people who didn't have inside information to do this kind of analysis.

Maybe a good idea would be to describe the amount of resources that would have had to have gone into the project for you to see the outcome as being reasonably successful ??? Idk seems hard to be well calibrated.

I expect some people to react negatively to this, and think that I am generally discouraging of criticism. I think that I feel moderately about most criticism, neither helpful nor particularly unhelpful. The few pieces of thoughtful criticism I see written up I think are very valuable, but thoughtful criticism in my view is hard to come by and requires substantial effort.

calebp

2y

6

(to be clear, I do donate I just haven't signed the pledge, and I'm confused about how much I am already donating)

I think the main things are:

whilst I think donating now > donating in the future + interest, the cost of waiting to donate is fairly low (if you're not worried about value drift)
I can think of many situations in the past where an extra $10k would have been extremely useful to me to move to more impactful work.
- I don't think that it always makes sense for funders to give this kind of money to people in my position.
- I now have friends who could probably do this for me, but it has some social cost.
I think it's important for me to be able to walk away from my job without worrying about personal finances.
- My job has a certain kind of responsibility that sometimes makes me feel uneasy, and being able to walk away without having another reason not to seems important.
I think I've seen several EAs make poor decisions from a place of poor personal finance and unusual financial security strategies. I think the epistemic effects of worrying about money are pretty real for me.

Also:

If I were trying to have the most impact with my money via donations, I think I would donate to various small things that I sometimes see that funders aren't well positioned to fund. This would probably mean saving my money and not giving right now.
- (I think that this kind of strategy is especially good for me as I have a good sense of what funders can and can't fund - I think people tend to overestimate the set of things funders can't fund)
I don't see why the GWWC 10% number should generalise well to my situation. I don't think it's a bad number. I don't weigh the community prior very strongly relative to my inside view here.

calebp

3y

6

More EAs should give rationalists a chance

My first impression of meeting rationalists was at a AI safety retreat a few years ago. I had a bunch of conversations that were decidedly mixed and made me think that they weren’t taking the project of doing a large amount of good seriously, reasoning carefully (as opposed to just parroting rationalist memes) or any better at winning than the standard EA types that I felt were more ‘my crowd’.

I now think that I just met the wrong rationalists early on. The rationalists that I most admire:

Care deeply about their values
Are careful reasoners, and actually want to work out what is true
Are able to disentangle their views from themselves, making meaningful conversations much more accessible
Are willing to seriously consider weird views that run against their current views

Calling yourself a rationalist or EA is a very cheap signal and I made an error early on (insensitivity to small samples sizes etc.) dismissing their community. Whilst there is still some stuff that I would change, I think that the median EA could move several steps in a ’rationalist’ direction.

Having a rationalist/scout mindset + caring a lot about impact are pretty correlated with me finding someone promising. It’s not essential to having a lot of impact but I am starting to think that EA is doing the altruism (A) part of EA super well and the rationalist are doing the effective (E) part of EA super well.

My go to resources are probably:

The scout mindset - Julia Galef
The codex - Scott Alexander
The sequences highlights - Eliezer Yudkowsky/Less Wrong
The Less Wrong highlights

calebp

3y

6

I adjust upwards on EAs who haven't come from excellent groups

I spend a substantial amount of my time interacting with community builders and doing things that look like community building.

It's pretty hard to get a sense of someone's values, epistemics, agency .... by looking at their CV. A lot of my impression of people that are fairly new to the community is based on a few fairly short conversations at events. I think this is true for many community builders.

I worry that there are some people who were introduced to some set of good ideas first, and then people use this as a proxy for how good their reasoning skills are. On the other hand, it's pretty easy to be in an EA group where people haven't thought hard about different cause areas/interventions/... And come away with the mean take that's not very good despite being relatively good reasoning wise.

When I speak to EAs I haven't met before I try extra hard to get a sense of why they think x and how reasonable a take that is, given their environment. This sometimes means I am underwhelmed by people who come from excellent EA groups, and impressed by people who come from mediocre ones.

You end up winning more Caleb points if your previous EA environment was 'bad' in some sense, all else equal.

(I don't defend why I think a lot of the causal arrow points from the EA environment quality to the EA quality - I may write something on this, another time.)

Jordan Pieters 🔸

3y

4

It's all about the Caleb points man

calebp

3y

6

(crosspost of a comment on imposter syndrome that I sometimes refer to)

I have recently found it helpful to think about how important and difficult the problems I care about are and recognise that on priors I won't be good enough to solve them. That said, the EV of trying seems very very high, and people that can help solve them are probably incredibly useful.

So one strategy is to just try and send lots of information that might help the community work out whether I can be useful, into the world (by doing my job, taking actions in the world, writing posts, talking to people ...) and trust the EA community to be tracking some of the right things.

I find it helpful to sometimes be in a mindset of "helping people reject me is good because if they reject me then it was probably positive EV and that means that the EA community is winning therefore I am winning (even if I am locally not winning).

calebp

3y

3

‘EA is too elitist’ criticisms seem to be more valid from a neartermist perspective than a longtermist one

I sometimes see criticisms around

EA is too elitist
EA is too focussed on exceptionally smart people

I do think that you can have a very outsized impact even if you're not exceptionally smart, dedicated, driven etc. However I think that from some perspectives focussing on outliery talent seems to be the right move.

A few quick claims that push towards focusing on attracting outliers:

The main problems that we have are technical in nature (particularly AI safety)
Most progress on technical problems historically seems to be attributable to a surprisingly small set of the total people working on the problem
We currently don't have a large fraction of the brightest minds working on what I see as the most important problems

If you are more interested in neartermist cause areas I think it's reasonable to place less emphasis on finding exceptionally smart people. Whilst I do think that very outliery-trait people have a better shot at very outliery impact, I don't think that there is as much of an advantage for exceptionally smart people over very smart people.

(So if you can get a lot of pretty smart people for the price of one exceptionally smart person then it seems more likely to be worth it.)

This seems mostly true to me by observation, but I have some intuition that motivates this claim.

AIS is a more novel problem than most neartermist causes, there's a lot of working going in to getting more surface area on the problem as opposed to moving down a well defined path.
Being more novel also makes the problem more first mover-y so it seems important to start with a high density of good people to push onto good trajectories.
The resources for getting up to speed on the latest stuff seemless good than in more established fields.