I'm a computational physicist, I generally donate to global health. I am skeptical of AI x-risk and of big R Rationalism, and I intend on explaining why in great detail.
I am a little concerned that you have immediately jumped to respond to the most controversial part of the post, rather than respond to any of the 9 uncontroversial reasons put forward that your decision to hire a convicted fraudster was a terrible decision.
I personally still don't trust Hanania, and I certainly don't think whether one supports shrimp welfare should have any bearing at all about how we feel about racism accusations, but is it really more important to discuss this rather than the fact that half of the two employees at Manifund have committed billion dollars worth of fraud?
One thing that worries me about discussions around reputation is that it seems like there is an assumption that ones reputation is independent of whether one is actually doing good or not.
Very often, the reason organisations get a bad reputation is because they are doing bad things, and the wider world is correctly associating them with those bad things. Obviously the general public can be irrational or wrong, but there are a significant amount of moral and intelligent people outside the ingroup, and their opinions deserve to be considered.
Often, the reason that decisions are perceived as bad is because they really are bad!
To add further context: The Manifund team currently consists of just two employees: CEO Austin Chen and "Carol N", the pseudonym for Caroline Ellison. Literally 50% of the employees in the organization are convicted financial fraudsters.
Now when people have their funds approved, they will know that decision was made by a convicted fraudster. The article recruiting new hires was written by a convicted fraudster (who was hiding it at the time). Irrespective of her actual role, the fact that she makes up a large percentage of the employee base gives her a significant power in the organization: it's crazy that she is apparently the first hire here.
This is a terrible decision on every dimension you can look at it. It's certainly a brave decision: that does not make it a sane or moral one.
For context, Caroline Ellison has been recently convicted for financial fraud amounting to billions of dollars. Manifund has just hired her for a finance role. They couldn't find a single qualified person that hadn't committed billions of dollars worth of financial fraud?
Nobody has to be diplomatic or mince words here, or couch this in terms of "reputation": this decision is rank incompetence, plainly obvious to anyone with a whit of common sense. Nobody should trust Manifund after this.
I think critiquing EA for a lack of moral imagination is very off the mark. Effective altruism has moral imagination in abundance, and it's one of the movements most admirable qualities.
I would say my problems with the movement is in the opposite direction: is that it lacks sufficient skepticism to temper it's vast moral imagination. Most novel ideas, including moral ideas, are simply wrong for one reason or another. The EA valorisation of novel moral ideas lets ideas be entrenched that have not passed the standards of scrutiny that is present in the actual scientific process that EA emulates.
If you look at the examples given in the article linked, they discuss a new great moral development occuring every couple of centuries. Do you truly believe that EA has come up with 5 or so such revolutionary ideas in a mere two decades?
Three anecdotes from a blog post is not exactly strong evidence. I think these anecdotes do not make the case very well. The economy returned to trend in world war II, but would it have done the same if the Axis had won? It says the american revolution had no effect on the development of freedom and democracy, but is there proof of that? It certainly played a role in the french revolution, for example.
Overall, despite the weak case, there probably are some historical trends that exist, but this doesn't mean that the world can't be radically altered by small changes. The latter is absolutely true. There is no contradiction between the two positions.
I definitely agree that Bayesianism is overrated and overused. I do wonder if some of these pitfalls could be avoided if people started using actual Bayesian statistics, rather than this fake pop-bayes thing where people make single numerical estimates (like saying "10% chance of apocalypse") and nudge the number up and down a bit in response to news.
Maybe I'm being a grumbly math nerd here, but outside of toy examples, actual bayesianism involves updating distributions, not single numbers. And you need to actually run calculations using bayes formula, not guesstimate it.
Another point in favour of your position is that there are a lot of people like me, who think that extinction risk is massively overblown, but are seriously concerned about smaller-scale harms from AI. I would say this is a majority view among the people I know in real life, and is the dominant view on left-wing social media networks such as bluesky.
I will not join a movement that is solely focussed on human extinction risk, because I think the arguments for it are wrong, and I believe such a movement would backfire anyway by helping AI companies with their marketing hype. I would join a movement that cares about the negative effects of AI that are plainly visible right now.
I can add four more to the list of possible dangers:
Effective altruists tend to be more utilitarian, more "maximizing" with regards to ethics, and more willing to bite philosophical bullets, compared to the general population. EA's involvement in AI safety could end up spreading these beliefs to future AI. It's well established how this sort of utilitarian reasoning can go wrong (utility monsters, etc), so this could increase the chance of disaster.
Effective altruists also have an implicit epistemology that differs from typical subject matter experts. If there are flaws in parts of the epistemology of EA or other groups involved in AI safety (which I believe to be true), then these flaws could be absorbed by future AI systems, and give rise to negative outcomes due to poor decisonmaking, etc.
People that are informed by AI safety arguments could decide that extreme actions are necessary to avoid AI doom, resulting in violence or rash actions that cause extreme harm.
The focus on AI safety in particular could result in less attention being paid to other threats to humanity that turn out to be more pressing, but are neglected due to the overrepresentation of AI safety people in the existential threat field.
Point 2 is something that confused me at first as well. The reason they are different is that we are looking at the performance of the top apparent threat. If we were perfectly good estimators, this would be the same as the top actual threat, but we aren't: the threat of the top pick is generally going to be lower than the top actual threat due to uncertainty and the curse.
For the grounded estimator, the process of ranking threats gives useful information, and it means that the top threat picked is much higher than you would get from picking at random. Whereas in the speculative case, we are much closer to just picking at random, and thats reflected in the yellow curve which looks a lot like the power law sampling we are drawing from.
I am a little concerned that you have immediately jumped to respond to the most controversial part of the post, rather than respond to any of the 9 uncontroversial reasons put forward that your decision to hire a convicted fraudster was a terrible decision.
I personally still don't trust Hanania, and I certainly don't think whether one supports shrimp welfare should have any bearing at all about how we feel about racism accusations, but is it really more important to discuss this rather than the fact that half of the two employees at Manifund have committed billion dollars worth of fraud?
One thing that worries me about discussions around reputation is that it seems like there is an assumption that ones reputation is independent of whether one is actually doing good or not.
Very often, the reason organisations get a bad reputation is because they are doing bad things, and the wider world is correctly associating them with those bad things. Obviously the general public can be irrational or wrong, but there are a significant amount of moral and intelligent people outside the ingroup, and their opinions deserve to be considered.
Often, the reason that decisions are perceived as bad is because they really are bad!
To add further context: The Manifund team currently consists of just two employees: CEO Austin Chen and "Carol N", the pseudonym for Caroline Ellison. Literally 50% of the employees in the organization are convicted financial fraudsters.
Now when people have their funds approved, they will know that decision was made by a convicted fraudster. The article recruiting new hires was written by a convicted fraudster (who was hiding it at the time). Irrespective of her actual role, the fact that she makes up a large percentage of the employee base gives her a significant power in the organization: it's crazy that she is apparently the first hire here.
This is a terrible decision on every dimension you can look at it. It's certainly a brave decision: that does not make it a sane or moral one.
For context, Caroline Ellison has been recently convicted for financial fraud amounting to billions of dollars. Manifund has just hired her for a finance role. They couldn't find a single qualified person that hadn't committed billions of dollars worth of financial fraud?
Nobody has to be diplomatic or mince words here, or couch this in terms of "reputation": this decision is rank incompetence, plainly obvious to anyone with a whit of common sense. Nobody should trust Manifund after this.
I think critiquing EA for a lack of moral imagination is very off the mark. Effective altruism has moral imagination in abundance, and it's one of the movements most admirable qualities.
I would say my problems with the movement is in the opposite direction: is that it lacks sufficient skepticism to temper it's vast moral imagination. Most novel ideas, including moral ideas, are simply wrong for one reason or another. The EA valorisation of novel moral ideas lets ideas be entrenched that have not passed the standards of scrutiny that is present in the actual scientific process that EA emulates.
If you look at the examples given in the article linked, they discuss a new great moral development occuring every couple of centuries. Do you truly believe that EA has come up with 5 or so such revolutionary ideas in a mere two decades?
Three anecdotes from a blog post is not exactly strong evidence. I think these anecdotes do not make the case very well. The economy returned to trend in world war II, but would it have done the same if the Axis had won? It says the american revolution had no effect on the development of freedom and democracy, but is there proof of that? It certainly played a role in the french revolution, for example.
Overall, despite the weak case, there probably are some historical trends that exist, but this doesn't mean that the world can't be radically altered by small changes. The latter is absolutely true. There is no contradiction between the two positions.
I definitely agree that Bayesianism is overrated and overused. I do wonder if some of these pitfalls could be avoided if people started using actual Bayesian statistics, rather than this fake pop-bayes thing where people make single numerical estimates (like saying "10% chance of apocalypse") and nudge the number up and down a bit in response to news.
Maybe I'm being a grumbly math nerd here, but outside of toy examples, actual bayesianism involves updating distributions, not single numbers. And you need to actually run calculations using bayes formula, not guesstimate it.
Another point in favour of your position is that there are a lot of people like me, who think that extinction risk is massively overblown, but are seriously concerned about smaller-scale harms from AI. I would say this is a majority view among the people I know in real life, and is the dominant view on left-wing social media networks such as bluesky.
I will not join a movement that is solely focussed on human extinction risk, because I think the arguments for it are wrong, and I believe such a movement would backfire anyway by helping AI companies with their marketing hype. I would join a movement that cares about the negative effects of AI that are plainly visible right now.
I can add four more to the list of possible dangers:
Point 1 was a typo, thanks for pointing it out!
Point 2 is something that confused me at first as well. The reason they are different is that we are looking at the performance of the top apparent threat. If we were perfectly good estimators, this would be the same as the top actual threat, but we aren't: the threat of the top pick is generally going to be lower than the top actual threat due to uncertainty and the curse.
For the grounded estimator, the process of ranking threats gives useful information, and it means that the top threat picked is much higher than you would get from picking at random. Whereas in the speculative case, we are much closer to just picking at random, and thats reflected in the yellow curve which looks a lot like the power law sampling we are drawing from.