And it's not so much that I think I have zero evidence: I keep up with progress in AI to some degrees, I have some idea of what the remaining gaps are to general intelligence, I've seen the speed at which capabilities have improved in recent years etc. It's that how to evaluate that evidence is not obvious, and so simply presenting a skeptic with it probably won't move them, especially as the skeptic-in this case you-probably already has most of the evidence I have anyway. If it was just some random person who had never heard of AI asking why I thought the chance of mildly-over-human level AI in 10 years was not far under 1%, there are things I could say. It's just you already know those things, probably, so there's not much point in my saying them to you. 

Yeah, I agree that in some sense saying "we should instantly reject a theory that recommends WD" doesn't not combine super-well with belief in classical U, for the reasons you give. That's compatible with classical U's problems with WD being less bad than NU's problem's with it, is all I'm saying. 

"I'm generally against this sort of appeal to authority. While I'm open to hear the arguments of smart people, we should evaluate those arguments themselves and not the people giving them. So far, I've heard no argument that would change my opinion on this matter."

I think this attitude is just a mistake if your goal is to form the most accurate credences you can. Obviously, it is always good practice to ask people for their arguments rather than only taking what they say on trust. But your evaluation of other people's arguments is fallible, and you know it is fallible. So you should distribute some of your confidence to cases where your personal evaluations of credible people's arguments are just wrong. This isn't the same as failing to question purported experts. I can question an expert, and even disagree with them overall, and still move my credences somewhat towards theirs. (I'm much more confident about this general claim than I am about what credences in ASI in the next decade are or aren't reasonable, or how much credibility anyone saying ASI is coming in the next decade should get.) 

"It all comes down to the question of whether the current tech is relevant for ASI or not. In my estimation, it is not – something else entirely is required. The probability for us discovering that something else just now is low." 

I think Richard's idea is that you shouldn't have *super-high* confidence in your estimation here, but should put some non-negligible credence on the idea that it is wrong, and current progress is relevant. Why be close to certainty about a question that you probably think is hard and that other smart people disagree about being the reasoning? And once you open yourself up to a small chance that current progress is in fact relevant, it then becomes at least somewhat unclear that you should be way below 1% in the chanc of AGI in the relatively near term or in current safety work being relevant. (Not necessarily endorsing the line of thought in this paragraph myself.) 

It seems like if you find it incredible to deny and he doesn't, it's very hard to make further progress :(  I'm on your side about the chance being over 1% in the next decade, I think, but I don't know how I'd prove it to a skeptic, except to gesture and say that capabilities have improved loads in a short time, and it doesn't seem like the are >20 similar sized jumps before AGI. But when I ask myself what evidence I have for "there are not >20 similar sized jumps before AGI" I come up short. I don't necessarily think the burden of proof here is actually on people arguing that the chance of AGI in the next decade is non-negligible though: it's a goal of some serious people within the relevant science, and they are not making zero progress, and some identifiable quantifiable individual capabilities have improved very fast. Plus the extreme difficulty of forecasting technological breakthroughs over more than a couple of years cuts both ways. 

Saying for "other thoughts on why NU doesn't recommend extinction" is a bit of a misnomer here. The Knutsson argument you've just state doesn't even try to show NU doesn't recommend extinction, it just makes a case that it is part of a wider class of more popular theories that also sometimes do this. 

An obvious response to Knutsson is that it also matters in what circumstances a theory recommends extinction, and that NU probably recommends extinction in a wider variety of circumstances where other forms of consequentialism don't, including ones where it is especially counterintuitive, i.e. utopian situations where the possible future risk of extreme suffering  is not literally zero* but very low and every sentient being is aware of this but wants their lives to continue anyway. Also, versions of NU that give some positive weight to happiness even though it can't be outweighed by extreme suffering actually share with classical utilitarians the results that we should kill everyone painlessly when they could be replaced with happier beings, so it's not really a case of both views have counterintuitive implications, about killing everyone, but just in different cases. (Though I guess there are cases where CU would recommend killing everyone but NU does not.) 

*I'm assuming that if extreme suffering can't be outweighed neither can any non-zero chance of extreme suffering. Maybe that's not true on all versions of NU? But I'm guess dropping this will bring other trouble. 

Also, I notice no references to anything about the concentration of power or wealth here. Isn't that probably something we want to avoid if we want to reach a good destination, at least all things being equal?  

Even if we are bad at answering the "what would utopia look like" question, what's the reason to think we'd be any better answering the "what would viatopia look like" question? If we are just as bad or worse at answering the second question, it's either useless or actively counterproductive to switch from utopian to viatopian planning. 

N=1, but I looked at an ARC puzzle https://arcprize.org/play?task=e3721c99, and I couldn't just do it in a few minutes, and I have a PhD from the University of Oxford. I don't doubt that most of the puzzles are trivial for some humans, and some of the puzzles are trivial for most humans or that I could probably outscore any AI across the whole ARC-2 data set. But at the same time, I am a general intelligence, so being able to solve all ARC puzzles doesn't seem like a necessary criteria. Maybe this is the opposite of how doing well on benchmarks doesn't always generalize to real world tasks, and I am just dumb at these but smart overall, and the same could be true for an LLM.

 

 Gavi do vaccines, something that governments and other big bureaucratic orgs sure seem to handle well in other cases. Government funding for vaccines is how we eliminated smallpox, for example. I think "other vaccination programs" are a much better reference class for Gavi than the nebulous category of "social programs" in general. Indeed the Rossi piece you've linked to actually says "In the social program field, nothing has yet been invented which is as effective in its way as the smallpox vaccine was for the field of public health." I'm not sure it is even counting public health stuff as "social programs" that fall under the iron law.

That's not to say that Gavi can actually save a life for $1600, or save millions at $1600 each, or that GiveWell should fund them. But impact of literally zero here seems very implausible.