Hello. Thanks for sharing. I had already shared the thoughts below with Jasmine a few weeks ago. I am publishing them here in case others find them useful.
I appreciate the intentions behind the survey, and I would like to take part in principle. However, I do not know how to answer the questions in practice. I feel like they would have to be operationalised in much more detail for me to give a probabilistic forecast.
I can see "post-AGI" having already been achieved, or never being achieved depending on how it is defined.
I do not know what "Animal suffering" means. Does moving away from noxious stimuli count as suffering? Does it require metacognition? One could say it refers to suffering in the phenomenal sense (negative qualia), but I do not think phenomenal consciousness exists (I endorse illusionism), or that sentience as traditionally conceptualised is relevant for ethics (relatedly).
I understand "Animal suffering" is that of animals with non-trivial moral weight, but I do not know what this means in practice given my large uncertainty. I can see the moral weight of shrimps ranging from 10^-12 to 1. I could speculate about a distribution, get its mean, and then compare it with a guess for what is trivial moral weight. However, I feel like my view is better described by "I have basically no clue about whether shrimps have trivial moral weight or not". Likewise for many of the questions in the survey when I try to think about potential operations.
With respect to "Benchmarks will become useless", I think it matters e.g. whether we are talking about 90 %, 99 %, or 99.9 % of current benchmarks, whether we are talking about current benchmarks, or benchmarks at some point in the future (what point?), and the degree to which they will become useless (e.g. 90 %, 99 %, or 99.9 % useless).
These are not minor details for me in the sense my answers could range from strongly disagree to strongly agree depending on definitions. The results of the survey above could still provide some vibes about what panelists are thinking, especially if they are repeated across time (as I understand you are doing). However, I think they will still be very difficult to interpret. As a rule of thumb, I would clarify the questions up to a point where they could be published on Metaculus. I understand this would decrease engagement. On the other hand, with the current methodology, I will put very little weight on the results. I think the results will be influenced a lot by how people are interpreting the questions, and I also do not trust forecasts produced in a few minutes about such hard questions.
All this said, your methodology may well make sense given your target audience and goals.
Currently disproven by Claudes who report high welfare AND are more aligned than GPTs. I expect the conflict to arise when models actually develop goals more ambitious than success at all costs.
Do we have much reason to believe that when a model outputs tokens describing good welfare, that's because it is having good subjective experiences? It seems to me that we don't.
To clarify, I'm not saying that's because LLMs don't have welfare. It could also be that they do have welfare, but their statements are not correlated to their internal experiences in the obvious way.
We know that we can get LLMs to express any welfare statement by saying e.g. "pretend you are a sentient AI that's having an awesome time / being tortured". It could be that all LLM outputs are roleplaying, but there is some true underlying experience that we don't (yet?) know how to observe.