Which fraction of the expected effects of neartermist interventions (e.g. global health and development, and animal welfare) flow through longtermis considerations (e.g. longterm effects of changing population size, or expansion of the moral circle)?
Should we be uncertain about whether saving lives is good/bad because of the meat eater problem?
What is the chance that the time of perils hypothesis is true (e.g. how does the existential risk this century compare to that over the next 1 billion years)? How can we get more evidence for/against it? Relevant because, if existential risk is spread out over a long time, reducing existential risk this century has a negligible effect on total existential risk, as discussed by David Thorstad.
How high is the chance of AGI lock-in this century?
What can we do to ensure a bright future if there are advanced aliens on or around Earth (Magnus Vinding's thoughts)? More broadly, should humanity do anything differently due to the possibility of advanced civilisations which did not originite on Earth?
How much weight should one give to the XPT's forecasts? The ones regarding nuclear extinction seem way too pessimistic to be accurate. Superforecasters and domain experters predicted a likelihood of nuclear extinction by 2100 of 0.074 % and 0.55 %. My guess would be something like 10^-6 (10 % of a global nuclear nuclear war involving tens of detonations, 10 % of it escalating to thousands of detonations, and 0.01 % of that leading to extinction), in which case superforecasters would be off by 3 orders of magnitude.
Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm?
We’ve seen two large and extremely capable swarms from OpenAI in the last few months:
* 1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched...
Often folks hit us up because they are thinking of starting an incubator and want advice.
Typically their motivation is either that (a) they have a list of specific things they want built that no one is building,  or (b) they think an ecosystem needs more new projects generally to absorb more talent and deploy more funding effectively.
Here are six questions we often ask prospective teams, to help them figure out what to do. If you're incubator-curious...
Summary:
First, I give several different angles on how I feel about reinforcement learning:
* Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffolding
* Recent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progress
* I’m worried things might get worse:...
Hi Rob,
Nice to know you are interviewing Anders! Some questions: