In general, I'm a big fan of approaches that are optimized around Value of Information. Given EA/longtermism's rapidly growing resources (people and $), I expect that acquiring information to make use of resources in the future is a particularly high EV use of resources today.
I'm excited for this series! I'm a big believer in EAs doing more things out in the world, both for the direct impacts but probably even more for the information value.
For example, I'm thrilled that Longview is getting into nuclear security grantmaking. I think this is:
(disclosure that I contract for Longview on something totally different and learned about this when everyone else did).
I think the sociology of EA will make us overly biased towards research and away from action, even when action would be more effective, in the near and long term. For example, I think there are major limitations to developing AI governance strategies in the absence of working with and talking to governments.
TBC, research is extremely important, and I'm glad the community is so focused on asking and answering important questions, but I'd be really happy to see more people "get after it" the way you have.
Very interesting! I'm often worried about these facets of Longtermism. I also find it a bit weird that lots of longtermists are aware of the explore/exploit trade-off and the importance of encouraging exploration for minimising regret - yet don't nearly encourage it enough. Not that I'm immune, of course.
As a side note, I also liked your writing.
Edit: I find it fitting to link to this xkcd: https://xkcd.com/2353/
Wow, thanks a lot for the work, and for sharing your insights here, I'm really impressed you were able to get involved and contribute on such a massive scale!
Minor thing I stumbled on:
> reaching millions of educated readers with our arguments
If this is based on the upper bound of 20 million followers of the accounts who tweeted about the paper, I'm somewhat sceptical that more than 10% of those have actually read even one of your arguments. Would expect that maybe 5% have read the specific tweet and .1% have gone more in depth on the paper?
Also, I wonder what you think of forecasting as another route to tie longtermists to the reality-mast. It seems like it's much less effortful, but also won't provide nearly as much high quality feedback compared when you actually interact with the systems that you're interested in understanding better.
[Epistemic status: massive extrapolation from a few hard numbers.]
Yeah that paper is just the big one, and just its Twitter audience; there are 7 papers, 100 or so major newspaper spots, and a dozen big Wiki spots. (e.g. The masks paper was on the BBC, ACX, NYT, Wired, Guardian, Mail, MR...) I've not actually estimated the total audience but I would eyeball a 95% CI as like [6m, 300m] using a weak operational audience of "people who read 1+ of our main claims presented as having good evidence".
As for in-depth readers: ~160k people downloaded the papers, 6k of which saved it to Mendeley, the poor sods. 20k deep readers sounds about right.
Millions is probably a safe bet/lower bound: majority won't be via direct twitter reads, but via mainstream media using it in their writing.
With twitter, we have a better overview in the case of our other research on seasonality (still in review!). Altmetric estimate is it was shared with accounts with an upper bound of 13M followers. However, in this case, almost all the shares were due to people retweeting my summary. Per twitter stats, it got 2M actual impressions. Given the fact the NPI research was shared and referenced more, it's probably more >1M reads just on twitter.
Re: forecasting (or bets). In a broad sense, I do agree. In practice I'm a bit skeptical that a forecasting mindset is that good for generating ideas about "what actions to take". "Successful planning and strategy" is often something like "making a chain of low-probability events happen", which seems distinct, or even at tension with typical forecasting reasoning. Also, empirically, my impression is that forecasting skills can be broadly decomposed into two parts - building good models / aggregates of other peoples models, and converting those models into numbers. For most people, the "improving at converting non-numerical information into numbers" part has initially much better marginal returns (e.g. just do calibration trainings...), but I suspect doesn't do that much for the "model feedback".
Thanks for the response, seems like a safe bet, yeah. :)
Re forecasting, "making low-probability events happen" is a very interesting framing, thanks! I still am maybe somewhat more positive about forecasting:
This series explains my part in the EA response to COVID, my reasons for switching from AI alignment work to the COVID response for a full year, and some new ideas the experience gave me. While it is written from my (Jan Kulveit's) personal perspective, I co-wrote the text with Gavin Leech, with input from many others.
The first post covers my main motivation: experimental longtermism.
Possibly the main problem with longtermism and x-risk reduction is the weak and slow feedback loop.
(You work on AI safety; at some unknown time in the future, an existential catastrophe happens, or doesn’t happen, as a result of your work, or not as a result of your work.)
Most longtermists and existential risk people openly admit that the area doesn't have good feedback loops. Still, I think the community at large underappreciates how epistemically tricky our situation is. Disciplines that lack feedback from reality are exactly the ones that can easily go astray.
But most longtermist work is based on models of how the world works - or doesn’t work. These models try to explain why such large risks are neglected, the ways institutions like government or academia are inadequate, how various biases influence public perception and decision making, how governments work during crises, and so on. Based on these models, we take further steps (e.g. writing posts like this, uncovering true statements in decision theory, founding organisations, working at AI labs, going into policy, or organising conferences where we explain to others why we believe the long-term future is important and x-risk is real).
Claim: COVID presented an unusually clear opportunity to put some of our models and theory in touch with reality, thus getting more "experimental" data than is usually possible, while at the same time helping to deal with pandemic. The impact of the actions I mentioned above is often unclear even after many years, whereas in the case of COVID impact of similar actions was observable within weeks and months.
For me personally, there was one more pull. My background is in physics, and in many ways, I still think like a physicist. Physics - in contrast to most of maths and philosophy - has the advantage of being able to put its models in touch with reality, and to use this signal as an important driver in finding out what's true. In modern maths, (basically) whatever is consistent is true, and a guiding principle for what's important to work on is a sense of beauty. To a large extent, the feedback signal in philosophy is what other philosophers think. (Except when a philosophy turns into a political movement - then the signal comes from outcomes such as greater happiness, improved governance, large death tolls, etc.) In both maths and philosophy, the core computation mostly happens "in” humans. Physics has the advantage that in its experiments, "reality itself" does the computation for us.
I miss this feedback from reality in my x-risk work. Note that many of the concrete things longtermists do, like posting on the Alignment Forum or explaining things at conferences, actually do have feedback loops. But these are usually more like maths or philosophy: they provide social feedback, including intuitions about what kinds of research are valuable. One may wonder about the problems with these feedback loops, and what kind of blind-spots or biases they entail.
At the beginning of the COVID crisis, it seemed to me that some of our "longtermist" models were making fairly strong predictions about specific things that would fail - particularly about inadequate research support for executive decision-making. After some hesitation, I decided that if I trusted these models for x-risk mitigation, it made sense to use them to solve COVID as well. And in pretty much every scenario, I learn something.
Over the next year, I and many collaborators tried a number of interventions to limit the damage associated with COVID. While we were motivated by trying to help, basically every intervention was also an experiment, putting some specific model in touch with reality, or attempting to fix some perceived inadequacy. Our efforts have had some direct impact, but from the longtermist perspective, the main source of value is ‘value of information’.
A more detailed description of our work is forthcoming, but briefly: we focused on inadequacies in the world’s modeling, forecasting, and decision support. Legible outputs include our research on non-pharmaceutical interventions, advising major vaccine manufacturers, advising multiple governments, sometimes at the executive level, consulting with bodies such as the European CDC and multiple WHO offices, and reaching millions of educated readers with our arguments, with mostly unknowable effects. I'm fairly confident the efforts made at least one country’s COVID policy not suck during at least one epidemic wave, and moderately confident our efforts influenced multiple countries toward marginally better decisions.
Concretely, here’s a causal graph of some of our efforts:
(Every edge has value of information.)
The sequence of posts, to be released over the next couple of weeks, will cover more detail:
Part of the value of my COVID year depends on whether I can pass on the data I collected, and the updates I made from them. The posts to come discuss some of these.
A year of intense work on COVID likely gave me more macrostrategy ideas, governance insights, and general world-modelling skills than the counterfactual (which would have been mostly solo research from my home office and occasional zoom calls with colleagues from FHI). My general conclusion is that such "experimental longtermist" work is useful, and relatively neglected.
One reason for neglectedness may be the type of reasoning where a longtermist compares the "short-term direct impacts" of similar work with the potential "long-term direct impacts" of a clearly longtermist project, and neglects the value of information term. (Note that a longtermist prioritisation taking value of information into account will often look different from a prioritisation focused on maximising direct impact - e.g. optimising for the value of information will lead to exploring more possible interventions).
My rough guess of the total value of information is a >10% improvement in my decision-making ability about large matters. Adding in what I hope you learn from me, it seems a clearly good investment.
On the margin, more longtermists should do experiments in this spirit; for the future, seize the day.
Some brain noise: "Hard data are the opposite of memes. Nature abhors a vacuum: if you don't have data then memes will rush in to replace it."