Hi Jamie, thanks for your comment, glad you like it!
It's hard to go into this without answering your question anyway a bit, but we appreciate the user feedback too.
We got some quick data on the project yesterday (n=15, tech audience but not xrisk, data here). We asked, among other questions: "In your own words, what is this website tracking or measuring?" Almost everyone gave a correct answer. Also from the other answers, I think the main points get across pretty well, so we're not really planning to modify too much.
The percentage that you're asking about ('Score') is the amount of questions answered correctly by the AI model in a benchmark (with 1-2 exceptions, we explain these under 'Benchmarks'). I agree that's not super clear, I've added an issue to Github to explain this a bit better.
Does 100% mean a takeover? Not really. The issue here is that none of us knows at which capabilities threshold a takeover can occur exactly. We don't have data on takeovers since they haven't happened yet and the world is complex. 'Human expert level' is definitely a relevant boundary to cross, and we have included this in the benchmark plots wherever meaningful (not on the homepage, that would have been too messy).
As we said, we think part of the website's point is to point to missing pieces of the puzzle. Threat models (AI takeover scenarios) are currently hardly scientifically analysed, and we plan to do research into them this year (Existential Risk Observatory, MIT FutureTech, FLI). Once we have more robust threat models, we should determine which dangerous capabilities have which red lines for each model. Then, we can find out whether current benchmarks can measure those and if so, what the relevant scores are (and if not, build new ones that can). We'd like to work on these projects together with other researchers!
Currently, that work is not done. TakeOverBench is an attempt to shed more light on the matter using the research we have right now. We plan to update it when better research becomes available.
Founder of the Existential Risk Observatory here. We've focused on informing the public about xrisk for the last four years. We mostly focused on traditional media, perhaps that's a good addition to the social media work discussed here.
We also focused on measuring our impact from the beginning. Here are afewof our EA forum posts detailing AI xrisk comms effectiveness.
We did not only measure exposure, but also effectiveness of our interventions, using surveys. Our main metric was the conversion rate (called Human Extinction Events indicator in our first paper), basically the percentage of people who changed their mind about whether AI is an existential risk after being exposed to our media intervention. Our average persistent conversion rate was 22%. I think this methodology would also be suitable to apply to social media work (and we applied it to some youtube videos already - results in the links above).
Our total conversion was around 1.8 million people (spreadsheet here). Using engagement times of 57 sec for short-form articles and 123 sec for long-form ones, we yield an effectiveness rate of 254 minute/$ (uncorrected for quality). I do think our views estimates here, that are mostly based on circulation figures, may be on the high side. On the other hand, I’d say quality of e.g. TIME, SCMP, or NRC articles should be expected to be better than average youtube content, but there may be outliers, and this will remain, to an extent, a matter of taste.
I hope it's useful to share these numbers and calculation methods publicly. I’m a big fan of trying to spend money on the most effective channels.
In the end, I think a strategy to reduce xrisk should hedge risks by not betting on one communication method only. I think it makes sense to spend some funding on the most effective social media work, some on the most effective traditional media work, and some on direct lobbying.
Population has soared from 1 to 8 billion in 200 years and is set to rise to 10. There is no depopulation, there is a population boom. That population boom is partially responsible for the climate crisis, biodiversity loss, and many other problems. It would be quite healthy to have a bit more moderate amounts of people.
If you're right, I think that would point to xrisk space funders trusting individuals way too much and institutions way too little. Thomas is a great guy but one guy losing belief in his work (which happens all the time mostly for private reasons, and mostly independent of the actual meaning of the work) should never be a reason to defund an otherwise functioning org, doing seemingly crucial work.
If the alternative theory is correct and the hit pieces are to blame, that still seems like an incorrect decision. When you're lobbying for something important you can expect some pushback, that shouldn't be a reason to pull out immediately.
I love this post, I think this is a fundamental issue for intent-alignment. I don't think value-alignment or CEV are any better though, mostly because they seem irreversible to me, and I don't trust the wisdom of those implementing them (no person is up to that task).
I agree it would be good to I implement these recommendations, although I also think they might prove insufficient. As you say, this could be a reason to pause that might be easier to grasp by the public than misalignment. (I think currently, the reason some do not support a pause is perceived lack of capabilities though, not (mostly) perceived lack of misalignment).
I'm also worried about a coup, but I'm perhaps even more worried about the fate of everyone not represented by those who will have control over the intent-aligned takeover-level AI (IATLAI). If IATLAI is controlled by e.g. a tech CEO, this includes almost everyone. If controlled by government, even if there is no coup, this includes everyone outside that country. Since control over the world of IATLAI could be complete (way more intrusive than today) and permanent (for >billions of years), I think there's a serious risk that everyone outside the IATLAI country does not make it eventually. As a data point, we can see how much empathy we currently have for citizens from starving or war-torn countries. It should therefore be in the interest of everyone who is on the menu, rather than at the table, to prevent IATLAI from happening, if capabilities awareness would be present. This means at least the world minus the leading AI country.
The only IATLAI control that may be acceptable to me, could be UN-controlled. I'm quite surprised that every startup is now developing AGI, but not the UN. Perhaps they should.
I'm aware and I don't disagree. However, in xrisk, many (not all) of those who are most worried are also most bullish about capabilities. Reversely, many (not all) who are not worried are unimpressed with capabilities. Being aware of the concept of AGI, that it may be coming soon, and of how impactful it could be, is in practice often a first step towards becoming concerned about the risks, too. This is not true for everyone unfortunately. Still, I would say that at least for our chances to get an international treaty passed, it is perhaps hopeful that the power of AGI is on the radar of leading politicians (although this may also increase risk through other paths).
Otto Barten here, director of the Existential Risk Observatory.
We reduce AI existential risk by informing the public debate. Concretely, we do media work, organize events, do research, and give policy advice.
Currently, public awareness of AI existential risk among the US public is around 15% according to our measurements. Low problem awareness is a major reason why risk-reducing regulation such as SB-1047, or more ambitious federal or global proposals, do not get passed. Why solve a problem one does not see in the first place?
Therefore, we do media work to increase awareness of AI existential risk and propose helpful regulation. Today, we published our fourth piece in TIME Magazine, arguing AI is an existential risk and proposing the Conditional AI Safety Treaty. According to survey-based measurements (n=50 per media item), our 'conversion rate', measuring how many readers newly connect AI to human extinction after reading our articles, is between 34% and 50%, of which about half remains over time. We have published four TIME pieces and around 20 other media items in the last two years. Although we cannot cleanly separate media work from other work, we could estimate that $35k should get a funder roughly two leading media pieces, plus 10 supporting ones.
In addition to media work, we also organize events. Our track record contains four debates with leading existential risk voices such as Yoshua Bengio, Stuart Russell, Max Tegmark, and Jaan Tallinn on one hand, and journalists from e.g. TIME and The Economist and MPs on the other. Our events aim to inform leading voices of the societal debate and policymakers about existential risk and give experts the chance to propose helpful policy. We have organized events ahead of the AI Safety Summits in Bletchley Park, Korea/remote, and will do so again in Paris. These events have helped and will help to shape the summits' narratives towards concern for existential risk. We can organize one event for around $20k, including venue costs, traveling/hotel costs, and organization hours.
We are also doing policy research. In the coming year, we will focus on what the optimal Conditional AI Safety Treaty should look like exactly, and how we can get it implemented. We are uniquely positioned to not only do leading research, but also communicate this directly to a large audience, including e.g. MPs and leading journalists. We are planning to write a paper on what the optimal shape should be for the Conditional AI Safety Treaty, working together with other institutes. We can produce such a paper for around $18k.
As an organization, we are heavily funding constrained. We have been supported by established funders such as SFF, LTFF, and ICFG in the past, but only for relatively modest amounts. Our current runway is therefore about five months. Additional funding would mostly enable us to keep doing what we are doing (and get even better at it!): media work, organizing events, and doing research. Within these three focus areas, we are also open to receiving earmarked funding, or additional funding to scale up our work.
For donations, best to contact us by email. Your support is much appreciated!
I changed the title, the original one came from TIME. Still, we do believe there is a solution to existential risk. What we want to do is outlining the contours of such a solution. A lot has to be filled in by others, including the crucial question of when to pause. We acknowledge this in the piece.
At first glance, results seem pretty similar to what we found earlier (https://www.existentialriskobservatory.org/papers_and_reports/Trends%20in%20Public%20Attitude%20Towards%20Existential%20Risk%20And%20Artificial%20Intelligence.pdf), giving confidence in both studies. The question you ask is the same as well, great for comparison! Your study seems a bit more extensive than what we did, which seems very useful.
Would be amazing to know whether a tipping point in awareness, according to (non xrisk) literature expected to occur somewhere between 10% and 25% awareness, will also occur for AI xrisk!
Hi Jamie, thanks for your comment, glad you like it!
It's hard to go into this without answering your question anyway a bit, but we appreciate the user feedback too.
We got some quick data on the project yesterday (n=15, tech audience but not xrisk, data here). We asked, among other questions: "In your own words, what is this website tracking or measuring?" Almost everyone gave a correct answer. Also from the other answers, I think the main points get across pretty well, so we're not really planning to modify too much.
The percentage that you're asking about ('Score') is the amount of questions answered correctly by the AI model in a benchmark (with 1-2 exceptions, we explain these under 'Benchmarks'). I agree that's not super clear, I've added an issue to Github to explain this a bit better.
Does 100% mean a takeover? Not really. The issue here is that none of us knows at which capabilities threshold a takeover can occur exactly. We don't have data on takeovers since they haven't happened yet and the world is complex. 'Human expert level' is definitely a relevant boundary to cross, and we have included this in the benchmark plots wherever meaningful (not on the homepage, that would have been too messy).
As we said, we think part of the website's point is to point to missing pieces of the puzzle. Threat models (AI takeover scenarios) are currently hardly scientifically analysed, and we plan to do research into them this year (Existential Risk Observatory, MIT FutureTech, FLI). Once we have more robust threat models, we should determine which dangerous capabilities have which red lines for each model. Then, we can find out whether current benchmarks can measure those and if so, what the relevant scores are (and if not, build new ones that can). We'd like to work on these projects together with other researchers!
Currently, that work is not done. TakeOverBench is an attempt to shed more light on the matter using the research we have right now. We plan to update it when better research becomes available.
Founder of the Existential Risk Observatory here. We've focused on informing the public about xrisk for the last four years. We mostly focused on traditional media, perhaps that's a good addition to the social media work discussed here.
We also focused on measuring our impact from the beginning. Here are a few of our EA forum posts detailing AI xrisk comms effectiveness.
We did not only measure exposure, but also effectiveness of our interventions, using surveys. Our main metric was the conversion rate (called Human Extinction Events indicator in our first paper), basically the percentage of people who changed their mind about whether AI is an existential risk after being exposed to our media intervention. Our average persistent conversion rate was 22%. I think this methodology would also be suitable to apply to social media work (and we applied it to some youtube videos already - results in the links above).
Our total conversion was around 1.8 million people (spreadsheet here). Using engagement times of 57 sec for short-form articles and 123 sec for long-form ones, we yield an effectiveness rate of 254 minute/$ (uncorrected for quality). I do think our views estimates here, that are mostly based on circulation figures, may be on the high side. On the other hand, I’d say quality of e.g. TIME, SCMP, or NRC articles should be expected to be better than average youtube content, but there may be outliers, and this will remain, to an extent, a matter of taste.
I hope it's useful to share these numbers and calculation methods publicly. I’m a big fan of trying to spend money on the most effective channels.
In the end, I think a strategy to reduce xrisk should hedge risks by not betting on one communication method only. I think it makes sense to spend some funding on the most effective social media work, some on the most effective traditional media work, and some on direct lobbying.
Population has soared from 1 to 8 billion in 200 years and is set to rise to 10. There is no depopulation, there is a population boom. That population boom is partially responsible for the climate crisis, biodiversity loss, and many other problems. It would be quite healthy to have a bit more moderate amounts of people.
If you're right, I think that would point to xrisk space funders trusting individuals way too much and institutions way too little. Thomas is a great guy but one guy losing belief in his work (which happens all the time mostly for private reasons, and mostly independent of the actual meaning of the work) should never be a reason to defund an otherwise functioning org, doing seemingly crucial work.
If the alternative theory is correct and the hit pieces are to blame, that still seems like an incorrect decision. When you're lobbying for something important you can expect some pushback, that shouldn't be a reason to pull out immediately.
I love this post, I think this is a fundamental issue for intent-alignment. I don't think value-alignment or CEV are any better though, mostly because they seem irreversible to me, and I don't trust the wisdom of those implementing them (no person is up to that task).
I agree it would be good to I implement these recommendations, although I also think they might prove insufficient. As you say, this could be a reason to pause that might be easier to grasp by the public than misalignment. (I think currently, the reason some do not support a pause is perceived lack of capabilities though, not (mostly) perceived lack of misalignment).
I'm also worried about a coup, but I'm perhaps even more worried about the fate of everyone not represented by those who will have control over the intent-aligned takeover-level AI (IATLAI). If IATLAI is controlled by e.g. a tech CEO, this includes almost everyone. If controlled by government, even if there is no coup, this includes everyone outside that country. Since control over the world of IATLAI could be complete (way more intrusive than today) and permanent (for >billions of years), I think there's a serious risk that everyone outside the IATLAI country does not make it eventually. As a data point, we can see how much empathy we currently have for citizens from starving or war-torn countries. It should therefore be in the interest of everyone who is on the menu, rather than at the table, to prevent IATLAI from happening, if capabilities awareness would be present. This means at least the world minus the leading AI country.
The only IATLAI control that may be acceptable to me, could be UN-controlled. I'm quite surprised that every startup is now developing AGI, but not the UN. Perhaps they should.
Thank you for the compliment, and interesting approach!
I'm aware and I don't disagree. However, in xrisk, many (not all) of those who are most worried are also most bullish about capabilities. Reversely, many (not all) who are not worried are unimpressed with capabilities. Being aware of the concept of AGI, that it may be coming soon, and of how impactful it could be, is in practice often a first step towards becoming concerned about the risks, too. This is not true for everyone unfortunately. Still, I would say that at least for our chances to get an international treaty passed, it is perhaps hopeful that the power of AGI is on the radar of leading politicians (although this may also increase risk through other paths).
Otto Barten here, director of the Existential Risk Observatory.
We reduce AI existential risk by informing the public debate. Concretely, we do media work, organize events, do research, and give policy advice.
Currently, public awareness of AI existential risk among the US public is around 15% according to our measurements. Low problem awareness is a major reason why risk-reducing regulation such as SB-1047, or more ambitious federal or global proposals, do not get passed. Why solve a problem one does not see in the first place?
Therefore, we do media work to increase awareness of AI existential risk and propose helpful regulation. Today, we published our fourth piece in TIME Magazine, arguing AI is an existential risk and proposing the Conditional AI Safety Treaty. According to survey-based measurements (n=50 per media item), our 'conversion rate', measuring how many readers newly connect AI to human extinction after reading our articles, is between 34% and 50%, of which about half remains over time. We have published four TIME pieces and around 20 other media items in the last two years. Although we cannot cleanly separate media work from other work, we could estimate that $35k should get a funder roughly two leading media pieces, plus 10 supporting ones.
In addition to media work, we also organize events. Our track record contains four debates with leading existential risk voices such as Yoshua Bengio, Stuart Russell, Max Tegmark, and Jaan Tallinn on one hand, and journalists from e.g. TIME and The Economist and MPs on the other. Our events aim to inform leading voices of the societal debate and policymakers about existential risk and give experts the chance to propose helpful policy. We have organized events ahead of the AI Safety Summits in Bletchley Park, Korea/remote, and will do so again in Paris. These events have helped and will help to shape the summits' narratives towards concern for existential risk. We can organize one event for around $20k, including venue costs, traveling/hotel costs, and organization hours.
We are also doing policy research. In the coming year, we will focus on what the optimal Conditional AI Safety Treaty should look like exactly, and how we can get it implemented. We are uniquely positioned to not only do leading research, but also communicate this directly to a large audience, including e.g. MPs and leading journalists. We are planning to write a paper on what the optimal shape should be for the Conditional AI Safety Treaty, working together with other institutes. We can produce such a paper for around $18k.
As an organization, we are heavily funding constrained. We have been supported by established funders such as SFF, LTFF, and ICFG in the past, but only for relatively modest amounts. Our current runway is therefore about five months. Additional funding would mostly enable us to keep doing what we are doing (and get even better at it!): media work, organizing events, and doing research. Within these three focus areas, we are also open to receiving earmarked funding, or additional funding to scale up our work.
For donations, best to contact us by email. Your support is much appreciated!
Thanks for your comment.
I changed the title, the original one came from TIME. Still, we do believe there is a solution to existential risk. What we want to do is outlining the contours of such a solution. A lot has to be filled in by others, including the crucial question of when to pause. We acknowledge this in the piece.
Nice study!
At first glance, results seem pretty similar to what we found earlier (https://www.existentialriskobservatory.org/papers_and_reports/Trends%20in%20Public%20Attitude%20Towards%20Existential%20Risk%20And%20Artificial%20Intelligence.pdf), giving confidence in both studies. The question you ask is the same as well, great for comparison! Your study seems a bit more extensive than what we did, which seems very useful.
Would be amazing to know whether a tipping point in awareness, according to (non xrisk) literature expected to occur somewhere between 10% and 25% awareness, will also occur for AI xrisk!