Do you have any recommendations for doing threat modeling well? In particular, resources that seem applicable to many different risks - nuclear, AI, bio.
Good find, thanks! I'm not very keen on instructing teams to run bug bounties and not other mechanisms, so am not particularly enthusiastic about this.
It looks like this would focus on infosecurity of the AI systems being used (i.e. can this weapon's AI be hacked?) rather than testing for potential vulnerabilities from the AI systems themself.
Yeah good find, I also think that passes the bar. Although I do think people have generally overestimated GPT's essay-writing ability compared to humans, and think I might be falling for that here.
I'm not planning to change the doc because Bing's AI wasn't released by Feb 23, but if you think it should be included (which would be reasonable given OpenAI pretty obviously made this before Feb 23), it would mean:
Experts expected 9 milestones to be met vs actually 11 milestones
The calibration curve looks four percentage points worse at the 10% mark
Bulls' Brier score: 0.29
Experts' Brier score: 0.24
Bears' Brier score: 0.29
I've added it to this tracker of milestones (feel free to request edit access).
The experts thought beating humans at Angry Birds would be relatively easy - they put a 90% chance of it being feasible by now. The surprise is that it has not been done - although I think this is mostly explained by no labs seriously trying it.
Hmm I disagree on the numbers - have I got something wrong in the below?
If you assigned 0% or 100% by coin flip, you would get a Brier score of 0.5 (half the time you would get 0, half the time you would get 1), and if you assigned a random probability between 0% and 100% for every question, you would get a Brier score of 0.33. If you put 50% on everything you would indeed get 0.25.
As the experts had to give 10%, 50%, and 90% forecasts, if they had done this at random they would have ended up with a score of 0.36 [1].
So I think they - including the bullish and bearish groups - still did a fair bit better than random, which would be 0.36 in this context. And all simulated groups did better than the 'randomized' group which got a Brier score of 0.31 in my randomization. This does seem like worthwhile context to add though.
I was also looking at https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Data (the dataset itself; the summary doesn't include the milestones). This new version seems like total garbage. The experts continue to predict several of the milestones are five years out, including milestones that were achieved by ChatGPT (ie a few months after the survey) and at least one milestone that had already clearly been achieved by the time the survey was released!
I've only given the data a quick look and found it hard to analyse - but yeah, many of the forecasts look bad. But some of the medians (I think- from eyeballing data) seem not terrible - the 'create top forty song' shifted from 10 year median to ~5 year median. The 'answer open-ended questions' shifted from 10 year median to ~3 years.
But like you say, for many of the milestones I resolved as being met before this survey went out, they still have medians >0 years from now so - if I'm right in my judgements - the experts seem pretty poorly clued up on recent developments across the field.
This is great - thanks for this comment! I've gone through each to explain my reasoning. Your comments/sources changed my opinion on Starcraft and Explain - I've updated the post and scores to reflect this, and think the conclusion is now the same but slightly weaker, because the experts' Brier score is 0.2 points worse, but the comparative Brier scores are also worse to a similar amount. There's also my reasoning for other milestones in the appendix (and I've copy-pasted some of them below).
Zach Stein-Perlman from AI Impacts said that he thought "efficiently sort very large lists" and "write good Python code" were false, because the questions said it had to be done in a certain way by a certain type of neural net, and that wasn't how it was done.
On "efficiently sort very large lists", I think Zach ended up deferring to me on this. But no one has red-teamed my thinking on this. Here's my source:
“We compared this algorithm against common sorting approaches and measured its performance for up to 1 billion normally-distributed double-precision keys. The results show that our approach yields an average 3.38x performance improvement over C++ STL sort, which is an optimized Quicksort hybrid, 1.49x improvement over sequential Radix Sort, and 5.54x improvement over a C++ implementation of Timsort, which is the default sorting function for Java and Python.”
On "write good Python code", Zach and I (at least initially) disagreed. The criteria specified Suppose the system is given only:
A specification of what counts as a sorted list
Several examples of lists undergoing sorting by quicksort
The concern was that LLMs like Codex are trained on open source code, including examples of quicksort algorithms. So they are given more than just ‘a specification of what counts’ and ‘examples of lists undergoing sorting’. I’m unsure, but I think 'given' in the requirement refers to the goal specification post-training. Otherwise, the wording should be 'suppose the system is trained only on...' Therefore, Codex does count as only being given the stated requirements (or less).
For example, the ‘Starcraft’ milestone uses the wording ‘Beat the best human Starcraft 2 players at least 50% of the time, given a video of the screen’. For this milestone, I think the AI can be trained on more than the video of the screen, it just can’t use more than this while playing.
But could definitely be convinced otherwise on this, I might be making a conceptual misunderstanding. And I haven't given Zach a chance to review my argument above.
I was planning to count "transcribe as well as humans" as false, based on https://interactiveaimag.org/columns/is-ai-at-human-parity-yet-a-case-study-on-speech-recognition/ . Maybe the top labs could achieve this with a year of work, but I think the question specifies they need to do as well as the best human transcriptionists, and right now they don't seem close.
The question specifies the AI needs to do "as well as the typical human" - I agree AI is not quite as good as the best human transcriptionists, but think it is better than the typical human.
OpenAI says Whisper “approaches human level robustness and accuracy on English speech recognition.” More detail is given in page 10 of the OpenAI report, which concludes “Whisper’s English ASR performance is not perfect but very close to human-level accuracy”.
However, this is when compared to “professional human transcribers”. Given Whisper sometimes performs better than professional human transcribers, and always performs to a similar level it seems very likely that Whisper regularly does "as well as a typical human”. I think Lorenzo's analysis below seems reasonable too.
I counted "translate as well as bilingual humans" as true based on a few quick tests of ChatGPT; I'm curious if you have some specific source for why it's false.
I have questionable anecdata! I asked two friends who are fluent in two languages but not skilled in translation, and one who has done a university module in translation, and all three thought they were better than AI at translating web pages (I think all three seem modest in general), although AI does a fine job for most uses.
This article notes the worse performance from AI when interpreting context and cultural nuances and says that “Le Monde explains that “the team (of its English language edition) is composed of eight journalists (…) responsible for selecting articles for translation, editing the translated versions and managing the English home page of the site. The translation is done by international agencies with the help of an artificial intelligence tool. Native English-speaking journalists do the selection and editing of the articles”.”
Not-skilled-in-translation humans are still paid to translate writing between English and Chinese (I know of one organisation that does this and the CEO chooses to use AI to assist but not replace human translators, where the human translators are bilingual students not trained in translation). It does seem possible that this is just business lagging behind though.
I don't think AI has won at Starcraft. The last word I've heard from this was https://www.extremetech.com/extreme/301325-deepminds-starcraft-ii-ai-can-now-defeat-99-8-percent-of-human-players , where AlphaStar could beat 99.8% of humans but not the absolute champions. I haven't seen any further progress on this since 2019. Again, it's possible that a year of concerted effort could change this, but that seems speculative. See also https://www.reddit.com/r/starcraft/comments/uakohx/why_cant_we_make_a_perfect_ai_for_starcraft/
I'm convinced by your argument/sources here, and this reddit thread. It does seem open to interpretation if "the best human players" refers to pros (which it did beat >50% of the time), or the literal very best player - but I think this should probably be interpreted as the very best. Given it reached a top 20 score but only beat Serral 2/3 times when heavy restrictions were placed on Serral, I've updated this to false.
I'm surprised you judged "high marks for a high school essay" as false; this seems like a central use case for ChatGPT and Bing/GPT4.
I was also surprised by this. Others disagreed with me and we discussed a bit more in this LW comment thread. Here's my reasoning:
I couldn't find this done and think, by now, someone would have submitted a fully ChatGPT-generated high school essay and talked about it publicly if it had gotten high marks.I've seen some evidence of cherry-picking paragraphs leading to a mid/low-level, e.g. this article describes someone who got a passing mark (53) on a university social policy essay.
This high school teacher judged two ChatGPT-generated history essays as “below average, scoring a 9/20 or lower”. This Guardian article says, uncited, that ‘academics have generated responses to exam queries that they say would result in full marks if submitted by an undergraduate’. I think, if this claim were true, there would be more evidence.
I was planning to judge "concisely explain game play" as true, based on https://www.forbes.com/sites/carlieporterfield/2022/11/22/metas-ai-gamer-beat-humans-in-diplomacy-using-strategy-and-negotiation/, which is testing basically this skill. Also, I was able to play a partial game of chess with ChatGPT where it explained all its moves - before it started hallucinating and making moves which were impossible. Still, it seemed to have the "explanation" skill down pat! I imagine if you asked it to explain why a chess engine made a given move, it would give a pretty plausible answer.
Good point! This is an update for me - I wouldn't have guessed ChatGPT could explain chess moves. The question specifies "For any computer game that can be played well by a machine". I just played Scrabble with ChatGPT and it explained its moves well. And I asked ChatGPT to imagine it was watching Agent57 play Montezuma's revenge for 10 seconds and it gave a very clear summary of what it imagined Agent57 was doing and why.
So I'm just about convinced enough that it could do this for all games - and have changed to True.
I have a rough (i.e. considered for <15 minutes) take: if top labs one year ago had attempted these particular milestones, and had the same policies on disclosing capabilities as they currently seem to, then there's a 40-50% chance they would have achieved 2 of Angry Birds,Atari fifty ,Laundry and Go low by now. But I don't put much weight on my prediction, whereas I put a lot more weight on my analysis of what has happened (though this is also somewhat subjective!).
I agree though that checking what has actually happened ends up being more conservative than the original "Ignore the question of whether they would choose to" , which makes the expert forecasts worse than they seem to be here. This is a weakness of this analysis! And of the resolvability of the original survey.
Do you have an estimate of how many of the tasks would have been achieved by now if labs tried a year ago?
neglectedness of the latter outweighing increased tractability of the former
Do you have any recommendations for doing threat modeling well? In particular, resources that seem applicable to many different risks - nuclear, AI, bio.
Good find, thanks! I'm not very keen on instructing teams to run bug bounties and not other mechanisms, so am not particularly enthusiastic about this.
It looks like this would focus on infosecurity of the AI systems being used (i.e. can this weapon's AI be hacked?) rather than testing for potential vulnerabilities from the AI systems themself.
How unusual does he think the current policy interest in AI safety is? Will this be a temporary window or an ever-increasing level of interest?
Yeah good find, I also think that passes the bar. Although I do think people have generally overestimated GPT's essay-writing ability compared to humans, and think I might be falling for that here.
I'm not planning to change the doc because Bing's AI wasn't released by Feb 23, but if you think it should be included (which would be reasonable given OpenAI pretty obviously made this before Feb 23), it would mean:
I've added it to this tracker of milestones (feel free to request edit access).
The experts thought beating humans at Angry Birds would be relatively easy - they put a 90% chance of it being feasible by now. The surprise is that it has not been done - although I think this is mostly explained by no labs seriously trying it.
Hmm I disagree on the numbers - have I got something wrong in the below?
If you assigned 0% or 100% by coin flip, you would get a Brier score of 0.5 (half the time you would get 0, half the time you would get 1), and if you assigned a random probability between 0% and 100% for every question, you would get a Brier score of 0.33. If you put 50% on everything you would indeed get 0.25.
As the experts had to give 10%, 50%, and 90% forecasts, if they had done this at random they would have ended up with a score of 0.36 [1].
So I think they - including the bullish and bearish groups - still did a fair bit better than random, which would be 0.36 in this context. And all simulated groups did better than the 'randomized' group which got a Brier score of 0.31 in my randomization. This does seem like worthwhile context to add though.
[(1-0.1)^2 + (0-0.1)^2 + (1-0.5)^2 + (0-0.5)^2 + (1-0.9)^2 + (1-0.1)^2] / 6
I've only given the data a quick look and found it hard to analyse - but yeah, many of the forecasts look bad. But some of the medians (I think- from eyeballing data) seem not terrible - the 'create top forty song' shifted from 10 year median to ~5 year median. The 'answer open-ended questions' shifted from 10 year median to ~3 years.
But like you say, for many of the milestones I resolved as being met before this survey went out, they still have medians >0 years from now so - if I'm right in my judgements - the experts seem pretty poorly clued up on recent developments across the field.
This is great - thanks for this comment! I've gone through each to explain my reasoning. Your comments/sources changed my opinion on Starcraft and Explain - I've updated the post and scores to reflect this, and think the conclusion is now the same but slightly weaker, because the experts' Brier score is 0.2 points worse, but the comparative Brier scores are also worse to a similar amount. There's also my reasoning for other milestones in the appendix (and I've copy-pasted some of them below).
On "efficiently sort very large lists", I think Zach ended up deferring to me on this. But no one has red-teamed my thinking on this. Here's my source:
On "write good Python code", Zach and I (at least initially) disagreed. The criteria specified
Suppose the system is given only:
The concern was that LLMs like Codex are trained on open source code, including examples of quicksort algorithms. So they are given more than just ‘a specification of what counts’ and ‘examples of lists undergoing sorting’. I’m unsure, but I think 'given' in the requirement refers to the goal specification post-training. Otherwise, the wording should be 'suppose the system is trained only on...' Therefore, Codex does count as only being given the stated requirements (or less).
For example, the ‘Starcraft’ milestone uses the wording ‘Beat the best human Starcraft 2 players at least 50% of the time, given a video of the screen’. For this milestone, I think the AI can be trained on more than the video of the screen, it just can’t use more than this while playing.
But could definitely be convinced otherwise on this, I might be making a conceptual misunderstanding. And I haven't given Zach a chance to review my argument above.
The question specifies the AI needs to do "as well as the typical human" - I agree AI is not quite as good as the best human transcriptionists, but think it is better than the typical human.
OpenAI says Whisper “approaches human level robustness and accuracy on English speech recognition.” More detail is given in page 10 of the OpenAI report, which concludes “Whisper’s English ASR performance is not perfect but very close to human-level accuracy”.
However, this is when compared to “professional human transcribers”. Given Whisper sometimes performs better than professional human transcribers, and always performs to a similar level it seems very likely that Whisper regularly does "as well as a typical human”. I think Lorenzo's analysis below seems reasonable too.
I have questionable anecdata! I asked two friends who are fluent in two languages but not skilled in translation, and one who has done a university module in translation, and all three thought they were better than AI at translating web pages (I think all three seem modest in general), although AI does a fine job for most uses.
This article notes the worse performance from AI when interpreting context and cultural nuances and says that “Le Monde explains that “the team (of its English language edition) is composed of eight journalists (…) responsible for selecting articles for translation, editing the translated versions and managing the English home page of the site. The translation is done by international agencies with the help of an artificial intelligence tool. Native English-speaking journalists do the selection and editing of the articles”.”
Not-skilled-in-translation humans are still paid to translate writing between English and Chinese (I know of one organisation that does this and the CEO chooses to use AI to assist but not replace human translators, where the human translators are bilingual students not trained in translation). It does seem possible that this is just business lagging behind though.
I'm convinced by your argument/sources here, and this reddit thread. It does seem open to interpretation if "the best human players" refers to pros (which it did beat >50% of the time), or the literal very best player - but I think this should probably be interpreted as the very best. Given it reached a top 20 score but only beat Serral 2/3 times when heavy restrictions were placed on Serral, I've updated this to false.
I was also surprised by this. Others disagreed with me and we discussed a bit more in this LW comment thread. Here's my reasoning:
I couldn't find this done and think, by now, someone would have submitted a fully ChatGPT-generated high school essay and talked about it publicly if it had gotten high marks. I've seen some evidence of cherry-picking paragraphs leading to a mid/low-level, e.g. this article describes someone who got a passing mark (53) on a university social policy essay.
This high school teacher judged two ChatGPT-generated history essays as “below average, scoring a 9/20 or lower”. This Guardian article says, uncited, that ‘academics have generated responses to exam queries that they say would result in full marks if submitted by an undergraduate’. I think, if this claim were true, there would be more evidence.
Good point! This is an update for me - I wouldn't have guessed ChatGPT could explain chess moves. The question specifies "For any computer game that can be played well by a machine". I just played Scrabble with ChatGPT and it explained its moves well. And I asked ChatGPT to imagine it was watching Agent57 play Montezuma's revenge for 10 seconds and it gave a very clear summary of what it imagined Agent57 was doing and why.
So I'm just about convinced enough that it could do this for all games - and have changed to True.
Thanks Adam :)
I have a rough (i.e. considered for <15 minutes) take: if top labs one year ago had attempted these particular milestones, and had the same policies on disclosing capabilities as they currently seem to, then there's a 40-50% chance they would have achieved 2 of Angry Birds, Atari fifty , Laundry and Go low by now. But I don't put much weight on my prediction, whereas I put a lot more weight on my analysis of what has happened (though this is also somewhat subjective!).
I agree though that checking what has actually happened ends up being more conservative than the original "Ignore the question of whether they would choose to" , which makes the expert forecasts worse than they seem to be here. This is a weakness of this analysis! And of the resolvability of the original survey.
Do you have an estimate of how many of the tasks would have been achieved by now if labs tried a year ago?