Peer-review is important but it isn't what alone decides if a study is good or not. Replications and further studies are what count. A paper without a peer review can still be good, and a paper with one can be bad.
Peer-reviews are often taken onto pedestal since their lack demonstrates the anti-academia tendencies in the AI safety field. Lack of peer review indicates that the study is not presented in the academic community in the standard way, which means it will not get the scrutiny it deserves for it to be validated by further research. It's a systematic issue, not an issue with any single study.
What METR should do is to get other people, unaffiliated with them, to continue this angle or research.
There is a limit to how much a for-profit company can care about safety. It is of existential importance to make profit, so you have to sell your product. It's just how it is.
OpenAI and Anthropic are technically public benefit companies that have other objectives than just profit, but to my understanding they are formulated in a way that makes them not relevant currently, so profit is more important.
I don't say that there aren't people in these companies that care about safety, or that safety is completely ignored when developing products, just that there is a limit to how far the companies will consider safety.
Anthropic and OpenAI have been quite successful at promoting the view that their models are very capable or even dangerous. By exaggerating the risks of these models they create a reputation as morally conscious, safety-oriented companies that nevertheless have the best models that everyone should be aware of. This gets the sympathies of investors and AI safety people alike.
But of course they must sell their models to get their money back.
I often hear the suggestion that people should short stock when they don't believe in a company. I don't think that it is a very good piece of advice.
Shorting is notoriously difficult and carries the possibility of unlimited loss. Even if you believe that a stock will crash, small errors such as timing the crash one year too early or using a miscalculated stop order to stop shorting too early can lead to massive losses. Determining the actual risk involved is not very straightforward. Shorting is often accompanied by risk-managing tactics such as hedging that also require careful considerations.
Due to all this, even if you believe a stock is going to crash, shorting is usually a bad idea. Basically, the skillset required to short properly is different from the skills used to predict whether or not AI will succeed commercially or not. An expert in AI can do the latter prediction, but the former requires you to also be specialized in complex investments. If you are "only" an AI expert, bets offer a much more managed risk profile.
In the context of my post, the term "narrative argument" was really clearly used to refer to fictional stories embedded in a text as part of its argument, such as AI2027's short story or the parables in Y&S's book. My text simply does not have that.
By completely misunderstanding even basic concepts such as "narrative argument", your comment was just a snarky jab and had little value. It didn't comment the actual content of my text, instead, and largely unjustified by any real analysis, it tried to "get back at me" by blaming me for things I blame others for, like non-evidence based argumentation and narrative arguments.
Since I find it difficult to believe that you really don't understand what a narrative argument is, I reasonably count a high probability for you simply acting in bad faith.
I don't know if "snitching" you is allowed or not, and I apologize for the moderators if it's not, but I think using examples of bad behavior is relevant in discussions of what is allowed on the forum.
I feel that there could be greater transparency in how these moderation decisions are done. In my opinion, although Yarrow has been quite confrontational at times, they haven't really done anything that would necessitate such a punishment. While the forum is allowed to moderate its users in any way it likes, one shouldn't be rate-limited just because they have disagreements, especially if their opinions go against the mainstream opinion on the forum. This puts the neutrality of the forum moderation at question.
Personally, I have pretty bad experiences writing on this forum. I get highly-upvoted comments from people who seemingly haven't even read what I wrote and instead throw in bad-faith arguments, which I find borderline insulting and certainly unwelcoming. See, for example, this comment that was given as a reply to my text, where I pointed out that many sources use narrative arguments (stories, parables, etc.) instead of technical evidence. The commenter accused me of writing a narrative argument... which is simply not true, as my text was not written in a form of a story. Either they didn't read at all what I wrote or intentionally misunderstood it. And that is the most upvoted comment!
In general, the quality of discussion on this forum is very low. I feel it is pretty harsh to say this, but it is my experience and it strongly discourages me from participating at all. In this atmosphere, rate-limiting Yarrow appears not as an impartial decision, since many other, much more problematic comments appear in the forum all the time.
I don't really see why anyone would use AI other than maybe translate phrases if one does not know English (or the language they are writing) well.
Language models add unnecessary fluff to the text that takes space away from actual content. I'd ask the following question: is the text written by the AI longer than the prompt? If the answer is yes, then please do not use that AI text. All extra text written by the model is fluff and meaningless.
Language models are also really bad at writing. They tend to overuse stylistic devices to the point that the text is heavy and difficult to read. They split the text into unnatural chapters, tend to add more subheadings than necessary, overuse tables and lists, and in general write horribly. Some people claim that AI helps people who are bad at writing to be able to take part in discussions, and that argument really makes no sense since AI cannot write either and I'd rather read bad English than fluff. The only exception to this is using AI for translation, since that is obviously a real blocker.
Lastly, there is the question of effort. To put an effort into writing is a proof that someone cares about this enough to write it down, so I should also care about reading it. If I recognize that a text is AI written, my motivation to go through it drops immediately. In a world of constant spam and sensory – and literary – overload, my time is valuable, and effort is the currency that gets me to read your text.
Your argument is very similar to creationist and other pseudoscientific/conspiracy theory-style arguments.
A creationist might argue that the existence of life, humanity, and other complex phenomena is "evidence" for intelligent design. If we allow this to count as "limited" evidence (or whatever term we choose to use), it is possible to follow through a Pascal's wager-style argument and posit that this "evidence", even if it has high uncertainty, is enough to merit an action.
It is always possible to come up with "evidence" for any claim. In evidence-based decision making, we must set a bar for evidence. Otherwise, the word "evidence" would lose it's meaning, and we'd be wasting our resources considering every piece of knowledge there exists as "evidence".
You could have moderate-quality studies that withstand scutiny
If the studies withstand scrutiny, then they are high-quality studies. Of course, it is possible that the study has multiple conclusions, and some of them are undermined by scrutiny and some are not, or that there are errors that do not undermine the conclusions. These studies can of course be used as evidence. I used "high-quality" as the opposite of "low-quality", and splitting hair about "moderate-quality" is uninteresting.
you could have preliminary studies which are suggestive but which haven't been around long enough for scrutiny to percolate up
This is a good basis when, e.g., funding new research, as confirming and replicating recent studies is an important part of science. In this case, it doesn't matter that much if the study's conclusions end up being true or false, as confirming either way is valuable. Researching interesting things is good, and even bad studies are evidence that the topic is interesting. But they are not evidence that should be used for other kind of decision-making.
The fact that you are using the word "evidence" in this way is causing you to misinterpret the quoted statement.
You are again splitting a hair about the meanings of words. The important thing is that they are advocating for making decisions without sufficient evidence, which is something I oppose. Their report is long and contains many AI risks, some of which (like deepfakes) have high-quality studies behind them, while others (like X-risks) do not. As a whole, the report "has some evidence" that there are risks associated with AI. So they talk about "limited evidence". What is important is that they imply this "limited evidence" is not sufficient for making decisions.
But "limited evidence" isn't the same as near-zero evidence
Splitting a hair. You can call your evidence limited evidence if you want. It won't get you a free pass that your argument should be considered. If it has too much uncertainty or doesn't withstand scrutiny, it shouldn't be taken in as evidence. Otherwise we end up in the creationist situation.
If you believe that evidence that does not withstand scrutiny (that is, evidence that does not meet basic quality standards, contains major methodological errors, is statistically insignificant, is based on fallacious reasoning, or any other reason why the evidence is scrutinized) is evidence that we should use, then you are advocating for pseudoscience. The expected value of benefits based on such evidence is near zero.
I'm sorry if criticizing pseudoscience is frustrating, but that kind of thinking has no place in rational decision-making.
Your summary of the quoted text is inaccurate. You claim that this is an arguement that evidence is not something that in inherently required, but the quote says no such thing. Instead, it references "a large body of scientific evidence" and "stronger evidence" vs "limited evidence". This quote essential makes the same arguement I do above. How can we square the differences in these interpretations?
The quoted text implies that the evidence would not be sufficient under normal circumstances, hence the "evidence dilemma". If the amount of evidence was sufficient, there would be no question about what is the correct action. While the text washes its hands from making the actual decision to rely on insufficient evidence, it clearly considers this as a serious possibility, which is not something that I believe anyone should advocate.
You are splitting hairs about the difference between "no evidence" and "limited evidence". The report considers a multitude of different AI risks, some of which have more evidence and some of which have less. What is important is that they bring up the idea that policy should be made without proper evidence.
The "scientific" phrasing frustrates me because I feel like it is often used to suggest high rigor without actually demonstrating that such rigor actually applies to a give situation, and because I feel like it is used to exclude certain categories of evidence when those categories are relevant, even if they are less strong compared to other kinds of evidence. I think we should weigh all relevant evidence, not exclude cetain pieces because they aren't scientific enough.
Again, you are attacking me because of the word "scientific" instead of attacking my arguments. As I have many, many times said, studies should be weighted based on their content and the scrutiny it receives. To oppose the word "science" just because of the word itself is silly. Your idea that works are arbitrarily sorted to "scientific" and "non-scientific" based on "style points" instead of assessing their merits is just wrong and a straw-man argument.
I don't think my argument leads to this conclusion. I'm just saying that AI risk has some evidence behind it, even if it isn't the most rigorous evidence! That's why I'm being such a stickler about this! If it were true that AI risk has actually zero evidence then of course I wouldn't buy it! But I don't think there actually is zero evidence even if AI risk advocates sometimes overestimate the strength of the evidence.
Where have I ever claimed that there is no evidence worth considering? In the start of my post, I write:
What unites many of these statements is the thorough lack of any evidence.
There are some studies that are rigorously conducted that provide some meager evidence. Not really enough to justify any EA intervention. But instead of referring to these studies, people use stuff like narrative arguments and ad-hoc models, which have approximately zero evidential value. That is the point of my post.
What about these statements makes you think that I don't believe uncertainty affects decision making? It seems like I say that it does affect decision making in my comment.
If you believe this, I don't understand where you disagree with me, other than you weird opposition to the word "scientific".
Peer-review is important but it isn't what alone decides if a study is good or not. Replications and further studies are what count. A paper without a peer review can still be good, and a paper with one can be bad.
Peer-reviews are often taken onto pedestal since their lack demonstrates the anti-academia tendencies in the AI safety field. Lack of peer review indicates that the study is not presented in the academic community in the standard way, which means it will not get the scrutiny it deserves for it to be validated by further research. It's a systematic issue, not an issue with any single study.
What METR should do is to get other people, unaffiliated with them, to continue this angle or research.
There is a limit to how much a for-profit company can care about safety. It is of existential importance to make profit, so you have to sell your product. It's just how it is.
OpenAI and Anthropic are technically public benefit companies that have other objectives than just profit, but to my understanding they are formulated in a way that makes them not relevant currently, so profit is more important.
I don't say that there aren't people in these companies that care about safety, or that safety is completely ignored when developing products, just that there is a limit to how far the companies will consider safety.
That's marketing.
Anthropic and OpenAI have been quite successful at promoting the view that their models are very capable or even dangerous. By exaggerating the risks of these models they create a reputation as morally conscious, safety-oriented companies that nevertheless have the best models that everyone should be aware of. This gets the sympathies of investors and AI safety people alike.
But of course they must sell their models to get their money back.
I often hear the suggestion that people should short stock when they don't believe in a company. I don't think that it is a very good piece of advice.
Shorting is notoriously difficult and carries the possibility of unlimited loss. Even if you believe that a stock will crash, small errors such as timing the crash one year too early or using a miscalculated stop order to stop shorting too early can lead to massive losses. Determining the actual risk involved is not very straightforward. Shorting is often accompanied by risk-managing tactics such as hedging that also require careful considerations.
Due to all this, even if you believe a stock is going to crash, shorting is usually a bad idea. Basically, the skillset required to short properly is different from the skills used to predict whether or not AI will succeed commercially or not. An expert in AI can do the latter prediction, but the former requires you to also be specialized in complex investments. If you are "only" an AI expert, bets offer a much more managed risk profile.
In the context of my post, the term "narrative argument" was really clearly used to refer to fictional stories embedded in a text as part of its argument, such as AI2027's short story or the parables in Y&S's book. My text simply does not have that.
By completely misunderstanding even basic concepts such as "narrative argument", your comment was just a snarky jab and had little value. It didn't comment the actual content of my text, instead, and largely unjustified by any real analysis, it tried to "get back at me" by blaming me for things I blame others for, like non-evidence based argumentation and narrative arguments.
Since I find it difficult to believe that you really don't understand what a narrative argument is, I reasonably count a high probability for you simply acting in bad faith.
I don't know if "snitching" you is allowed or not, and I apologize for the moderators if it's not, but I think using examples of bad behavior is relevant in discussions of what is allowed on the forum.
I feel that there could be greater transparency in how these moderation decisions are done. In my opinion, although Yarrow has been quite confrontational at times, they haven't really done anything that would necessitate such a punishment. While the forum is allowed to moderate its users in any way it likes, one shouldn't be rate-limited just because they have disagreements, especially if their opinions go against the mainstream opinion on the forum. This puts the neutrality of the forum moderation at question.
Personally, I have pretty bad experiences writing on this forum. I get highly-upvoted comments from people who seemingly haven't even read what I wrote and instead throw in bad-faith arguments, which I find borderline insulting and certainly unwelcoming. See, for example, this comment that was given as a reply to my text, where I pointed out that many sources use narrative arguments (stories, parables, etc.) instead of technical evidence. The commenter accused me of writing a narrative argument... which is simply not true, as my text was not written in a form of a story. Either they didn't read at all what I wrote or intentionally misunderstood it. And that is the most upvoted comment!
In general, the quality of discussion on this forum is very low. I feel it is pretty harsh to say this, but it is my experience and it strongly discourages me from participating at all. In this atmosphere, rate-limiting Yarrow appears not as an impartial decision, since many other, much more problematic comments appear in the forum all the time.
I don't really see why anyone would use AI other than maybe translate phrases if one does not know English (or the language they are writing) well.
Language models add unnecessary fluff to the text that takes space away from actual content. I'd ask the following question: is the text written by the AI longer than the prompt? If the answer is yes, then please do not use that AI text. All extra text written by the model is fluff and meaningless.
Language models are also really bad at writing. They tend to overuse stylistic devices to the point that the text is heavy and difficult to read. They split the text into unnatural chapters, tend to add more subheadings than necessary, overuse tables and lists, and in general write horribly. Some people claim that AI helps people who are bad at writing to be able to take part in discussions, and that argument really makes no sense since AI cannot write either and I'd rather read bad English than fluff. The only exception to this is using AI for translation, since that is obviously a real blocker.
Lastly, there is the question of effort. To put an effort into writing is a proof that someone cares about this enough to write it down, so I should also care about reading it. If I recognize that a text is AI written, my motivation to go through it drops immediately. In a world of constant spam and sensory – and literary – overload, my time is valuable, and effort is the currency that gets me to read your text.
Your argument is very similar to creationist and other pseudoscientific/conspiracy theory-style arguments.
A creationist might argue that the existence of life, humanity, and other complex phenomena is "evidence" for intelligent design. If we allow this to count as "limited" evidence (or whatever term we choose to use), it is possible to follow through a Pascal's wager-style argument and posit that this "evidence", even if it has high uncertainty, is enough to merit an action.
It is always possible to come up with "evidence" for any claim. In evidence-based decision making, we must set a bar for evidence. Otherwise, the word "evidence" would lose it's meaning, and we'd be wasting our resources considering every piece of knowledge there exists as "evidence".
If the studies withstand scrutiny, then they are high-quality studies. Of course, it is possible that the study has multiple conclusions, and some of them are undermined by scrutiny and some are not, or that there are errors that do not undermine the conclusions. These studies can of course be used as evidence. I used "high-quality" as the opposite of "low-quality", and splitting hair about "moderate-quality" is uninteresting.
This is a good basis when, e.g., funding new research, as confirming and replicating recent studies is an important part of science. In this case, it doesn't matter that much if the study's conclusions end up being true or false, as confirming either way is valuable. Researching interesting things is good, and even bad studies are evidence that the topic is interesting. But they are not evidence that should be used for other kind of decision-making.
You are again splitting a hair about the meanings of words. The important thing is that they are advocating for making decisions without sufficient evidence, which is something I oppose. Their report is long and contains many AI risks, some of which (like deepfakes) have high-quality studies behind them, while others (like X-risks) do not. As a whole, the report "has some evidence" that there are risks associated with AI. So they talk about "limited evidence". What is important is that they imply this "limited evidence" is not sufficient for making decisions.
Splitting a hair. You can call your evidence limited evidence if you want. It won't get you a free pass that your argument should be considered. If it has too much uncertainty or doesn't withstand scrutiny, it shouldn't be taken in as evidence. Otherwise we end up in the creationist situation.
If you believe that evidence that does not withstand scrutiny (that is, evidence that does not meet basic quality standards, contains major methodological errors, is statistically insignificant, is based on fallacious reasoning, or any other reason why the evidence is scrutinized) is evidence that we should use, then you are advocating for pseudoscience. The expected value of benefits based on such evidence is near zero.
I'm sorry if criticizing pseudoscience is frustrating, but that kind of thinking has no place in rational decision-making.
The quoted text implies that the evidence would not be sufficient under normal circumstances, hence the "evidence dilemma". If the amount of evidence was sufficient, there would be no question about what is the correct action. While the text washes its hands from making the actual decision to rely on insufficient evidence, it clearly considers this as a serious possibility, which is not something that I believe anyone should advocate.
You are splitting hairs about the difference between "no evidence" and "limited evidence". The report considers a multitude of different AI risks, some of which have more evidence and some of which have less. What is important is that they bring up the idea that policy should be made without proper evidence.
Again, you are attacking me because of the word "scientific" instead of attacking my arguments. As I have many, many times said, studies should be weighted based on their content and the scrutiny it receives. To oppose the word "science" just because of the word itself is silly. Your idea that works are arbitrarily sorted to "scientific" and "non-scientific" based on "style points" instead of assessing their merits is just wrong and a straw-man argument.
Where have I ever claimed that there is no evidence worth considering? In the start of my post, I write:
There are some studies that are rigorously conducted that provide some meager evidence. Not really enough to justify any EA intervention. But instead of referring to these studies, people use stuff like narrative arguments and ad-hoc models, which have approximately zero evidential value. That is the point of my post.
If you believe this, I don't understand where you disagree with me, other than you weird opposition to the word "scientific".