Since you're the only one mentioning LLMs I thought I'd ask a couple of things I'd love to understand. Maybe you can point me somewhere to learn more.
Forecasting is supposed to help us prepare for the future. I just learnt that LLMs seem to be increasingly be used for forecasting - turns out that they're surprisingly good at it. And LLMs-for-forecasting are used for AI safety work.
This sounds to me very close to assigning a suspected criminal to guide the team of investigators that are investigating the suspected crime.
Furthermore, I hear that this is admitted to be risky some years in the future. While also admitting that we don't know how the current AI is already so surprisingly good at it. Why isn't it considered to be risky right now?
Even further, I hear that the reason LLMs are used is because there's not enough humans to work on this. But given the circularity of these things, I'll bet that the use of LLMs is also making it difficult for new humans to grow into the job. So this would paint a future of better LLMs forecasting for less humans. Is this so?
In all, this sounds to me like a doom machine that will only fail if forecasting turns out by sheer luck to be useless and/or LLMs turn out by sheer luck to go nowhere.
Or in other words, AI safety work with a good chance of making the problem worse by spawning a double agent.
How much of a post are you comfortable for AI to write?
As a reader, each whiff of LLM usage is a red flag. If the writer used it, it's highly probable that I should use another LLM to summarize it.
And if the final copy used it, surely previous stages used it even more, with all that implies. (I assume it's not the moment to go into that)
As a writer, it's symmetric. Why should I expect readers to spend effort that I didn't? Even if they do, how can I be sure that there wasn't a drift from my original intended meaning?
Reference: Institute of Medicine. 1998. Dietary Reference Intakes for Thiamin, Riboflavin, Niacin, Vitamin B6, Folate, Vitamin B12, Pantothenic Acid, Biotin, and Choline. Washington, DC: The National Academies Press. https://doi.org/10.17226/6015.
I couldn't find any discussion or consensus about this, so I'll ask.
I am surprised to find seemingly well-thought-out articles openly citing LLMs as research. For contrast, no one (who knows how to use Wikipedia) would cite Wikipedia: you are supposed to go to its sources. That's why [citation needed] is a thing: you don't want to build on unsubstantiated BS. So I find it hard to believe that people would cite infamously hallucination/BS-prone LLMs instead, expecting some reasoning built on that to be accepted.
If the author did check for actual sources, they would surely cite them directly instead of citing the LLM, right? E.g., if Claude says X and cites source Y, then either Y is a valid source and then why care for Claude's X; or Y is not valid and then of course why care for Claude's X. In both cases, X is useless.
And if the author couldn't be bothered to check this, why should anyone care for their reasoning?
I find it particularly surprising in a community worried about AI. Are authors really swallowing whole whatever an LLM says, and thus being its conduit, without checking if it's manipulating them?
EDIT: Citaception: Turns out that LLMs also don't know how to use Wikipedia even when in "research" mode: they are happy to quote and build on any claim X in there, even when the citations in the very same Wikipedia refute X.
I agree with the "strong title + unconvincing = social punishment" part. But you seem to only apply it to the "value judgement" in the title, and I disagree with that.
The post being critiqued has a bold, unapologetic title: no "model" or "forecast" or "this could be", just this is "AI 2027", you deal with it. And is published on its own website with high production values. It's borderline arrogant.
In that context, a response article (not a website!) named "A deep critique... of bad timeline models" sounds comparatively level-headed to me.
I wrote much of this in January and February before it became apparent how much of an incredible own-goal we were going to make by undermining our alliances.
Personally, not addressing this came through as USA-absorbed; largely lacking self-awareness and making me itch to find a Chinese counterpoint.
As an European living in Australia and interested in China, right now I look worriedly at what will USA-backed Israel preemptively bomb next while everything in Western news seems to be about how Iran's leader is "vulnerable" but with an "iron grip"[1] and how exactly will he be killed.
So I find it hard to take seriously a post that is all worried about Xi and China vs Taiwan while the USA are just some kind of abstract defender that just "maybe yes, maybe not" will join wars just because somehow it happens to have bases next door to China. And Russia. And Iran. Etc, etc.
The referring to CIA intelligence (just like with Iraq's WMDs[2]) and seeing that the Plan A video starts with a nuclear strike from Russia (never USA, of course) doesn't help.
I guess my question is: how much risk reduction would there be if USA stopped messing with the world?
ADDED 2 days later: USA "spy chief" repeats there's no evidence Iran is building nuclear weapons. Trump disavows her. 12 hours later, there's news about USA bombing Iran. Why should I be worried about Xi again?
You provide a lot of examples of companies and studies already using various flavors of AI, ML - and in many cases things get thin enough that feels like they are using AI-washed databases and simulations. At the risk of getting cynical, the take I end up with is "lots of companies and studies have been using software and statistics for years to develop alternative proteins, and many are happy to embrace AI-washing".
This feels ironic given that you yourself mention AI-washing as a risk.
So I guess my question is, why the AI focus in the post, and what is the future implication that I'm missing?
Reminds me of the part in Douglas Adams' "The Restaurant at the End of the Universe" where a cow-like being is eager to be eaten, describes how she had been overfeeding to fatten herself, and suggests to the Earthlings dishes made of parts of its body. They end up horrified and ordering a salad instead.
I don't expect that Adams wrote it to defend veganism, but he was good at laughing at this kind of absurdity / hypocrisy.
Since you're the only one mentioning LLMs I thought I'd ask a couple of things I'd love to understand. Maybe you can point me somewhere to learn more.
In all, this sounds to me like a doom machine that will only fail if forecasting turns out by sheer luck to be useless and/or LLMs turn out by sheer luck to go nowhere.
Or in other words, AI safety work with a good chance of making the problem worse by spawning a double agent.
What am I getting wrong? Where can I learn more?
As a reader, each whiff of LLM usage is a red flag. If the writer used it, it's highly probable that I should use another LLM to summarize it.
And if the final copy used it, surely previous stages used it even more, with all that implies. (I assume it's not the moment to go into that)
As a writer, it's symmetric. Why should I expect readers to spend effort that I didn't? Even if they do, how can I be sure that there wasn't a drift from my original intended meaning?
On top of the others' comments on B12, it's worth noting that there's a common blanket recommendation for everyone over 50 to supplement B12, no matter their diet, because absorption gets worse with age. So it's strange to insist on lack of B12 and need of medicine as a problem only for vegans.
Reference: Institute of Medicine. 1998. Dietary Reference Intakes for Thiamin, Riboflavin, Niacin, Vitamin B6, Folate, Vitamin B12, Pantothenic Acid, Biotin, and Choline. Washington, DC: The National Academies Press. https://doi.org/10.17226/6015.
If an Anthropic data scientist in a high-profile legal case can be hoodwinked by bad citations, I don't think that it is realistic at all to think that anyone can have a "good sense of what sorts of facts LLMs are unlikely to hallucinate".
And I thought we all have heard about lists of fun things to do full of non-existent restaurants in the way to non-existent towns?
When is it acceptable to cite an LLM?
I couldn't find any discussion or consensus about this, so I'll ask.
I am surprised to find seemingly well-thought-out articles openly citing LLMs as research. For contrast, no one (who knows how to use Wikipedia) would cite Wikipedia: you are supposed to go to its sources. That's why [citation needed] is a thing: you don't want to build on unsubstantiated BS. So I find it hard to believe that people would cite infamously hallucination/BS-prone LLMs instead, expecting some reasoning built on that to be accepted.
If the author did check for actual sources, they would surely cite them directly instead of citing the LLM, right? E.g., if Claude says X and cites source Y, then either Y is a valid source and then why care for Claude's X; or Y is not valid and then of course why care for Claude's X. In both cases, X is useless.
And if the author couldn't be bothered to check this, why should anyone care for their reasoning?
I find it particularly surprising in a community worried about AI. Are authors really swallowing whole whatever an LLM says, and thus being its conduit, without checking if it's manipulating them?
EDIT: Citaception: Turns out that LLMs also don't know how to use Wikipedia even when in "research" mode: they are happy to quote and build on any claim X in there, even when the citations in the very same Wikipedia refute X.
I agree with the "strong title + unconvincing = social punishment" part. But you seem to only apply it to the "value judgement" in the title, and I disagree with that.
The post being critiqued has a bold, unapologetic title: no "model" or "forecast" or "this could be", just this is "AI 2027", you deal with it. And is published on its own website with high production values. It's borderline arrogant.
In that context, a response article (not a website!) named "A deep critique... of bad timeline models" sounds comparatively level-headed to me.
Personally, not addressing this came through as USA-absorbed; largely lacking self-awareness and making me itch to find a Chinese counterpoint.
As an European living in Australia and interested in China, right now I look worriedly at what will USA-backed Israel preemptively bomb next while everything in Western news seems to be about how Iran's leader is "vulnerable" but with an "iron grip"[1] and how exactly will he be killed.
So I find it hard to take seriously a post that is all worried about Xi and China vs Taiwan while the USA are just some kind of abstract defender that just "maybe yes, maybe not" will join wars just because somehow it happens to have bases next door to China. And Russia. And Iran. Etc, etc.
The referring to CIA intelligence (just like with Iraq's WMDs[2]) and seeing that the Plan A video starts with a nuclear strike from Russia (never USA, of course) doesn't help.
I guess my question is: how much risk reduction would there be if USA stopped messing with the world?
The enemy is weak and strong at the same time - just like Umberto Eco warned about fascist movements. Who are the baddies again?
ADDED 2 days later: USA "spy chief" repeats there's no evidence Iran is building nuclear weapons. Trump disavows her. 12 hours later, there's news about USA bombing Iran. Why should I be worried about Xi again?
Thank you for this interesting post.
You provide a lot of examples of companies and studies already using various flavors of AI, ML - and in many cases things get thin enough that feels like they are using AI-washed databases and simulations. At the risk of getting cynical, the take I end up with is "lots of companies and studies have been using software and statistics for years to develop alternative proteins, and many are happy to embrace AI-washing".
This feels ironic given that you yourself mention AI-washing as a risk.
So I guess my question is, why the AI focus in the post, and what is the future implication that I'm missing?
I was surprised by this statement. Isn't it easy to say the same for coal power plants and mines?
Reminds me of the part in Douglas Adams' "The Restaurant at the End of the Universe" where a cow-like being is eager to be eaten, describes how she had been overfeeding to fatten herself, and suggests to the Earthlings dishes made of parts of its body. They end up horrified and ordering a salad instead.
I don't expect that Adams wrote it to defend veganism, but he was good at laughing at this kind of absurdity / hypocrisy.