Great catch, this seems important and I didn't realize it. The ForecastBench paper has some comparisons between humans and LLMs which probably don't have access to human forecasts. In the tables below, they're the rows where "information provided" is "news." These models don't have open access to the internet; they can only pull summaries of news articles through a custom API, so unless those news articles are citing prediction markets, the LLM isn't getting information about prediction market forecasts.
The Brier score difference between LLM forecasters with and without access to human crowd forecasts is roughly the same as the Brier score difference between the superforecaster median and the public median. (Though I'm not sure how to interpret that, a Brier score is a weird metric.)
Agreed this seems like an important shortcoming of existing research. I'd love to see future work that measures the accuracy of LLM forecasters with access to the internet but no access to prediction markets or human crowd forecasts. This could be implemented by instructing the LLM not to look at crowd forecasts when surfing the internet, then asking another LLM to verify that the instruction was followed, and resampling if not.
We’re looking for someone to write the grant recommendations, memos, and explainers that help us move money to AI safety. You'll basically work to understand what we're funding and why, and write it up for donors. Some donors are just learning about AI risk, but many are extremely high context on AI safety and will notice if you get a technical detail wrong, so it's important to get someone who can write excellent, high-fidelity content. We want someone with strong AI safety knowledge who writes well and fast, including same-day turnarounds.
It’s an hourly contract, initially for three months, 20+ hrs/week preferred, with potential to extend or convert to full-time. I think it’s a highly directly impactful role and a good learning opportunity for people with good prior knowledge of AI safety.Apply here by EOD Monday March 16.
Nice, I think this is a great perspective. One comment on "becoming known": I used to think trying to become well known was mostly zero-sum—you're just competing against other candidates for a fixed pool of jobs and marketing yourself to beat them. That's definitely part of the story, but it misses a key positive sum benefit of becoming better known.
Employers have a pessimistic prior on job applicants and struggle to tell whether someone is truly excellent, so making your skills legible (e.g. by writing in public, getting credentials building relationships with domain experts who can vouch for you) allows employers to hire you when they otherwise wouldn't have the confidence to hire anyone. From the perspective of an employer, one candidate known to be very good is often better than a bunch of candidates who might have better skills in expectation but whose skills are extremely difficult to verify, because you can actually hire the first person whereas you might not be able to make a hire from the second pool.
The topline comparison between LLMs and superforecasters seems a bit unfair. You compare a single LLM's forecast against the median from a crowd of superforecasters. But we know the median from a crowd is typically more accurate than any particular member of the crowd. Therefore I think it'd be more fair to compare a single LLM to a single superforecaster, or a crowd of LLMs against a crowd of superforecasters. Do we know whether the best LLM is better than the best individual forecaster in your sample, or how the median LLM compares to the median forecaster?
(Nitpick aside, this is very interesting research, thanks for doing it.)
Agreed it's super useful. I think it's probably significantly underestimating the size of the field though, as I think there are dozens of orgs doing at least some work on AI safety not listed here.
Agreed with the other answers on the reasons why there's no GiveWell for AI safety. But in case it's helpful, I should say that Longview Philanthropy offers advice to donors looking to give >$100K per year to AI safety. Our methodology is a bit different from GiveWell’s, but we do use cost-effectiveness estimates. We investigate funding opportunities across the AI landscape from technical research to field-building to policy in the US, EU, and around the world, trying to find the most impactful opportunities for the marginal donor. We also do active grantmaking, such as our calls for proposals on hardware-enabled mechanisms and digital sentience. More details here. Feel free to reach out to [email protected] or [email protected] if you'd like to learn more.
Now, Anthropic, OpenAI, Google DeepMind, and xAI say their most powerful models might have dangerous biology capabilities and thus could substantially boost extremists—but not states—in creating bioweapons.
I think the "not states" part of this is incorrect in the case of OpenAI, whose Deep Research system card said: "Our evaluations found that deep research can help experts with the operational planning of reproducing a known biological threat, which meets our medium risk threshold."
One other potential suggestion: Organizers should consider focusing on their own career development rather than field-building if their timelines are shortening and they think they can have a direct impact sooner than they can have an impact through field-building. Personally I regret much of the time I spent starting an AI safety club in college because it traded off against building skills and experience in direct work. I think my impact through direct work has been significantly greater than my impact through field-building, and I should've spent more time on direct work in college.
What about corporations or nation states during times of conflict - do you think it's accurate to model them as roughly as ruthless in pursuit of their own goals as future AI agents?
They don't have the same psychological makeup as individual people, they have a strong tradition and culture of maximizing self-interest, and they face strong incentives and selection pressures to maximize fitness (i.e. for companies to profit, for nation states to ensure their own survival) lest they be outcompeted by more ruthless competitors. On average, while I'd expect that these entities tend to show some care for goals besides self-interest maximization, I think the most reliable predictor of their behavior is the maximization of their self-interest.
If they're roughly as ruthless as future AI agents, and we've developed institutions that somewhat robustly align their ambitions with pro-social action, then we should have some optimism that we can find similarly productive systems for working with misaligned AIs.
Great catch, this seems important and I didn't realize it. The ForecastBench paper has some comparisons between humans and LLMs which probably don't have access to human forecasts. In the tables below, they're the rows where "information provided" is "news." These models don't have open access to the internet; they can only pull summaries of news articles through a custom API, so unless those news articles are citing prediction markets, the LLM isn't getting information about prediction market forecasts.
The Brier score difference between LLM forecasters with and without access to human crowd forecasts is roughly the same as the Brier score difference between the superforecaster median and the public median. (Though I'm not sure how to interpret that, a Brier score is a weird metric.)
Agreed this seems like an important shortcoming of existing research. I'd love to see future work that measures the accuracy of LLM forecasters with access to the internet but no access to prediction markets or human crowd forecasts. This could be implemented by instructing the LLM not to look at crowd forecasts when surfing the internet, then asking another LLM to verify that the instruction was followed, and resampling if not.
Longview is hiring an AI Safety Content Specialist (contractor, remote, $40–75/hr)
We’re looking for someone to write the grant recommendations, memos, and explainers that help us move money to AI safety. You'll basically work to understand what we're funding and why, and write it up for donors. Some donors are just learning about AI risk, but many are extremely high context on AI safety and will notice if you get a technical detail wrong, so it's important to get someone who can write excellent, high-fidelity content. We want someone with strong AI safety knowledge who writes well and fast, including same-day turnarounds.
It’s an hourly contract, initially for three months, 20+ hrs/week preferred, with potential to extend or convert to full-time. I think it’s a highly directly impactful role and a good learning opportunity for people with good prior knowledge of AI safety. Apply here by EOD Monday March 16.
Nice, I think this is a great perspective. One comment on "becoming known": I used to think trying to become well known was mostly zero-sum—you're just competing against other candidates for a fixed pool of jobs and marketing yourself to beat them. That's definitely part of the story, but it misses a key positive sum benefit of becoming better known.
Employers have a pessimistic prior on job applicants and struggle to tell whether someone is truly excellent, so making your skills legible (e.g. by writing in public, getting credentials building relationships with domain experts who can vouch for you) allows employers to hire you when they otherwise wouldn't have the confidence to hire anyone. From the perspective of an employer, one candidate known to be very good is often better than a bunch of candidates who might have better skills in expectation but whose skills are extremely difficult to verify, because you can actually hire the first person whereas you might not be able to make a hire from the second pool.
The topline comparison between LLMs and superforecasters seems a bit unfair. You compare a single LLM's forecast against the median from a crowd of superforecasters. But we know the median from a crowd is typically more accurate than any particular member of the crowd. Therefore I think it'd be more fair to compare a single LLM to a single superforecaster, or a crowd of LLMs against a crowd of superforecasters. Do we know whether the best LLM is better than the best individual forecaster in your sample, or how the median LLM compares to the median forecaster?
(Nitpick aside, this is very interesting research, thanks for doing it.)
Agreed it's super useful. I think it's probably significantly underestimating the size of the field though, as I think there are dozens of orgs doing at least some work on AI safety not listed here.
Agreed with the other answers on the reasons why there's no GiveWell for AI safety. But in case it's helpful, I should say that Longview Philanthropy offers advice to donors looking to give >$100K per year to AI safety. Our methodology is a bit different from GiveWell’s, but we do use cost-effectiveness estimates. We investigate funding opportunities across the AI landscape from technical research to field-building to policy in the US, EU, and around the world, trying to find the most impactful opportunities for the marginal donor. We also do active grantmaking, such as our calls for proposals on hardware-enabled mechanisms and digital sentience. More details here. Feel free to reach out to [email protected] or [email protected] if you'd like to learn more.
That’s the new PF. The old (December 2023) version defined a medium risk threshold which Deep Research surpassed.
https://cdn.openai.com/openai-preparedness-framework-beta.pdf
I think the "not states" part of this is incorrect in the case of OpenAI, whose Deep Research system card said: "Our evaluations found that deep research can help experts with the operational planning of reproducing a known biological threat, which meets our medium risk threshold."
One other potential suggestion: Organizers should consider focusing on their own career development rather than field-building if their timelines are shortening and they think they can have a direct impact sooner than they can have an impact through field-building. Personally I regret much of the time I spent starting an AI safety club in college because it traded off against building skills and experience in direct work. I think my impact through direct work has been significantly greater than my impact through field-building, and I should've spent more time on direct work in college.
What about corporations or nation states during times of conflict - do you think it's accurate to model them as roughly as ruthless in pursuit of their own goals as future AI agents?
They don't have the same psychological makeup as individual people, they have a strong tradition and culture of maximizing self-interest, and they face strong incentives and selection pressures to maximize fitness (i.e. for companies to profit, for nation states to ensure their own survival) lest they be outcompeted by more ruthless competitors. On average, while I'd expect that these entities tend to show some care for goals besides self-interest maximization, I think the most reliable predictor of their behavior is the maximization of their self-interest.
If they're roughly as ruthless as future AI agents, and we've developed institutions that somewhat robustly align their ambitions with pro-social action, then we should have some optimism that we can find similarly productive systems for working with misaligned AIs.