Summary
This post gives my personal take on “AI for epistemics” and how important it might be to work on.
Some background context:
So: How can we affect AI to contribute to better epistemic processes? When looking at concrete projects, here, I find it helpful to distinguish between two different categories of work:
Working to increase AIs’ epistemic capabilities, and in particular, differentially advancing them compared to other AI capabilities. Here, I also include technical work to measure AIs’ epistemic capabilities.
- Efforts to enable the diffusion and appropriate trust of AI-discovered information. This is focused on social dynamics that could cause AI-produced information to be insufficiently or excessively trusted. It’s also focused on AIs’ role in communicating information (as opposed to just producing it). Examples of interventions, here, include “create an independent organization that evaluates popular AIs’ truthfulness”, or “work for countries to adopt good (and avoid bad) legislation of AI communication”.
I’d be very excited about thoughtful and competent efforts in this second category. However, I talk significantly more about efforts in the first category, in this post. This is just an artifact of how this post came to be, historically — it’s not because I think work on the second category of projects is less important.
For the first category of projects: Technical projects to differentially advance epistemic capabilities seem somewhat more “shovel-ready”. Here, I’m especially excited about projects that differentially boost AI epistemic capabilities in a manner that’s some combination of durable and/or especially good at demonstrating those capabilities to key actors.
Durable means that projects should (i) take the bitter lesson into account by working on problems that won’t be solved-by-default when more compute is available, and (ii) work on problems that industry isn’t already incentivized to put huge efforts into (such as “making AIs into generally better agents”). (More on these criteria here.)
Two example projects that I think fulfill these criteria (I discuss a lot more projects here):
- Experiments on what sort of arguments and decompositions make it easier for humans to reach the truth in hard-to-verify areas. (Strongly related to scalable oversight.)
- Using AI to generate large quantities of forecasting data, such as by automatically generating and resolving questions.
Separately, I think there’s value in demonstrating the potential of AI epistemic advice to key actors — especially frontier AI companies and governments. When transformative AI (TAI) is first developed, it seems likely that these actors will (i) have a big advantage in their ability to accelerate AI-for-epistemics via their access to frontier models and algorithms, and (ii) that I especially care about their decisions being well-informed. Thus, I’d like these actors to be impressed by the potential of AI-for-epistemics as soon as possible, so that they start investing and preparing appropriately.
If you, above, wondered why I group “measuring epistemic capabilities” into the same category of project as “differentially advancing AI capabilities”, this is now easier to explain. I think good benchmarks could be both a relatively durable intervention for increasing capabilities, via inspiring work to beat the benchmark for a long time, and that they’re a good way of demonstrating capabilities.
Structure of the post
In the rest of this post, I:
Previous work
Here is an incomplete list of previous work on this topic:
Why work on AI for epistemics?
Summary
I think there’s very solid grounds to believe that AI’s influence on epistemics is important. Having good epistemics is super valuable, and human-level AI would clearly have a huge impact on our epistemic landscape. (See here for more on importance.)
I also think there are decent plausibility arguments for why epistemics may be important: Today, we are substantially less epistemically capable than our technology allows for, due to various political and social dynamics which don’t all seem inevitable. And I think there are plausible ways in which poor epistemics can be self-reinforcing (because it makes it harder to clearly see what’s the direction towards better epistemics). And vice-versa that good epistemics can be self-reinforcing. (See here for more on path-dependence.)
That’s not very concrete though. To be more specific, I will go through some more specific goals that I think are both important and plausible path-dependent:
- Good norms & practices for AI-as-knowledge-producers. Such as transparency of how AI-based science/investigations work, minimal censorship of AI-produced research results, and maximizing the number of actors who can trust some technically sophisticated institution to verify AI methods’ trustworthiness. (E.g. by having many such institutions with different political affiliations.)
- Good norms & practices for AI-as-communicatiors. Such as transparency of how AIs decide what to communicate, independent evaluators who measure AIs’ truthfulness, and laws that limit the degree to which AIs can present contradictory arguments to different people or be paid-off to present biased views.
- Differentially high epistemic capabilities. Such as high alignable capabilities (compared to underlying capabilities), relative strength at persuading people of true beliefs compared to false beliefs, and relative strength at understanding & predicting the world compared to building new technologies.
Let’s go through all of this in more detail.
Importance
I think there’s very solid grounds to believe that AI’s influence on epistemics is important.
- AI’s influence on human epistemic abilities will eventually be huge. Briefly:
- AI will eventually automate epistemic labor. This includes both knowledge production work and communication work. (The latter which includes both good and bad persuasion of humans.)
- AIs’ epistemic work won’t just replace humans’ 1-for-1. AI comes with special capabilities that will change the epistemic ecosystem:
- Cheaper & easier to delegate epistemic labor.
- It could be cheaper to delegate epistemic labor to AIs (than to humans) because you can copy software for free.
- If we develop methods to train reliably truth-seeking AIs, it will be easier to delegate epistemic labor to AIs (than to humans), because you would have to worry less about being deceived.
- More delegation could lead to more equitable distribution of epistemic capabilities, but also to reduced incentives and selection for humans to have reasonable beliefs and epistemic practices (because AIs make all decisions that matter for your power).
- Better epistemic science.
- You can more easily control what information AIs have and have not seen and thereby run reproducible experiments on what epistemic strategies work best.
- Epistemic capabilities during and after the development of TAI are very valuable. Briefly:
Most AI takeover risk comes from “unforced errors”. A vast majority of powerful people don’t want AI to take over, but I think that many underestimate the risk. If I thought that people were going to have reasonable, well-calibrated beliefs about AI takeover risk, my subjective probability of AI takeover would more than halve.
Most extinction risk comes from “unforced errors”. Just as above: A vast majority of powerful people don’t want extinction, and (I strongly suspect) would be capable of preventing the exceptions from being able to cause extinction.
- Strong epistemic capabilities seem great for moral deliberation. For example via: Helping you better imagine the realistic consequences of various moral principles; by letting you forecast what sort of deliberation procedures will go off-the-rails; and by teaching you about the underlying empirical reasons for moral disagreement (so you can choose which drivers of moral intuition you trust more).
Path-dependence
While less solid than the arguments for importance, I think there are decent plausibility arguments for why AI’s role in societal epistemics may be importantly path-dependent.
- Comparing with the present. Today, I think that our epistemics are significantly worse than they “could have been”. We aren’t just constrained by high-quality labor or evidence — there are also significant political and self-serving forces/incentives that actively distort people’s beliefs. These won’t automatically go away in the future.
Feedback loops. People often choose to learn the truth when the choice is presented sufficiently clearly and unambiguously to them. But with poor enough epistemic starting abilities, it won’t be clear what methods are more or less truth-seeking. So poor epistemic capabilities can be self-reinforcing, and vice versa.
Veil of ignorance. Conversely, people may be more enthusiastic to invest in novel, strong epistemic methods while they think that those methods will come to support their current beliefs (which would be the default, if they actually believe their current beliefs). Whereas if they first learn that the methods are going to contradict their current beliefs, then they may oppose them.
Early investment. I can easily imagine both a future where frontier AI projects either (i) spend continuous effort on making their AIs strong forecasters and strategic analysts, and distributes those capabilities to other key institutions, or (ii) almost exclusively focus on using their AI systems for other tasks, such as technical R&D. My being able to imagine both might just be a fact of my own ignorance — but it’s at least suggestive that both futures are plausible, and could come about depending on our actions.
Distribution of epistemic capabilities. Even without changing the pace at which powerful AI epistemics are developed, the question of whether important decisions are made with or without AI epistemic assistance may depend on how quickly different actors get access to those capabilities. It seems probably great for those epistemic capabilities to quickly be made widely available, and if they’re powerful enough, it could be essential for multiple key players (such as AI companies, governments, and opposition parties) to get access to them at a similar time, so they can provide checks on each others’ new capabilities.
To be more concrete
Now, let’s be more specific about what goals could be important to achieve in this area. I think these are the 3 most important instrumental goals to be working towards:
- Society adopts good norms & practices for AI-as-knowledge-producers, i.e., norms & practices that allow insights from AI-as-knowledge-producers to be widely spread and appropriately trusted.
- Society adopts good norms & practices for AI-as-communicators, i.e., norms & practices that make it systematically easy for AIs to spread true information and relatively more difficult for AIs to illegitimately persuade people of falsehoods.
- For a given amount of general capabilities, we have high “epistemic capabilities” and high justified trust in those capabilities.
Let’s go through these in order.
Good norms & practices for AI-as-knowledge-producers
Let’s talk about norms and practices for AIs as knowledge-producers. With this, I mean AIs doing original research, rather than just reporting claims discovered elsewhere. (I.e., AIs doing the sort of work that you wouldn’t get to publish on Wikipedia.)
Here are some norms/institutions/practices that I think would contribute to good usage of AI-as-knowledge-producers:
Good norms & practices for AI-as-communicators
Now let’s talk about norms for AIs as communicators. This is the other side of the coin from “AI as knowledge producers”. I’m centrally thinking about AIs talking with people and answering their questions.
Here are some norms/institutions/practices that I think would enable good usage of AI-as-communicators:
- Transparency about how AIs decide what to communicate.
- E.g. via publishing information about AIs’ constitutions or model spec.
- Independent evaluators publishing reports on AI truthfulness, including…
- Fraction of statements that the evaluators believe to be clearly true, debatable, vs. clearly false.
- Results from AI lie-detection tests on whether the AI is being dishonest.
- The degree to which AIs contradict themselves in different contexts or when talking with different audiences.
- The degree to which AI is misleading via behaving differently (e.g. being more or less evasive, using a different tone, or relying on very different sources) on questions that are similar except for their implications about a topic that the AI may want to mislead users about (e.g. something political or something where the AI developer has commercial interests).
- Experiments on whether humans tend to be better or worse at answering questions about a certain topic after conversing the AI about related topics. If humans are systematically worse, that suggests that the AI may be systematically misleading.
- Certain laws about AI communication may be helpful, such as:
- Conversely, it’s also important to avoid bad laws. For example, laws that forbid AIs from saying blatant falsehoods may be good if they were judged in a reasonable way, and had the threshold for “blatant” set highly enough, but they could also be very bad if they became a tool for pushing political agendas.
Differentially high epistemic capabilities
Finally: I want AIs to have high epistemic capabilities compared to their other capabilities. (Especially dangerous ones.) Here are three metrics of “epistemic capabilities” that I care about (and what “other capabilities” to contrast them with):
In order for these distinctions to be decision-relevant, there needs to be ways of differentially accelerating one side of the comparison compared to the other. Here are two broad categories of interventions that I think have a good shot at doing so:
- Scalable Oversight, Weak-to-Strong Generalization (W2S generalization), and Elicing Latent Knowledge (ELK).
- I list these three domains together because they’re strongly related.
- These research areas push forward “aligned capabilities” via letting us elicit stronger capabilities towards arbitrary goals.
- This pushes towards “asymmetric persuasion” via letting people increase their skepticism of unsupported AI statements, while still being able to believe AI statements backed up by a spot-checked decomposed argument (scalable oversight), by a scientific understanding of generalization (W2S generalization), or by methods that directly eliciting AI’s latent knowledge.
- This pushes forward “understanding” over “building technology” via being disproportionately helpful for boosting capabilities in areas with poor feedback loops. (Whereas I think “building technology” typically has better feedback loops.)
- Building and iteratively improving capabilities on “understanding”-loaded tasks, such as forecasting and strategic analysis.
- (This partly overlaps with the first point, because you might want to practice using scalable-oversight/W2S-generalization/ELK on these tasks in particular.)
- Examples of how you might do this includes:
- Making the models do forecasts of unseen data, and iterating to improve their performance.
- This becomes more interesting if you can train highly capable models on only old data, since this would let you test and iterate the models on more long-range forecasting.
- Training models using experts’ (superforecasters, policy analysts, AI strategy researchers) feedback.
- Either in a baseline RLHF sort-of way, or going further towards scalable oversight & W2S generalization.
- Experimentally determine what sort of procedures and arguments tend to lead humans towards truth.
- For example, via the methodology that Tom Davidson outlines in this post. Note that this might be meaningfully different than the procedures that work well in technical domains, because of the fuzzier topics and increased political biases.
- I think this differentially pushes forward “aligned capabilities” in “understanding”-loaded domains, because I expect that models will (via generalization from pre-training) start out with some baseline understanding of these domains. Effort on these tasks will go towards some mix of increasing model capabilities and improving our ability to elicit existing capabilities, and I expect the net-effect will be to somewhat reduce the amount of capabilities that we can’t elicit. (But I don’t feel fully confident in this.)
- This can push towards “asymmetric persuasion” in these domains insofar as developers take care to develop truth-seeking methods rather than just indiscriminately iterating to improve models’ ability to persuade people.
- This clearly differentially pushes towards “understanding”.
Heuristics for good interventions
Having spelled-out what we want in the way of epistemic capabilities, practices for AI-as-knowledge-producers, and AI-as-communicators: Let’s talk about how we can achieve these goals. This section will talk about broad guidelines and heuristics, while the next section will talk about concrete interventions. I discuss:
Direct vs. indirect strategies
One useful distinction is between direct and indirect strategies. While direct strategies aim to directly push for the above goals, indirect strategies instead focus on producing demos, evals, and/or arguments indicating that epistemically powerful AI will soon be possible, in order to motivate further investment & preparation pushing toward the above goals.
My current take is that:
Indirect value generation
One possible path-to-impact from building and iteratively improving capabilities on “understanding”-loaded tasks is that this gives everyone an earlier glimpse of a future where AIs are very epistemically capable. This could then motivate:
- Investment into making AIs even more epistemically capable.
- Increased concern about AI companies’ upcoming epistemic advantage over other actors, motivating stronger demands for transparency and increased urgency of actors acquiring their own ability to verify AI epistemic methods.
- The production of further evals for exactly how epistemically capable AIs are.
- More attention, as well as more-informed attention, going towards developing norms for how AIs should communicate information.
- Also: It would directly enable AIs to start gathering a good epistemic track record earlier, which could be helpful for evaluating AIs’ trustworthiness later.
- Also: It might help build a track record for a particular organization, which could be helpful for evaluating that organization’s track-record later.
The core advantage of the indirect approach is that it seems way easier to pursue than the direct approach.
- Demos and evals for epistemically capable AI are very easily measurable/testable which gives you great feedback loops.
- Because the path-to-impact is indirect, it’s ok if methods or evals don't generalize to future, more powerful AI systems. They can still “wake people up”.
- Because the path-to-impact is indirect, it’s ok if the work takes place in organizations that will eventually be marginalized/outcompeted by the well-resourced AI company or other actors. They can still “wake people up”.
Core questions about the indirect approach: Are there really any domain-specific demos/evals that would be convincing to people here, on the margin? Or will people’s impressions be dominated by “gut impression of how smart the model is” or “benchmark performance on other tasks” or “impression of how fast the model is affecting the world-at-large”? I feel unsure about this, because I don’t have a great sense of what drives people’s expectations, here.
A more specific concern: Judgmental forecasting hasn’t “taken off” among humans. Maybe that indicates that people won’t be interested in AI forecasting? This one I feel more skeptical of. My best-guess is that AI forecasting will have an easier time of becoming widely adopted. Here’s my argument.
I don’t know a lot of why forecasting hasn’t been more widely adopted. But my guess would be that the story if something like:
- Many forecasting practices are useful. (Such as assigning probabilities to questions that are highly relevant to your decisions, keeping track of how well you’re doing, and keeping track of how well people-who-you-listen-to are doing.)
- However, they’re not useful enough that people who use these tools but who don’t have experience in a profession can easily outcompete people without experience in that profession.
- And it takes time for people in existing professions to adopt good practices. (Cultural change is slow.)
For AIs, these problems seem smaller:
- There’s already a culture of measuring AI performance numerically, so you don’t need much of a cultural shift to get AIs to quantify their probability estimates and be scored on them.
- And AI advantages will eventually deliver lots of advantages over existing human experts, so there will eventually be strong incentives to shift over to using AI.
Overall, I feel somewhat into “indirect” approaches as a path-to-impact, but only somewhat. But it at least seems worth pursuing the most leveraged efforts here: Such as making sure that we always have great forecasting benchmarks and getting AI forecasting services to work with important actors as soon as (or even before) they start working well.
On long-lasting differential capability improvements
It seems straightforward and scalable to boost epistemic capabilities in the short run. But I expect a lot of work that leads to short-run improvements won’t matter after a couple of year. (This completely ruins your path-to-impact if you’re trying to directly improve long-term capabilities — but even if you’re pursuing an indirect strategy, it’s worse for improvements to last for months than for them to last for years.)
So ideally, we want to avoid pouring effort into projects that aren’t relevant in the long-run. I think there’s two primary reasons for why projects may become irrelevant in the long run: either due to the bitter lesson or due to other people doing them better with more resources.
- The bitter lesson: This suggests that we want to avoid “leveraging human knowledge” and instead work on projects that “leverage computation”.
- This inherently gets easier over time — as investments go up and more compute becomes available. But you can still test out prototypes now.
- Example projects that are especially good for “leveraging computation” are those that explore how we could generate large amounts of data to train on, given access to large amounts of compute.
- Other people doing them better: Epistemic capabilities will be significantly boosted by efforts that other people have huge incentives to pursue, such as learning how to train models on synthetic data, or making AI agents who can pursue tasks over longer periods of time, and who can easily navigate interfaces that were made for humans. Such projects therefore don’t seem worth prioritizing.
- (Though it might be valuable to make sure that epistemic projects are in close touch with people working on more general capabilities, in order to plan around their advances and make use of their unreleased prototypes. This is one reason why “AI for epistemics” projects may benefit from being inside of frontier AI companies.)
That said, even if we mess this one up, there’s still some value in projects that temporarily boost epistemic capabilities, even if the technological discoveries don’t last long: The people who work on the project may have developed skills that let them improve future models faster, and we may get some of the indirect sources of value mentioned above.
Ultimately, key guidelines that I think are useful for this work are:
- Inform people who can influence the future of AI-for-epistemics. (As discussed in the previous section on indirect approaches.)
- Leverage computation rather than human knowledge.
- Avoid building stuff that others will build better.
Painting a picture of the future
To better understand which ones of today’s innovations will be more/less helpful for boosting future epistemics, it’s helpful to try to envision what the systems of the future will look like. In particular: It’s useful to think about the systems that we especially care about being well-designed. For me, these are the systems that can first provide a very significant boost on top of what humans can do alone, and that get used during the most high-stakes period around TAI-development.
Let’s talk about forecasting in particular. Here’s what I imagine such future forecasting systems will look like:
- In contrast to today’s systems, I don’t think they’ll have a series of hard-coded steps. I think they’ll be much more flexible in going back and forth between different types of considerations. But even if they don’t have e.g. a hard-coded “baserate generation”-step, they’ll probably still use “baserate generation” in a similar sense as human forecasters do: As a useful, flexible input into their forecasting.
- For short-horizon forecasting (e.g.: what will tomorrow’s newspaper report about ongoing events?) I imagine that they will rely a lot on hard-to-express heuristics that they’ve learned from fine-tuning. Because there’s enough data that this is feasible.
- Most of this data will be model-generated. AIs will formulate questions about tomorrow’s newspaper, that other AIs will forecast, and that other AIs will resolve after reading the next day’s paper.
- For medium-horizon forecasting (e.g. what will happen in 1-12 months) I think there won’t be enough data for finetuning to build in a lot of really opaque heuristics. But we will be able to learn what types of reasoning tends to lead to relatively better vs. worse forecasts, by instructing different systems to use different strategies (and verifying that they do so, c.f. measuring faithfulness in chain-of-thought reasoning). Then we can teach models to use the strategies and heuristics that work best. (E.g. via few-shot prompting them with good examples, or by doing supervised learning on passages of chain-of-thought that empirically did well (similar to the Halawi paper), or by using constitutional-AI-style-training to generate many more examples of similar passages to finetune on.)
For long-horizon forecasting (e.g. what will happen in >1 year) we won’t have any ground-truth data to train on, so we’ll have to rely on human feedback. In order to know what kind of feedback to give, here, we’ll want to use medium-horizon forecasting as a “lab” to test different hypotheses about what sort of AI-human interactions tend to lead to accurate forecasts, and what types of arguments tend to work well in practice.
In order to generalize sensibly from the medium-horizon-forecasting case, we’ll want the learnings from this to be as human-interpretable as possible. E.g. “arguments from analogy tends to work well/poorly”, not “this 100k word long prompt tends to give good results, and no human can understand why”.
Long-horizon forecasters can reference all kinds of evidence, including the forecasts of short- and medium-horizon forecasters, insofar as they’re relevant.
When using medium-horizon forecasting as a “lab”: We’ll want to run both (i) studies where we try to get as good forecasting abilities as we can, including by relying substantially on good generalization from AIs, and (ii) studies where a red-team tries to make the AIs maximally subtly misleading, and see whether humans who are getting AI advice can notice this, or whether they get tricked into believing terrible forecasts.
If the latter tests lead to humans making terrible forecasts, then we should assume that scheming AIs would be able to mislead us about both medium-term and long-term forecasts. (And probably also short-term forecasts in recognizably rare, high-stakes situations.)
C.f. control evaluations.
- Medium- and long-horizon forecasters may be finetuned copies of models that were previously fine-tuned on short-horizon forecasting, insofar as that tends to instill some useful intuitions.
- Ideally, someone will have trained a competent model that hasn’t seen any data from the last 3 years, or maybe 30 years, or possibly 300 years. (The longer time frames would require a lot of synthetic data.) We could use such a model as a “lab” to test hypotheses about what types of reasoning tends to do well or poorly on long-range forecasting.
Concrete projects for differentially advancing epistemic capabilities
Now, let’s talk about concrete projects for differentially advancing epistemic capabilities, and how well they do according to the above criteria and vision.
Here’s a summary/table-of-contents of projects that I feel excited about (no particular order). More discussion below.
Effort to provide AI forecasting assistance (or other ambitious epistemic assistance) to governments is another category of work that I’d really like to happen eventually. But I’m worried that there will be more friction in working with governments, so that it’s better to iterate outside them first and then try to provide services to them once they’re better. This is only a weakly held guess, though. If someone who was more familiar with governments thought they had a good chance of usefully working with them, I would be excited for them to try it.
In the above paragraph, and the above project titles, I refer to AI forecasting or “other ambitious epistemic assistance”. What do I mean by this?
- “Ambitious epistemic assistance” is meant to include projects that don’t do forecasting specifically, but that still leverage AIs to do a large amount of epistemic labor, in a way that could be scaled up to be extremely valuable.
- For example, I would want to include AIs that assist with threat modeling or policy analysis in flexible, scalable ways.
- On the other hand, an example of something that would be insufficiently ambitious is a tool that was narrowly targeted at a particular well-scoped type of analysis. Which could be perfectly automated without providing that much of an acceleration to overall strategic efforts. E.g. a highly structured literature review process like “automatically find and combine randomized trials of medical interventions”.
Now for more detail on the projects I’m most excited about.
Evals/benchmarks for forecasting (or other ambitious epistemic assistance)
- I generally think that evaluations and benchmarks are pretty leveraged for motivating work and for making it clear when models are getting seriously good.
- Some ongoing work on this includes.
- Further work on this would ideally check in with these efforts and see if there’s important angles that they don’t cover, that would be good to evaluate.
Automate forecasting question-generation and -resolution
- I.e.: Train AI systems to automatically formulate forecasting questions, and train AI systems to automatically seek out information about what happened and resolve them.
- This is strongly related to the above “evals/benchmark” category. It's a tool that can be used to generate really large amounts of questions that models can be evaluated on. (Or trained on.)
- Other than general reasons why I like evals/benchmarks, I like this angle because:
- It’s also squarely in the domain of “leveraging computation rather than human knowledge”, as a way of improving forecasting abilities. (Via training models on the automatically generated questions.)
- I think that “automatically generated and resolved forecasting questions” is a core part of what it would eventually look like to have a flourishing science of AI forecasting. And it seems great to have prototypes of all the most core parts, as early as possible, so that:
- We can track how close we are to the full vision.
- It gets easier to demo & explain what the future might look like.
- We can encounter obstacles and start working on them early
- My main concern is that the models might not quite be good enough to make this really easy, yet. So it might be significantly easier to do with generations that are one step further along.
Logistics of past-casting
- Another angle for getting more forecasting data is to exploit all the data in the past. If we could train competent AI systems without “spoiling” them on recent events, then we could run experiments on what methodologies work best for long-range forecasting.
- One question to answer here is: What methods do we have for determining the date of past data, and how reliable are they?
- Can we date past internet-data by checking for its presence in older crawls? (E.g. past versions of the common crawl.)
- By looking at website meta-data?
- By having AI models read it and guess?
- Some existing work is already trying to address this problem, and my impression is that it’s surprisingly annoying to do in practice.
- In particular, a mini-version of “past-casting” is to take existing models with training-data cutoffs several months ago, and see how well they can forecast events since then.
- Even here, you have to deal with questions about how to date information. You’d like to give AIs the option to read-up on newspaper articles and similar that are relevant to the events they’re forecasting — but it turns out to be non-trivial to ensure that e.g. the articles haven’t been updated in ways that leak information about the future.
- The fact that this is surprisingly difficult is part of why I’m excited for people to start working on it early.
- (C.f. here where Halawi et al. solves the problem by white-listing a small number of websites where it’s easy to determine dates. This means that their system cannot use data from most of the web.)
- Different methods can be evaluated by comparing them against each other.
- Note that you don’t need 100% reliability. It’s generally ok to accidentally date older data as actually being newer; it just means that models will have a bit less old data to access. And it might be ok to date some new data as old if the reason that you’re doing it is that it’s very difficult to recognise as new data — because that probably means that it’s not leaking much information about the present.
- One way to get a lot of different models that have been trained on different amounts of history would be to order all of the pre-training data chronologically and then train on it in that order. It seems useful to explore and address various problems with this.
For really long-range experiments (where we avoid spoiling AIs on the past 100+ years) we would need to be able to do pretraining with mostly synthetic data. “How to usefully pre-train models on synthetic data) is not something I recommend working on, because I think it would be very useful for AI capabilities. So I expect capabilities researchers to be good at exploring it on their own.
However, it might be useful to consider how you would prevent leaking information from the present if you could usefully pre-train models on synthetic data.
In particular, the synthetic data would probably be constructed by models that have a lot of knowledge about the present. So you would have to prevent that knowledge from leaking into the synthetic data.
(This research project may be easier to do once we understand more about good methods of training on synthetic data. I’m personally not sure what the SOTA is, here.)
- Another question someone could answer is: How much data do we even have from various time periods? (I’m not sure.)
Start efforts inside of AI companies for AI forecasting or other ambitious epistemic assistance
- It seems great for epistemic tools to be developed in close contact with users, so that the tools fill real needs. And people inside AI companies are important customers who I eventually want to be aided by epistemic assistance.
- Also, if external AI efforts ever become uncompetitive with the big companies (because of the greater AI capabilities available inside of labs) then I want people in those companies to already be working on this.
- A variant of this would be to start an effort outside of AI companies, but consult with them to understand what they’re interested in and so that they can get impressed with the technology. Such that, if external projects become uncompetitive, then the people inside of AI labs are interested in starting up similar AI & epistemics efforts inside the labs (or to provide external ones with privileged access to company models).
- My main concern is that the current technology might not actually be good enough to be really helpful, yet. Or that (in order to be sufficiently good) the current technology needs a ton of schlep-work that won’t generalize to future models.
Scalable oversight / weak-to-strong-generalization / ELK
- It’s possible that we’ll develop powerful AI systems which themselves have excellent epistemic abilities, without us being able to use those abilities for everything that we want to use them for. For example, if you trained an AI to predict the stock market, it could develop all kinds of powerful epistemic methods and interesting hypotheses about the world — but all that you would see as its developer would be its projected stock prices.
- In order for AIs to provide great epistemic assistance to humans, we want to be able to elicit and verify all the knowledge and heuristics that AI systems develop.
- This also overlaps heavily with alignment research, since if AI have abilities that humans can’t elicit from them, that makes it difficult for humans to train AIs to behave well, and more dangerous if AIs try to seize power.
- Eliciting Latent Knowledge (ELK) is a framing from the Alignment Research Center. While they work on it from a theoretical perspective, it can also be tackled experimentally. It’s the problem of eliciting knowledge that AIs “knows” even when you can’t provide any direct feedback signal that incentivizes honesty.
- Weak to Strong Generalization is a framing from OpenAI: “Our setting can be viewed as a general methodology for empirically studying problems like ELK and honesty across a wide range of tasks.”
- A related area includes lie detection of AI system, see e.g.:
- Scalable Oversight refers to attempts to amplify the overseers of an AI system such that they are more capable than the system itself (typically by having the overseers use the system itself). If successful, this could give us a feedback signal with which to train powerful AI systems while its reasoning remains understandable to (amplified) humans.
- See here for some discussion about its relationship to weak-to-strong generalization.
- There are teams working on this at Anthropic and DeepMind.
Experiments on what type of arguments and AI interactions tend to lead humans toward truth vs. mislead them
- As a final category of experiments that could be useful to run, I wanted to flag experiments on what kinds of arguments and interactions-with-AIs tend to lead humans towards the truth, and what kinds tend to mislead them.
- One example of this is Tom Davidson’s proposal described in Let’s use AI to harden human defenses against AI manipulation:
- “tldr: AI may manipulate humans; we can defend against that risk better by optimizing AIs to manipulate humans, seeing what manipulation techniques they use, and learning to detect those techniques.”
- Compared to e.g. automated forecasting, note that this centers the fact that AIs will assist humans in decision-making, and we ultimately want them to lead humans towards truth, rather than just be right themselves.
- Note also that the human component makes experiments here more time-sensitive. For AIs, we might be able to use past-casting to do experiments later on. But it’s difficult to run past-casting experiments for humans. So if you want time on forecasting questions, here, we may have to start soon.
- This category is arguably a subset of scalable oversight research (mentioned above), since an “amplified overseer” will typically be an AI-assisted human, and it’s critical for such a human to be led towards truth rather than misled by AIs. But I’m still calling it out separately because:
- Even separately from humans’ role in overseeing (i.e., providing a training signal to) AI systems, it would be great to better understand how AIs can lead humans towards truth rather than mislead them. (For example, this kind of knowledge could be important when people use AIs that they didn’t train themselves.)
- I think the results may be importantly different depending on whether you’re looking at a more technical area (e.g.: “does this code have a backdoor?” or an area that’s more interdisciplinary and politically charged (e.g.: “what would happen if the US made this deal with China on how to handle TAI?”). The latter seems especially important for “AI for epistemics”.
Concluding thoughts
The development of AI systems with powerful epistemic capabilities presents both opportunities and significant challenges for our society. Transformative AI will have a big impact on our society’s epistemic processes, and how good or bad this impact is may depend on what we do today.
I started out this post by distinguishing between efforts to differentially increase AI capabilities and efforts to enable the diffusion and appropriate trust of AI-discovered information. While I wrote a bit about this second category (characterizing it as good norms & practices for AI as knowledge producers and communicators), I will again note that the relative lack of content on it doesn’t mean that I think it’s any less important the first category.
On the topic of differentially increasing epistemic AI capabilities, I’ve argued that work on this today should (i) focus on methods that will complement rather than substitute for greater compute budgets, (ii) prioritize problems that industry isn’t already trying hard to solve, and (iii) be especially interested to show people what the future have in store by demonstrating what’s currently possible and prototyping what’s yet to come. I think that all the project ideas I listed do well according to these criteria, and I’d be excited to see more work on them.
Do you have any arguments for why this would be more important rather than working on evals of deceptive AI or evals of cybersecurity capabilities? Asking in general, I'm trying to figure out how one should think about prioritizing things like that.