@Carrickflynn,
It's been 3 years now. Is it possible to do a retake evaluating the current situation of the Disentanglement and if there has been growth in possibilities to work in AI strategy & policy (be in implementation or research)?
Thanks.
Carrick, this is an excellent post. I agree with most of the points that you make. I would, however, like to call attention to the wide consensus that exists in relation to acting prematurely.
As you observe, there are often path dependencies at play in AI strategy. Ill-conceived early actions can amplify the difficulty of taking corrective action at a later date. Under ideal circumstances, we would act under as close to certainty as possible. Achieving this ideal, however, is impractical for several interrelated reasons:
AI strategy is replete with wicked problems. The confidence that we can have in many (most?) of our policy recommendations must necessarily be relatively low. If the marginal costs of further research are high, then undertaking that research may not be worthwhile.
Delaying policy recommendations can sometimes be as harmful as or more harmful than making sub-par policy recommendations. There are several reasons for this. First, there are direct costs (e.g., lives lost prior to implementing sanitary standards). Second, delays allow other actors--most of whom are less concerned with rigor and welfare--to make relative gains in implementing their favored policies. If outcomes are path dependent, then inaction from AI strategists can lead to worse effects than missteps. Third, other actors are likely to gain influence if AI strategists delay. Opaque incentive structures and informal networks litter the path from ideation to policymaking. Even if there are not path dependencies baked into the policies themselves, there are sociopolitical path dependencies in the policymaking process. Gaining clout at an early stage tends to increase later influence. If AI strategists are unwilling to recommend policies, others will do so and reap the reputational gains entailed. Inversely, increased visibility may confer legitimacy to AI strategy as a discipline.
Policy communities in multiple countries are becoming more aware of AI, and policymaking activity is poised to increase. China's national AI strategy, released several months ago, is a long-range plan, the implementation of which is being carried out by top officials. For the CCP, AI is not a marginal issue. Westerners will look to Chinese policies to inform their own decisions. In Washington, think tanks are increasingly recognizing the importance of AI. The Center for a New American Security, for example, now has a dedicated AI program (https://www.cnas.org/research/technology-and-national-security/artificial-intelligence) and is actively hiring. Other influential organizations are following suit. While DC policymakers paid little attention to AlphaGo, they definitely noticed Putin's comments on AI's strategic importance earlier this month. As someone with an inside vantage point, I can say with a high degree of confidence that AI will not remain neglected for long. Inaction on the part of AI strategists will not mean an absence of policy; it will mean the implementation of less considered policy.
As policy discussions in relation to AI become more commonplace and more ideologically motivated, EAs will likely have less ability to influence outcomes, ceteris paribus (hence Carrick's call for individuals to build career capital). Even if we are uncertain about specific recommendations--uncertainty that may be intractable--we will need to claim a seat at the table or risk being sidelined.
There are also many advantages to starting early. To offer a few:
If AI strategists are early movers, they can wield disproportionate influence in framing the discourse. Since anchoring effects can be large, introducing policymakers to AI through the lens of safety rather than, say, national military advantage is probably quite positive in expectation.
Making policy recommendations can be useful in outsourcing cognitive labor. Once an idea becomes public, others can begin working on it. Research rarely becomes policy overnight. In the interim period, proponents and critics alike can refine thinking and increase the analytical power brought to bear on a topic. This enables greater scrutiny for longer-range thought that has no realistic path to near-term implementation, and may result in fewer unidentified considerations.
Taking reversible harmful actions at an early stage allows us to learn from our mistakes. If these mistakes are difficult to avoid ex ante, and we wait until later to make them, the consequences are likely to be more severe. Of course, we may not know which actions are reversible. This indicates to me that researching path dependence in policymaking would be valuable.
This is not a call for immediate action, and it is not to suggest that we should be irresponsible in making recommendations. I do, however, think that we should increasingly question the consensus around inaction and begin to consider more seriously how much uncertainty we are willing to accept, as well as when and how to take a more proactive approach to implementation.
I think it is important to note that in the political world there is the vision of two phases of AI development, narrow AI and general AI.
Narrow AI is happening now. The 30+% job loss predictions in the next 20 years, all narrow AI. This is what people in the political sphere are preparing for, from my exposure to it.
General AI is conveniently predicted more that 20 years away, so people aren't thinking about it because they don't know what it will look like and they have problems today to deal with.
Getting this policy response right to narrow AI does have a large impact. Large scale unemployment could destabilize countries, causing economic woes and potentially war.
So perhaps people interested in general AI policy should get involved with narrow AI policy, but make it clear that this is the first battle in a war, not the whole thing. This would place them well and they could build up reputations etc. They could be be in contact with the disentanglers so that when the general AI picture is clearer, they can make policy recommendations.
I'd love it if the narrow-general AI split was reflected in all types of AI work.
Hi Carrick,
Thanks for your thoughts on this. I found this really helpful and I think 80'000 hours could maybe consider linking to it on the AI policy guide.
Disentanglement research feels like a valid concept, and it's great to see it exposed here. But given how much weight pivots on the idea and how much uncertainty surrounds identifying these skills, it seems like disentanglement research is a subject that is itself asking for further disentanglement! Perhaps it could be a trial question for any prospective disentanglers out there.
You've given examples of some entangled and under-defined questions in AI policy and provided the example of Bostrom as exhibiting strong disentanglement skills; Ben has detailed an example of an AI strategy question that seems to require some sort of "detangling" skill; Jade has given an illuminative abstract picture. These are each very helpful. But so far, the examples are either exclusively AI strategy related or entirely abstract. The process of identifying the general attributes of good disentanglers and disentanglement research might be assisted by providing a broader range of examples to include instances of disentanglement research outside of the field of AI strategy. Both answered and unanswered research questions of this sort might be useful. (I admit to being unable to think of any good examples right now)
Moving away from disentanglement, I've been interested for some time by your fourth, tentative suggestion for existing policy-type recommendations to
fund joint intergovernmental research projects located in relatively geopolitically neutral countries with open membership and a strong commitment to a common good principle.
This is a subject that I haven't been able to find much written material on - if you're aware of any I'd be very interested to know about it. It isn't completely clear whether or how to push for an idea like this. Additionally, based on the lack of literature, it feels like this hasn't received as much thought as it should, even in an exploratory sense (but being outside of a strategy research cluster, I could be wrong on this). You mention that race dynamics are easier to start than stop, meanwhile early intergovernmental initiatives are one of the few tools that can plausibly prevent/slow/stop international races of this sort. These lead me to believe that this 'recommendation' is actually more of a high priority research area. Exploring this area appears robustly positive in expectation. I'd be interested to hear other perspectives on this subject and to know whether any groups or individuals are currently working/thinking about it, and if not, how research on it might best be started, if indeed it should be.
For five years, my favorite subject to read about was talent. Unlike developmental psychologists, I did not spend most of my learning time on learning disabilities. I also did a lot of intuition calibration which helps me detect various neurological differences in people. Thus, I have a rare area of knowledge and an unusual skill which may be useful for assisting with figuring out what types of people have a particular kind of potential, what they're like, what's correlated with their talent(s), what they might need, and how to find and identify them. If any fellow EAs can put this to use, feel free to message me.
I would broadly agree. I think this is an important post and I agree with most of the ways to prepare. I think we are not there yet for large scale AI policy/strategy.
There are few things that I would highlight as additions. 1) We need to cultivate the skills of disentanglement. Different people might be differently suited, but like all skills it is one that works better with practice and people to practice with. Lesswrong is trying to place itself as that kind of place. It is having a little resurgence with the new website www.lesserwrong.com. For example there has been lots of interesting discussion on the problems of Goodheart's law, which will be necessary to at least somewhat solve if we are to get AISafety groups that actually do AISafety research and don't just optimise some research output metric to get funding.
I am not sure if lesswrong is the correct place, but we do need places for disentanglers to grow.
2) I would also like to highlight the fact that we don't understand intelligence and that there have been lots of people studying it for a long time (psychologists etc) that I don't think we do enough to bring into discussing artificial versions of the thing they have studied. Lots of work on policy side of AI safety models it as utility maximimising agent in the economic style. I am pretty skeptical that is a good model of humans or of the AIs we will create. Figuring out what better models might be, is on the top of my personal priority list.
Edited to add 3) It seems like a sensible policy is to fund a competition in the style of at super forecasting aimed at AI and related technologies. This should give you some idea of the accuracy of peoples view on technology development/forecasting.
I would caution that we are also in the space of wicked problems so it may be there is never a complete certainty of the way we should move.
Great article, thanks Carrick!
If you're an EA who wants to work on AI policy/strategy (including in support roles), you should absolutely get in touch with 80,000 Hours about coaching. Often, we've been able to help people interested in the area clarify how they can contribute, made introductions etc.
Apply for coaching here.
We need disentanglement research examples. I tried using Google to search intelligence.org and www.fhi.ox.ac.uk for the term "disentanglement" and received zero results for both. What I need to determine whether I should pursue this path is three examples of good disentanglement research. Before reading the study or book for the examples, I will need a very quick gist - a sentence or three that summarizes what each example is about. An oversimplification is okay as long as this is mentioned and we're given a link to a paper or something so we can understand it correctly if we choose to look into it further. Additionally, I need to be shown a list of open questions.
If I am the only person who asked for this, then your article has not been very effective at getting new people to try out disentanglement research. The obstacle of not even knowing what, specifically, disentanglement research looks like would very effectively prevent a new person from getting into it. I think it would be a really good idea to write a follow-up article that contains the three examples of disentanglement research, the quick gists of what's contained in each example, and the list of open questions. That information has a chance to get someone new involved.
Thanks for writing this. My TL;DR is:
AI policy is important, but we don’t really know where to begin at the object level
You can potentially do 1 of 3 things, ATM: A. “disentanglement” research: B. operational support for (e.g.) FHI C. get in position to influence policy, and wait for policy objectives to be cleared up
Get in touch / Apply to FHI!
I think this is broadly correct, but have a lot of questions and quibbles.
That's the TLDR that I took away from the article too.
I agree that "disentanglement" is unclear. The skillset that I previously thought was needed for this was something like IQ + practical groundedness + general knowledge + conceptual clarity, and that feels mostly to be confirmed by the present article.
It seems plausible that “start iterating and create feedback loops” is a better alternative to the “wait until things are clearer” strategy.
I have some lingering doubts here as well. I would flesh out an objection to the 'disentanglement'-focus as follows: AI strategy depends critically on government, some academic communities and some companies, that are complex organizations. (Suppose that) complex organizations are best understood by an empirical/bottom-up approach, rather than by top-down theorizing. Consider the medical establishment that I have experience with. If I got ten smart effective altruists to generate mutually exclusive collectively exhaustive (MECE) hypotheses about it, as the article proposes doing for AI strategy, they would, roughly speaking, hallucinate some nonsense, that could be invalidated in minutes by someone with years of experience in the domain. So if AI strategy depends in critical components on the nature of complex institutions, then what we need for this research may be, rather than conceptual disentanglement, something more like high-level operational experience of these domains. Since it's hard to find such people, we may want to spend the intervening time interacting with these institutions or working within them on less important issues. Compared to this article, this perspective would de-emphasize the importance of disentanglement, while maintaining the emphasis on entering these institutions, and increasing the emphasis on interacting with and making connections within these institutions.
Carrick,
Given your background, I will take as given your suggestion that disentanglement research is both very important and a very rare skill. With that said, I feel like there's a reasonable meta-solution here, one that's at least worth investigating. Since you've identified at least one good disentaglement researcher (eg, Nick Bolstrom), have you considered asking them to design a test to assess possible researchers?
The suggestion may sound a bit silly, so I'll elaborate. I read your article and found it compelling. I may or may not be a good disentanglement researcher, but per your article, I probably am not. So, your article has simultaneously raised my awareness of the issue and dissuaded me from possibly helping with it. The initial pessimism, followed by your suggestion to "Read around in the area, find something sticky you think you might be able to disentangle, and take a run at it", all but guarantees low followthrough from your audience.
Ben Garfinkel's suggestion in your footnote is a step in the right direction, but it doesn't go far enough. If your assessment that the skill is easily assessed is accurate, then this is fertile ground for designing an actual filter. I'm imagining a well-defined scope (for example, a classic 1-2 page "essay test") posing an under-defined DR question. There are plenty of EA-minded folk out there who would happily spend an afternoon thinking about, and writing up their response to, an interesting question for its own sake (cf. the entire internet), and even more who'd do it in exchange for receiving a "disentanglement grade". Realistically, most submissions will be clear F/D scores within a paragraph or two (perhaps modulo initial beta testing and calibration of the writing prompt), and the responses requiring a more careful reading will be worth your while for precisely the reason that makes this exercise interesting in the first place.
TLDR: don't define a problem and immediately discourage everyone from helping you solve it.
Your link [2] points to a .docx file in a folder on a computer. It isn't a usable download link. Was that the purpose?
'file:///C:/Users/futureuser/Downloads/Careers%20in%20AI%20strategy%20and%20policy%201.4.docx#_ftnref2'
I wrote this in a google doc and copy-pasted, without intending the numbers to be links to anything. I'm not really sure why it made them highlight like a hyperlink.
I run an independent rationality group on Facebook, Evidence and Reasoning Enthusiasts. This is targeted toward people with at least some knowledge of rationality or science and halfway decent social skills. As such, I can help "build up this community and its capacity" and would like to know what specifically to do.
Quantitative social science, such as economics or analysis of survey data
Can you elaborate more on this?
I agree "disentanglement research" is unclear. To me transdisciplinarity is easier to understand. My argument is transdisciplinary research is key to developing effective AI Policy and Strategy for Africa while disentanglement could be for the west.This primary because the transdisciplinarity approach is strongly woven into the fabric of the continent. However in Africa, solutions have to be broken down to ontology specificity and then the transdisciplinarity applied. I agree with klevanoff with his notion of wicked problem. I do not know how much sense it would make if we replaced the word disentanglement with transdiscplinarity.
I think an important thing for Ai strategy is to figure out ishow to fund empirical studies into questions that impinge on crucial considerations.
For example funding studies into the nature of IQ. I'll post an article on that later but wanted to flag it here as well.
To further support my argument on why AI needs to take a transdicplinarity research perspective in Africa. I notice that there existing institutions like politics,sciences and businesses that need to work together for development to occur [1]. The writer refers to them as object interfaces that need to be integrated together for a greater purpose "interrelatedness with in interests and values" [2]. He further argues that these objects must be weakly structured and easily malleable. In the context of Africa, some of the emerging technologies like AI have no existing policies and the policy implementing institutions are developing at a slower rate than the technology is. The technology on the other hand is developed to solve challenges in the western world and introduced to Africa for adoption. Such are examples of how loose the existing structures are.Secondly, the need for these structures to be malleable is quite evident because the technology advancements like AI that have strong impact on the general public must be regulated both in deployment and development. But how does one regulate what one does not understand.The risk in this approach is that one may enforce strict restrictions which may stifle the technology innovation. I think the complexity comes from integrating epistemology principles/
[1]https://i2insights.org/2017/05/09/transdisciplinary-integration-methods/
[2]https://complexitycontrol.org/methods-of-transdisciplinary-research/
Quick question: Is your term "disentanglement research" similar to the discipline of "systems thinking" and what are the differences? (Trying to get to grips with what you mean by "disentanglement research" ) (https://en.wikipedia.org/wiki/Systems_theory)
In fact a more general version of the above question is:
What are the existing research / consultancy / etc disciplines that are most similar to the kind of work you are looking for?
If you can identify that it could help people in local communities direct people to this kind of work.
The closest seems to be really well done analytic philosophy. I recommend Nick Bostrom's work as a great example of this.
I also think that is seems similar to what mathematicians, theoretical physicists, and model builders frequently do.
Other good examples would probably be Thomas Schelling in IR and nuclear security. Coase in economics. Maybe Feynman?
(3600 words. Reading time: approximately 15 minutes with endnotes.)
Intended audience: This post is aimed at EAs and other altruistic types who are already interested in working in AI strategy and AI policy because of its potential large scale effect on the future.[1]
Epistemic status: The below represents my current best guess at how to make good use of human resources given current constraints. I might be wrong, and I would not be surprised if my views changed with time. That said, my recommendations are designed to be robustly useful across most probable scenarios. These are my personal thoughts, and do not necessarily represent the views of anyone else in the community or at the Future of Humanity Institute.[2] (For some areas where reviewers disagreed, I have added endnotes explaining the disagreement.) This post is not me acting in any official role, this is just me as an EA community member who really cares about this cause area trying to contribute my best guess for how to think about and cultivate this space.
Why my thoughts might be useful: I have been the primary recruitment person at the Future of Humanity Institute (FHI) for over a year, and am currently the project manager for FHI’s AI strategy programme. Again, I am not writing this in either of these capacities, but being in these positions has given me a chance to see just how talented the community is, to spend a lot of time thinking about how to best utilize this talent, and has provided me some amazing opportunities to talk with others about both of these things.
There are lots of ways to slice this space, depending on what exactly you are trying to see, or what point you are trying to make. The terms and definitions I am using are a bit tentative and not necessarily standard, so feel free to discard them after reading this. (These are also not all of the relevant types or areas of research or work, but the subset I want to focus on for this piece.)[3]
To quickly clarify, as I mean to use the terms, AI strategy research is an area or field of research, a bit like quantum mechanics or welfare economics. Disentanglement research I mean more as a type of research, a bit like quantitative research or conceptual analysis, and is defined more by the character of the questions researched and the methods used to advance toward clarity. Disentanglement is meant to be field agnostic. The relationship between the two is that, in my opinion, AI strategy research is an area that at its current early stage, demands a lot of disentanglement-type research to advance.
Figuring out a good strategy for approaching the development and deployment of advanced AI requires addressing enormous, entangled, under-defined questions, which exist well outside of most existing research paradigms. (This is not all it requires, but it is a central part of it at its current stage of development.)[6] This category includes the study of multi-polar versus unipolar outcomes, technical development trajectories, governance design for advanced AI, international trust and cooperation in the development of transformative capabilities, info/attention/reputation hazards in AI-related research, the dynamics of arms races and how they can be mitigated, geopolitical stabilization and great power war mitigation, research openness, structuring safe R&D dynamics, and many more topics.[7] It also requires identifying other large, entangled questions such as these to ensure no crucial considerations in this space are neglected.
From my personal experience trying and failing to do good disentanglement research and watching as some much smarter and more capable people have tried and struggled as well, I have come to think of it as a particular skill or aptitude that does not necessarily correlate strongly with other talents or expertise. A bit like mechanical, mathematical, or language aptitude. I have no idea what makes people good at this, or how exactly they do it, but it is pretty easy to identify if it has been done well once the person is finished. (I can appreciate the quality of Nick Bostrom’s work, like I can appreciate a great novel, but how they are created I don’t really understand and can’t myself replicate.) It also seems to be both quite rare and very difficult to identify in advance who will be good at this sort of work, with the only good indicator, as far as I can tell, being past history of succeeding in this type of research. The result is that it is really hard to recruit for, there are very few people doing it full time in the AI strategy space, and this number is far, far fewer than optimal.
The main importance of disentanglement research, as I imagine it, is that it makes questions and research directions clearer and more tractable for other types of research. As Nick Bostrom and others have sketched out the considerations surrounding the development of advanced AI through “disentanglement”, tractable research questions have arisen. I strongly believe that as more progress is made on topics requiring disentanglement in the AI strategy field, more tractable research questions will arise. As these more tractable questions become clear, and as they are studied, strategic direction, and concrete policy recommendations should follow. I believe this then will open up the floodgates for AI policy implementation work.
While I think that our biggest need right now is disentanglement research, there are also certain other skills and knowledge sets that would be especially helpful for advancing AI strategy research. This includes expertise in:
I expect these skills and knowledge sets to help provide valuable insight on strategic questions including governance design, diplomatic coordination and cooperation, arms race dynamics, technical timelines and capabilities, and many more areas.
There is a wide consensus in the community, with which I agree, that aside from a few robust recommendations,[8] it is important not to act or propose concrete policy in this space prematurely. We simply have too much uncertainty about the correct strategic direction. Do we want tighter or looser IP law for ML? Do we want a national AI lab? Should the government increase research funding in AI? How should we regulate lethal autonomous weapons systems? Should there be strict liability for AI accidents? It remains unclear what are good recommendations. There are path dependencies that develop quickly in many areas once a direction is initially started down. It is difficult to pass a law that is the exact opposite of a previous law recently lobbied for and passed. It is much easier to start an arms race than to stop it. With most current AI policy questions, the correct approach, I believe, is not to use heuristics of unclear applicability to choose positions, even if those heuristics have served well in other contexts,[9] but to wait until the overall strategic picture is clear, and then to push forward with whatever advances the best outcome.
This is not as big an immediate problem as the AI strategy bottleneck, but it is an issue, and one that exacerbates the research bottleneck as well.[10] FHI alone will need to fill 4 separate operations roles at senior and junior levels in the next few months. Other organizations in this space have similar shortages. These shortages also compound the research bottleneck as they make it difficult to build effective, dynamic AI strategy research groups. The lack of institutional capacity also might become a future hindrance to the massive, rapid, “AI policy implementation” mobilization which is likely to be needed.
First, I want to make clear, that if you want to work in this space, you are wanted in this space. There is a tremendous amount of need here. That said, as I currently see it, because of the low tractability of disentanglement research, institutional constraints, and the effect of both of these things on the progress of AI strategy research, a large majority of people who are very needed in this area, even extremely talented people, will not be able to directly contribute immediately. (This is not a good position we are currently in, as I think we are underutilizing our human resources, but hopefully we can fix this quickly.)
This is why I am hoping that we can build up a large community of people with a broader set of skills, and especially policy implementation skills, who are in positions of influence from which they can mobilize quickly and effectively and take important action once the bottleneck clears and direction comes into focus.
Read all the things! There are a couple of publications in the pipeline from FHI, including a broad research agenda that should hopefully advance the field a bit. Sign up to FHI’s newsletter and the EA newsletter which will have updates as the cause area advances and unfolds. There is also an extensive reading list, not especially narrowly tailored to the considerations of interest to our community, but still quite useful. I recommend skimming it and picking out some specific publications or areas to read more about.[11] Try to skill up in this area and put yourself in a position to potentially advance policy when the time comes. Even if it is inconvenient, go to EA group meet-ups and conferences, read and contribute to the forums and newsletters, keep in the loop. Be an active and engaged community member.
FHI is recruiting, but somewhat capacity limited, and trying to triage for advancing strategy as quickly as possible.
If you have good reason to think you would be good at disentanglement research on AI strategy (likely meaning a record of success with this type of research) or have expertise in the areas listed as especially in demand, please get in touch.[12] I would strongly encourage you to do this even if you would rather not work at FHI, as there are remote positions possible if needed, and other organizations I can refer you to. I would also strongly encourage you to do this even if you are reluctant to stop or put on hold whatever you are currently doing. Please also encourage your friends who likely would be good at this to strongly consider it. If I am correct, the bottleneck in this space is holding back a lot of potentially vital action by many, many people who cannot be mobilized until they have a direction in which to push. (The framers need the foundation finished before they can start.) Anything you can contribute to advancing this field of research will have dramatic force multiplicative effects by “creating jobs” for dozens or hundreds of other researchers and implementers. You should also consider applying for one or both of the AI Macrostrategy roles at FHI if you see this before 29 Sept 2017.[13]
If you are unsure of your skill with disentanglement research, I would strongly encourage you to try to make some independent progress on a question of this type and see how you do. I realize this task itself is a bit under-defined, but that is also really part of the problem space itself, and the thing you are trying to test your skills with. Read around in the area, find something sticky you think you might be able to disentangle, and take a run at it.[14] If it goes well, whether or not you want to get into the space immediately, please send it in.
If you feel as though you might be a borderline candidate because of your relative inexperience with an area of in-demand expertise, you might consider trying to tool up a bit in the area, or applying for an internship. You might also err on the side of sending in a CV and cover letter just in case you are miscalibrated about your skill compared to other applicants. That said, again, do not think that you not being immediately employed is any reflection of your expected value in this space! Do not be discouraged, please stay interested, and continue to pursue this!
Being a contributor to this effort, as I imagine it, requires investing in yourself, your career, and the community, while positioning yourself well for action once the bottleneck unjams and a robust strategic direction is clear.
I also highly recommend investing in building up your skills and career capital. This likely means excelling in school, going to graduate school, pursuing relevant internships, building up your CV, etc. Additionally, stay in close communication with the EA community and keep up to date with opportunities in this space as they develop. (Several people are currently looking at starting programs specifically to on-ramp promising people into this space. One reason to sign up to the newsletters is to make sure you don't miss such opportunities.) To repeat myself from above, attend meet-ups and conferences, read the forums and newsletters, and be active in the community. Ideally AI strategy and policy will become a sub-community within EA and a strong self-reinforcing career network.
A good way to determine how to prepare and tool up for a career in either AI policy research or implementation is to look at the 80,000 Hours’ Guide to working in AI policy and strategy. Fields of study that are likely to be most useful for AI policy implementation include policy, politics and international relations, quantitative social sciences, and law.
Especially useful is finding roles of influence or importance, even with low probability but high expected value, within (especially the US federal) government.[15] Other potentially useful paths include non-profit management, project management, communications, public relations, grantmaking, policy advising at tech companies, lobbying, party and electoral politics and advising, political “staffing,” or research within academia, thinks tanks, or large corporate research groups especially in the areas of machine learning, policy, governance, law, defense, and related. A lot of information about the skills needed for various sub-fields within this area are available at 80,000 Hours.
Another important bottleneck in this space, though smaller in my estimation than the main bottleneck, is in institutional capacity within this currently tiny field. As mentioned already above, FHI needs to fill 4 separate operations roles at senior and junior levels in the next few months. (We are also in need of a temporary junior-level operations person immediately, if you are an EU citizen, consider getting in touch about this!)[16][17] Other organizations in this space have similar shortages. If you are an experienced manager, administrator, or similar, please consider applying or getting in touch for our senior roles. Alternatively, if you are freshly out of school, but have some proven hustle (especially proven by extensive extracurricular involvement, such as running projects or groups) and would potentially like to take a few years to advance this cause area before going to graduate school or locking in a career path, consider applying for a junior operations position, or get in touch.[18] Keep in mind that operations work at an organization like FHI can be a fantastic way to tool up and gain fluency in this space, orient yourself, discover your strengths and interests, and make contacts, even if one intends to move on to non-operations roles eventually.
The points I hope you can take away in approximate order of importance:
1) If you are interested in advancing this area, stay involved. Your expected value is extremely high, even if there are no excellent immediate opportunities to have a direct impact. Please join this community, and build up your capacity for future research and policy impact in this space.
2) If you are good at “disentanglement research” please get in touch, as I think this is our major bottleneck in the area of AI strategy research, and is preventing earlier and broader mobilization and utilization of our community’s talent.
3) If you are strong or moderately strong in key high-value areas, please also get in touch. (Perhaps err to the side of getting in touch if you are unsure.)
4) Excellent things to do to add value to this area, in expectation, include:
a) Investing in your skills and career capital, especially in high-value areas, such as studying in-demand topics.
b) Building a career in a position of influence (especially in government, global institutions, or in important tech firms.)
c) Helping to build up this community and its capacity, including building a strong and mutually reinforcing career network among people pursuing AI policy implementation from an EA or altruistic perspective.
5) Also of very high value is operations work and other efforts to increase institutional capacity.
Thank you for taking the time to read this. While it is very unfortunate that the current ground reality is, as far as I can tell, not well structured for immediate wide mobilization, I am confident that we can do a great deal of preparatory and positioning work as a community, and that with some forceful pushing on these bottlenecks, we can turn this enormous latent capacity into extremely valuable impact.
Let’s getting going “doing good together” as we navigate this difficult area, and help make a tremendous future!
[1] For those of you not yet interested in working in AI strategy or policy, who are interested in exploring why you might want to be, I recommend this short EA Global talk, the Policy Desiderata paper, and OpenPhil’s analysis. For a very short consideration on why the far future matters, I recommend this very short piece, and for a quick fun primer on AI as transformative I recommend this. Finally, once the hook is set, the best resource remains Superintelligence.
[2] On a related note, I want to thank Miles Brundage, Owen Cotton-Barratt, Allan Dafoe, Ben Garfinkel, Roxanne Heston, Holden Karnofsky, Jade Leung, Kathryn Mecrow, Luke Muehlhauser, Michael Page, Tanya Singh, and Andrew Snyder-Beattie for their comments on early drafts of this post. Their input dramatically improved it. That said, again, they should not be viewed as endorsing anything in this. All mistakes are mine. All views are mine.)
[3] There are some interesting tentative taxonomies and definitions of the research space floating around. I personally find the following, quoting from a draft document by Allan Dafoe, especially useful:
AI strategy [can be divided into]... four complementary research clusters: the technical landscape, AI politics, AI governance, and AI policy. Each of these clusters characterizes a set of problems and approaches, within which the density of conversation is likely to be greater. However, most work in this space will need to engage the other clusters, drawing from and contributing high-level insights. This framework can perhaps be clarified by analogy to the problem of building a new city. The technical landscape examines the technical inputs and constraints to the problem, such as trends in the price and strength of steel. Politics considers the contending motivations of various actors (such as developers, residents, businesses), the possible mutually harmful dynamics that could arise and strategies for cooperating to overcome them. Governance involves understanding the ways that infrastructure, laws, and norms can be used to build the best city, and proposing ideal masterplans of these to facilitate convergence on a common good vision. The policy cluster involves crafting the actual policies to be implemented to build this city.
In a comment on this draft, Jade Leung pointed out what I think is an important implicit gap in the terms I am using, which also highlights the importance of not treating these terms as either final, comprehensive, or especially applicable outside of this piece:
There seems to be a gap between [AI policy implementation] and 'AI strategy research' - where does the policy research feed in? I.e. the research required to canvas and analyse policy mechanisms by which strategies are most viably realised, prior to implementation (which reads here more as boots-on-the-ground alliance building, negotiating, resource distribution etc.)
[4] Definition lightly adapted from Allan Dafoe and Luke Muehlhauser.
[5]This idea owes a lot to conversations with Owen Cotton-Barratt, Ben Garfinkel, and Michael Page.
[6] I did not get a sense that any reviewer necessarily disagreed that this is a fair conceptualization of a type of research in this space, though some questioned its importance or centrality to current AI strategy research. I think the central disagreement here is on how many well-defined and concrete questions there are left to answer at the moment, how far answering them is likely to go in bringing clarity to this space and developing robust policy recommendations, and the relative marginal value of addressing these existing questions versus producing more questions through disentanglement of the less well defined areas.
[7] One commenter did not think these were a good sample of important questions. Obviously this might be correct, but in my opinion, these are absolutely among the most important questions to gain clarity on quickly.
[8] My personal opinion is that there are only three or maybe four robust policy-type recommendations we can make to governments at this time, given our uncertainty about strategy: 1) fund safety research, 2) commit to a common good principle, and 3) avoid an arms races. The fourth suggestion is both an extension of the other three and is tentative, but is something like: fund joint intergovernmental research projects located in relatively geopolitically neutral countries with open membership and a strong commitment to a common good principle.
I should note that this point was also flagged as potentially controversial by one reviewer. Additionally, Miles Brundage, quoted below, had some useful thoughts related to my tentative fourth suggestion:
In general, detailed proposals at this stage are unlikely to be robust due to the many gaps in our strategic and empirical knowledge. We "know" arms races are probably bad but there are many imaginable ways to avoid or mitigate them, and we don't really know what the best approach is yet. For example, launching big new projects might introduce various opportunities for leakage of information that weren't there before, and politicize the issue more than might be optimal as the details are worked out. As an example of an alternative, governments could commit to subsidizing (e.g. through money and hardware access) existing developers that open themselves up to inspections, which would have some advantages and some disadvantages over the neutrally-sited new project approach.
[9] This is an area with extreme and unusual enough considerations that it seems to break normal heuristics, or at least my normal heuristics. I have personally heard at least minimally plausible arguments made by thoughtful people that openness, antitrust law and competition, government regulation, advocating opposition to lethal autonomous weapons systems, and drawing wide attention to the problems of AI might be bad things, and invasive surveillance, greater corporate concentration, and weaker cyber security might be good things. (To be clear, these were all tentative, weak, but colourable arguments, made as part of exploring the possibility space, not strongly held positions by anyone.) I find all of these very counter-intuitive.
[10] A useful comment from a reviewer on this point: “These problems are related: We desperately need new institutions to house all the important AI strategy work, but we can't know what institutions to build until we've answer more of the foundational questions.”
[11] Credit for the heroic effort of assembling this goes mostly to Matthijs Maas. While I contributed a little, I have myself only read a tiny fraction of these.
[12] [email protected].
[13] Getting in touch is a good action even if you can not or would rather not work at FHI. In my opinion, AI strategy researchers would ideally cluster in one or more research groups in order to advance this agenda as quickly as possible, but there is also some room for remote scholarship. (The AI strategy programme at FHI is currently trying to become the first of these “cluster” research groups, and we are recruiting in this area aggressively.)
[14] I’m personally bad enough at this, that my best advice is something like read around in the area, find a topic, and “do magic.” Accordingly, I will tag in Jade Leung again for a suggestion of what a “sensible, useful deliverable of 'disentanglement research' would look like”:
A conceptual model for a particular interface of the AI strategy space, articulating the sub-components, exogenous and endogenous variables of relevance, linkages etc.; An analysis of driver-pressure-interactions for a subset of actors; a deconstruction of a potential future scenario into mutually-exclusive-collectively-exhaustive (MECE) hypotheses.
Ben Garfinkel similarly volunteered to help clarify “by giving an example of a very broad question that seem[s] to require some sort of "detangling" skill:”
What does the space of plausible "AI development scenarios" look like, and how do their policy implications differ?
If AI strategy is "the study of how humanity can best navigate the transition to a world with advanced AI systems," then it seems like it ought to be quite relevant what this transition will look like. To point at two different very different possibilities, there might be a steady, piecemeal improvement of AI capabilities -- like the steady, piecemeal improvement of industrial technology that characterized the industrial revolution -- or there might be a discontinuous jump, enabled by sudden breakthroughs or an "intelligence explosion," from roughly present-level systems to systems that are more capable than humans at nearly everything. Or -- more likely -- there might be a transition that doesn't look much like either of these extremes.
Robin Hanson, Eliezer Yudkowsky, Eric Drexler, and others have all emphasized different visions of AI development, but have also found it difficult to communicate the exact nature of their views to one another. (See, for example, the Hanson-Yudkowsky "foom" debate.) Furthermore, it seems to me that their visions don't cleanly exhaust the space, and will naturally be difficult to define given the fact that so many of the relevant concepts--like "AGI," "recursive self-improvement," "agent/tool/goal-directed AI," etc.--are currently so vague.
I think it would be very helpful to have a good taxonomy of scenarios, so that we could begin to make (less ambiguous) statements like, "Policy X would be helpful in scenarios A and B, but not in scenario C," or, "If possible, we ought to try to steer towards scenario A and away from B." AI strategy is not there yet, though.
A related, "entangled" question is: Across different scenarios, what is the relationship between short and medium-term issues (like the deployment of autonomous weapons systems, or the automation of certain forms of cyberattacks) and the long-term issues that are likely to arise as the space of AI capabilities starts to subsume the space of human capabilities? For a given scenario, can these two (rough) categories of issues be cleanly "pulled apart"?
[15] 80,000 hours is experimenting with having a career coach specialize in this area, so you might consider getting in touch with them, or getting in touch with them again, if you might be interested in pursuing this route.
[16] [email protected]. This is how I snuck into FHI ~2 years ago, on a 3 week temporary contract as an office manager. I flew from the US on 4 days notice for the chance to try to gain fluency in the field. While my case of “working my way up from the mail room” is not likely to be typical (I had a strong CV), or necessarily a good model to encourage (see next footnote below) it is definitely the case that you can pick up a huge amount through osmosis at FHI, and develop a strong EA career network. This can set you up well for a wise choice of graduate programs or other career direction decisions.
[17] One reviewer cautioned against encouraging a dynamic in which already highly qualified people take junior operations roles with the expectation of transitioning directly into a research position, since this can create awkward dynamics and a potentially unhealthy institutional culture. I think this is probably, or at least plausibly, correct. Accordingly, while I think a junior operations role is great for building skills and orienting yourself, it should probably not be seen as a way of immediately transitioning to strategy research, but treated more as a method for turning post-college uncertainty into a productive plan, while also gaining valuable skills and knowledge, and directly contributing to very important work.
[18] Including locking in a career path continuing in operations. This really is an extremely high-value area for a career, and badly overlooked and neglected.
*29/9: Small edits for small mistakes*
In Tetlock's book Superforecasting, he distinguishes between two skills related to forecasting: generating questions, and answering them. This "disentanglement research" business sounds more like the first sort of work. Unfortunately, Tetlock's book focuses on the second skill, but I do believe he talks some about the first skill (e.g. giving examples of people who are good at it).
I would imagine that for generating questions, curiosity and creativity are useful. Unfortunately, the Effective Altruism movement seems to be bad at creativity.
John Cleese gave this great talk about creativity in which he distinguishes between two mental modes, "open mode" and "closed mode". Open mode is good for generating ideas, whereas closed mode is good for accomplishing well-defined tasks. It seems to me that for a lot of different reasons, the topic of AI strategy might put a person in closed mode:
Ethical obligation - Effective altruism is often framed as an ethical obligation. If I recall correctly, a Facebook poll indicated that around half of the EA community sees EA as more of an obligation than an opportunity. Obligations don't typically create a feeling of playfulness.
Size of the problem - Paul Graham writes: "Big problems are terrifying. There's an almost physical pain in facing them." AI safety strategy is almost the biggest problem imaginable.
Big names - People like Nick Bostrom, Eliezer Yudkowsky, and Eric Drexler have a very high level of prestige within the EA community. (The status difference between them and your average EA is greater than what I've observed between the students & the professor in any college class I remember taking.) Eliezer in particular can get very grumpy with you if you disagree with him. I've noticed that I'm much more apt to generate ideas if I see myself as being at the top of the status hierarchy, and if there is no penalty for coming up with a "bad" idea (even a bad idea can be a good starting point). One idea for solving the EA community's creativity problem is to encourage more EAs to develop Richard Feynman-level indifference to our local status norms.
Urgency - As you state in this post, every second counts! Unfortunately urgency typically has the effect of triggering closed mode.
Difficulty - As you state in this post, many brilliant people have tried & failed. For some people, this fact is likely to create a sense of intimidation which precludes creativity.
For curiosity, one useful exercise I've found is Anna Salamon's practice of setting a 7-minute timer and trying to think of as many questions as possible within that period. The common pattern here seems to be "quantity over quality". If you're in a mental state where you feel a small amount of reinforcement for a bad idea, and a large amount of reinforcement for a good idea, don't be surprised if a torrent of ideas follows (some of which are good).
Another practice I've found useful is keeping a notebook. Harnessing "ambient thought" and recording ideas as they come to me, in the appropriate notebook page, seems to be much more efficient on a per-minute basis than dedicated brainstorming.
If I was attacking this problem, my overall strategic approach would differ a little from what you are describing here.
I would place less emphasis on intellectual centralization and more emphasis on encouraging people to develop idiosyncratic perspectives/form their own ontologies. Rationale: if many separately developed idiosyncratic perspectives all predict that a particular action X is desirable, that is good evidence that we should do X. There's an analogy to stock trading here. (Relatedly, the finance/venture capital industry might be the segment of society that has the most domain expertise related to predicting the future, modulo principle-agent problems that come with investing other peoples' money. Please let me know if you can think of other candidates... perhaps the intelligence community?)
Discipline could be useful for reading books & passing classes which expand one's library of concepts, but once you get to the original reasoning part, discipline gets less useful. Centralization could be useful for making sure that the space of ideas relevant to AI strategy gets thoroughly covered through our collective study, and for helping people find intellectual collaborators. But I would go for beers, whiteboards, and wikis with long lists of crowdsourced pros and cons, structured to maximize the probability that usefully related ideas will at one point or another be co-located in someone's working memory, before any kind of standard curriculum. I suspect it's better to see AI strategy as a fundamentally interdisciplinary endeavor. (It might be useful to look at successful interdisciplinary research groups such as the Santa Fe Institute for ideas.) And forget all that astronomical waste nonsense for a moment. We are in a simulation. We score 1 point if we get a positive singularity, 0 points otherwise. Where is the loophole in the game's rules that the designers didn't plan for?
[Disclaimer: I haven't made a serious effort to survey the literature or systematically understand the recommendations of experts on either creativity or curiosity, and everything in this comment is just made up of bits and pieces I picked up here and there. If you agree with my hunch that creativity/curiosity are a core part of the problem, it might be worth doing a serious lit review/systematically reading authors who write about this stuff such as Thomas Kuhn, plus reading innovators in various fields who have written about their creative process.]
Another thought: Given the nature of this problem, I wonder why the focus is on enabling EAs to discover AI strategy vs trying to gather ideas from experts who are outside the community. Most college professors have office hours you can go to and ask questions. Existing experts aren't suffering from any of the issues that might put EAs in closed mode, and they already have the deep expertise it would take years for us to accumulate. I could imagine an event like the Asilomar AI conference, but for AI safety strategy, where you invite leading experts in every field that seems relevant, do the beer and whiteboards thing, and see what people come up with. (A gathering size much smaller than the Asilomar conference might be optimal for idea generation. I think it'd be interesting to sponsor independent teams where each team consists of one deep learning expert, one AI venture capitalist, one game theory person, one policy person, one historian, one EA/rationalist, etc. and then see if the teams end up agreeing about anything.)
Are there any best practices for getting academics interested in problems?
I run a group for creatives on Facebook called Altruistic Ideas. In it, I have worked to foster a creative culture. I've also written about the differences between the EA and rationality cultures vs. the culture creatives need. If this might be useful for anyone's EA goals, please feel free to message me.
I agree that creativity is key.
I'd would point out that you may need discipline to do experiments based upon your creative thoughts (if the information you need is not available). If you can't check your original reasoning against the world, you are adrift in a sea of possibilities.
Yeah, that sounds about right. Research and idea generation are synergistic processes. I'm not completely sure what the best way to balance them is.
I strongly agree that independent thinking seems undervalued (in general and in EA/LW). There is also an analogy with ensembling in machine learning (https://en.wikipedia.org/wiki/Ensemble_learning).
By "independent" I mean "thinking about something without considering others' thoughts on it" or something to that effect... it seems easy for people's thoughts to converge too much if they aren't allowed to develop in isolation.
Thinking about it now, though, I wonder if there isn't some even better middle ground; in my experience, group brainstorming can be much more productive than independent thought as I've described it.
There is a very high-level analogy with evolution: I imagine sexual reproduction might create more diversity in a population than horizontal gene transfer, since in the latter case, an idea(=gene) which seems good could rapidly become universal, and thus "local optima" might be more of a problem for the population (I have no idea if that's actually how this works biologically... in fact, it seems like it might not be, since at least some viruses/bacteria seem to do a great job of rapidly mutating to become resistant to defences/treatments.)