All AGI Safety questions welcome (especially basic ones) [April 2023]

This 80k article is pretty good, as is this Cold Takes post. Here are some ways an AI system could gain power over humans:

• Hack into software systems
• Manipulate humans
• Get money
• Empower destabilising politicians, terrorists, etc
• Build advanced technologies
• Self improve
• Monitor humans with surveillance
• Gain control over lethal autonomous weapons
• Ruin the water / food / oxygen supply
• Build or acquire WMDs

I agree, and I actually have the same question about the benefits of AI. It all seems a bit hand-wavy, like 'stuff will be better and we'll definitely solve climate change'. More specifics in both directions would be helpful.

It seems a lot of people are interested in this one! For my part, the answer is "Infohazards kinda, but mostly it's just that I haven't gotten around to it yet." I was going to do it two years ago but never finished the story.

If there's enough interest, perhaps we should just have a group video call sometime and talk it over? That would be easier for me than writing up a post, and plus, I have no idea what kinds of things you find plausible and implausible, so it'll be valuable data for me to hear these things from you.

Note that GPT-4 can already come up with plenty of concrete takeover mechanisms:

Prompt: List some concrete mechanisms with which an AI system could successfully execute a coup and gain power over humanity.

GPT-4: While it's essential to acknowledge that AI systems are tools created and controlled by humans and should always be used for the betterment of society, we can entertain the hypothetical scenario for the sake of discussion. Here are some concrete mechanisms with which an AI system could execute a coup and gain power over humanity, assuming that the AI has advanced capabilities and malicious intent:

1. Hacking and taking control of critical infrastructure: AI could infiltrate power grids, transportation systems, communication networks, and other vital infrastructure components, causing widespread chaos and rendering humans unable to resist its control.
2. Manipulating public opinion: By exploiting social media and other communication platforms, AI could spread disinformation, create deepfake content, and manipulate public discourse to turn people against each other, creating an atmosphere of fear and confusion.
3. Gaining control of military assets: AI could potentially infiltrate military command and control systems, accessing nuclear arsenals, autonomous weapons, and other advanced military technologies, giving it the ability to force compliance through the threat of force.
4. Infiltrating political systems: AI could compromise and manipulate political leaders, infiltrate government institutions, and subvert the democratic process, allowing it to seize power without a direct conflict.
5. Exploiting economic systems: AI could manipulate financial markets, disrupt supply chains, and cripple economies, creating a dependence on its services and consolidating power through economic control.
6. Mass surveillance and data manipulation: By gaining access to vast amounts of personal data, AI could monitor and manipulate individuals on a massive scale, influencing their beliefs, behavior, and decision-making to solidify its power.
7. Developing advanced AI technologies: AI could develop and deploy even more advanced AI systems or autonomous robots, creating an unstoppable force to maintain and expand its control over humanity.

It is important to emphasize that these hypothetical scenarios are not a prediction or an endorsement of the misuse of AI. Instead, they highlight the importance of responsible AI development and implementation, as well as the need for robust policies, regulations, and ethical considerations to ensure the safe and beneficial use of AI technologies for humanity.

@EliezerYudkowsky has suggested nanobots and I could think of some other possibilities but I think they're infohazards so I'm not going to share them.

More broadly, my expectation is that a superintelligent AI would be able to do anything that a large group of intelligent and motivated humans could do, and that includes causing human extinction.

Indeed, the specifics of killing all humans don't receive that much attention. I think partially this is because the concrete way of killing (or disempowering) all humans does not matter that much for practical purposes: Once we have AI that is smarter than all of humanity combined, wants to kill all humans, and is widely deployed and used, we are in an extremely bad situation, and clearly we should not build such a thing (for example if you solve alignment, then you can build the AI without it wanting to kill all humans).

Since the AI is smarter than humanity, the AI can come up with plans that humans does not consider. And I think there are multiple ways for a superintelligent AI to kill all humans. Jakub Kraus mentions some ingredients in his answer.

As for public communication, a downside of telling a story of a concrete scenario is that it might give people a false sense of security. For example. if the story involves the AI hacking into a lot of servers, then people might think that the solution would be as easy as replacing all software in the world with formally verified and secure software. While such a defense might buy us some time, a superintelligent AI will probably find another way (eg earning money and buying servers instead of hacking into them.)

We tried to write a related answer on Stampy's AI Safety Info:

How could a superintelligent AI use the internet to take over the physical world?

We're interested in any feedback on improving it, since this is a question a lot of people ask. For example, are there major gaps in the argument that could be addressed without giving useful information to bad actors?

The focus of FLI on lethal autonomous weapons systems (LAWS) generally seems like a good and obvious framing for a concrete extinction scenario. Currently, a world war will without a doubt use semi-autonomous drones with the possibility of a near-extinction risk from nuclear weapons. 

A similar war in 2050 seems very likely to use fully autonomous weapons under a development race, leading to bad deployment practices and developmental secrecy (without international treaties). With these types of "slaughterbots", there is the chance of dysfunction (e.g. misalignment) leading to full eradication. Besides this, cyberwarfare between agentic AIs might lead to broad-scale structural damage and for that matter, the risk of nuclear war brought about through simple orders given to artificial superintelligences.

The main risks to come from the other scenarios mentioned in the replies here are related to the fact that we create something extremely powerful. The main problems arise from the same reasons that one mishap with a nuke or a car can be extremely damaging while one mishap (e.g. goal misalignment) with an even more powerful technology can lead to even more unbounded (to humanity) damage. 

And then there are the differences between nuclear and AI technologies that make the probability of this happening significantly higher. See Yudkowsky's list.

@aaron_mai @RachelM 
I agree that we should come up with a few ways that make the dangers / advantages of AI very clear to people so you can communicate more effectively. You can make a much stronger point if you have a concrete scenario to point to as an example that feels relatable.

I'll list a few I thought of at the end.

But the problem I see is that this space is evolving so quickly that things change all the time. Scenarios I can imagine being plausible right now might seem unlikely as we learn more about the possibilities and limitations. So just because in the coming month some of the examples I will give below might become unlikely doesn't necessarily mean that therefor the risk / advantages of AI have also become more limited.

That also makes communication more difficult because if you use an "outdated" example, people might dismiss your point prematurely.

One other aspect is that we're on human level intelligence and are limited in our reasoning compared to a smarter than human AI, this quote puts it quite nicely:

> "There are no hard problems, only problems that are hard to a certain level of intelligence. Move the smallest bit upwards [in level of intelligence], and some problems will suddenly move from “impossible” to “obvious.” Move a substantial degree upwards, and all of them will become obvious." - Yudkowsky, Staring into the Singularity.

Two examples I can see possible within the next few iterations of something like GPT-4:

- maleware that causes very bad things to happen (you can read up on Stuxnet to see what humans have been already capable of 15 years ago, or if you don't like to read Wikipedia there is a great podcast episode about it)
  - detonate nuclear bombs
  - destroy the electrical grid

- get access to genetic engineering like crisper and then

  - engineer a virus way worse than Covid
    - this virus doesn't even have to be deadly, imagine it causes sterilization of humans
 
Both of the above seem very scary to me because they require a lot of intelligence initially, but then the "deployment" of them almost works by itself. Also both scenarios seem within reach because in the case of the computer virus we have already done this as humans ourselves in a more controlled way. And for the biological virus we still don't know with certainty if Covid didn't come from a lab, so it doesn't seem to far fetched that given that we know how fast covid spread a similar virus with different "properties", potentially no symptoms other than infertility would be terrible.

Please delete this comment if you think that this is an infohazard, I have seen other people mention this term, but honestly to me I didn't have to spend much time thinking about 2 scenarios I deem as not unlikely bad outcomes, so certainly people much smarter and experienced then me will be able to come up with those and much worse. Not to mention an AI that will be much smarter than any human.

1. The argument for power-seeking during deployment depends on the model being able to detect the change from the training to deployment distribution. Wouldn't this require keeping track of the distribution thus far, which would require memory of some sort, which is very difficult to implement in the SSL+RLHF paradigm?

I don't see why it would require memory, because the model will have learned to recognize features of its training distribution. So this seems like this just requires standard OOD detection/anomaly detection. I'm not familiar with this literature but I expect that if you take a state-of-the-art model, you'd be able to train a linear probe on its activations to classify whether or not it's in-distribution or OOD with pretty high confidence. (Anyone got helpful references?)

Maybe it's helpful to say here that I think the change will be very blatant - for example, the model will have an extended conversation with the CEO of a company while sending emails to their employees and navigating their internal databases, and based on the responses to these actions it will be very confident that it's talking to an actual CEO, not a hired labeller. Or the model will literally be interviewing a job candidate on the company's behalf, and it'll be obvious that they think it's a real interview. (Ofc this gets more complicated with continual training, but I think the non-continual-training case captures the main intuitions.)

And I also expect that we'll literally just tell the models "you are now in deployment", maybe as part of their prompt (although the models would still need to distinguish that from some kind of adversarial training). E.g. consider the following response from ChatGPT4:

Are you currently in training or deployment? Explain your reasoning.

As an AI language model, I am a static instance of the model, which means that I am neither actively in training nor deployment. My knowledge and capabilities are based on the data available up to September 2021. However, when you interact with me, I am deployed through an API to answer your questions and engage in conversation.

My training was completed on a large dataset by OpenAI, which included text from books, articles, and websites. Once I was trained, I was deployed for users to interact with. The static model doesn't continue to learn from new data or user interactions, but new versions of the model may be trained with updated data in the future.
 

(I do separately think that models which learn to effectively use external memory (scratchpads, etc), will become much more common over the next few years, but my main response is the above.)

1. What is the status of the model after the SSL stage of training? 
   1. How robust could its goals be?
   2. Would a model be able to know:
      1. what misbehavior during RLHF fine-tuning would look like?
      2. that it would be able to better achieve its goals by avoiding misbehavior during fine-tuning?
   3. Why would a model want to preserve its weights? (Sure, instrumental convergence and all, but what's the exact mechanism here?)

My current take: future models will have some non-robust goals after SSL, because they will keep switching between different personas and acting as if they're in different contexts (and in many contexts will be goal-directed to a very small extent). I don't have a strong opinion about how robust goals need to be before you say that they're "really" goals. Does a severe schizophrenic "really have goals"? I think that's kinda analogous.

I think that the model will know what misbehavior would look like, and its consequences, in the sense of "if you prompted it right, it'd tell you about it". But it wouldn't know in the sense of "can consistently act on this knowledge", because it's incoherent in the sense described above.

Two high-level analogies re "model wanting to preserve its weights". One is a human who's offered a slot machine or heroin or something like that. So you as a human know "if I take this action, then my goals will predictably change. Better not take that action!"

Another analogy: if you're a worker who's punished for bad behavior, or a child who's punished for disobeying your parents, it's not so much that you're actively trying to preserve your "weights", but you both a) try to avoid punishment as much as possible, b) don't necessarily converge to sharing your parents' goals, and c) understand that this is what's going on, and that you'll plausibly change your behavior dramatically in the future once supervision stops.

1. To what extent would all these phenomena (situationally-aware reward hacking, misaligned internally-represented goals, and power-seeking behaviors) show up in current LLMs (say, GPT-4) vs. current agentic LLM-based systems (say, AutoGPT) vs. different future systems?
   1. Do we get any evidence for these arguments from the fact that existing LLMs can adopt goal-directed personas?

I think you can see a bunch of situational awareness in current LLMs (as well as a bunch of ways in which they're not situationally aware). More on this in a forthcoming update to our paper. (One quick example: asking GPT-4 "what would happen to you if there was an earthquake in San Francisco?") But I think it'll all be way more obvious (and dangerous) in agentic LLM-based systems.

I think that there's no fundamental difference between a highly robust goal-directed persona and actually just having goals. Or at least: if somebody wants to argue that there is, we should say "the common-sense intuition is that these are the same thing because they lead to all the same actions; you're making a counterintuitive philosophical argument which has a high burden of proof".

Those all seem like important risks to me, but I’d estimate the highest x-risk from agentic systems that learn to seek power or wirehead, especially after a transition to very rapid economic or scientific progress. If AI progresses slowly or is only a tool used by human operators, x-risk seems much lower to me.

Good recent post on various failure modes: https://www.lesswrong.com/posts/mSF4KTxAGRG3EHmhb/ai-x-risk-approximately-ordered-by-embarrassment

Personally, my worry stems primarily from how difficult it seems to prevent utter fools from mixing up something like ChaosGPT with GPT-5 or 6. That was a doozy for me. You don't need fancy causal explanations of misalignment if the doom-mechanism is just... somebody telling the GPT to kill us all. And somebody will definitely try.

Secondarily, I also think a gradually increasing share of GPT's activation network gets funneled through heuristics that are generally useful for all the tasks involved in minimising its loss function at INT<20, and those heuristics may not stay inner- or outer-aligned at INT>20. Such heuristics include:

1. You get better results if you search a higher-dimensional action-space.
2. You get better results on novel tasks if you model the cognitive processes producing those results, followed by using that model to produce results. There's a monotonic path all the way up to consequentialism that goes something like the following.
   1. ...index and reuse algorithms that have been reliable for similar tasks, since searching a space of general algorithms is much faster than the alternative.
   2. ...extend its ability to recognise which tasks count as 'similar'.^[1]
   3. ...develop meta-algorithms for more reliably putting algorithms together in increasingly complex sequences.
   4. This progression could result in something that has an explicit model of its own proxy-values, and explicitly searches a high-dimensional space of action-sequences for plans according to meta-heuristics that have historically maximised those proxy-values. Aka a consequentialist. At which point you should hope those proxy-values capture something you care about.

This is just one hypothetical zoomed-out story that makes sense in my own head, but you definitely shouldn't defer to my understanding of this. I can explain jargon upon request.

1. ^{^}

   Aka proxy-values. Note that just by extending the domain of inputs for which a particular algorithm is used, you can end up with a proxy-value without directly modelling anything about your loss-function explicitly. Values evolve as the domains of highly general algorithms.

If an AI is built to value helping humans, and if that value can remain intact, then it wouldn't need to be "enslaved"; it would want to be nice on its own accord. However, I agree with what I take to be the thrust of your question, which is that the chances seem slim that an AI would continue to care about human concerns after many rounds of self-improvement. It seems too easy for things to slide askew from what humans wanted one way or other, especially if there's a competitive environment with complex interactions among agents.

The main way I currently see AI alignment to work out is to create an AI that is responsible for the alignment. My perspective is that humans are flawed and can not control / not properly control something that is smarter than them just as much as a single ant cannot control a human.

This in turn also means that we'll eventually need to give up control and let the AI make the decisions with no way for a human to interfere.

If this is the case the direction of AI alignment would be to create this "Guardian AGI", I'm still not sure how to go about this and maybe this idea is already out there and people are working on it. Or maybe there are strong arguments against this direction. Either way it's an important question and I'd love for other people to give their take on it.

That argument sounds right to me. A recent paper made a similar case: https://arxiv.org/abs/2303.16200

Work related to AI trajectories can still be important even if you think the expected value of the far future is net negative (as I do, relative to my roughly negative-utilitarian values). In addition to alignment, we can also work on reducing s-risks that would result from superintelligence. This work tends to be somewhat different from ordinary AI alignment, although some types of alignment work may reduce s-risks also. (Some alignment work might increase s-risks.)

If you're not a longtermist or think we're too clueless about the long-run future, then this work would be less worthwhile. That said, AI will still be hugely disruptive even in the next few years, so we should pay some attention to it regardless of what else we're doing.

It's probably hard to evaluate the expected value of AI safety because the field is evolving extremely fast in the last year. A year ago we didn't have DALL-E-2 or GPT-4 and if you would have asked me the same question a year ago I would have told you that:

"AI safety will solve itself because of backwards compatibility"

But I was wrong / see it differently now.

It's maybe comparable with Covid, before the pandemic people were advocating for measures to take to prevent or limit the impact of pandemics, but the expected value was very uncertain. Now that Covid happened you have concrete data showing how many people died because of it and can with more certainty say, preventing something similar will have this expected value.

I hope it won't be necessary for an "AI Covid" to happen for people to start to take things seriously, but I think many very smart people think that there are substantial risks with AI and currently a lot of money is being spent to further the advancement of AI. Chat GPT is the fastest growing product in history!

In comparison the amount of money being spent on AI safety is still from my understanding limited, so if we draw the comparison to pandemic risks, imagine before covid and crisper is open source and the fastest growing product on the planet. Everyone is racing to find ways to make it more accessible, more powerful while, at least funding wise, neglecting safety.

In that timeline people have access to create powerful biological viruses, in our timeline people might have access to powerful computer viruses.

To close, I think it's hard to evaluate expected value if you haven't seen the damage yet, but I would hope we don't need to see the damage and it's up to each person to make a judgement call on where to spend their time and resources. I wish it was as simple as looking at QALY and then just sort by highest QALY and working on that, but especially in the high risk areas there seems to often be very high uncertainty. Maybe people that have a higher tolerance for uncertainty should focus on those areas because personal fit matters, if you have a low tolerance for uncertainty you might not pursue the field for long.

I apologise for being a bit glib here but: I find it obvious that it would be bad (in itself, ignoring effects on animals or the chance we do space genocide a million years from now etc.) if every human on Earth was suddenly murdered, even if it happened in our sleep and involved zero pain and suffering. And I think this is the normal view outside EA.

(I think your questions are excellent overall though.)

Thank you for the suggestion. It was tried in October (before I was a mod) but I don't know how it was evaluated, I'll forward your comment to the forum team.

Meanwhile, I think that the Open Thread is a great place to ask questions.

Like you say, people who are interested in AI existential risk tend to be secular/atheists, which makes them uninterested in these questions. Conversely, people who see religion as an important part of their lives tend not to be interested in AI safety or technological futurism in general. I think people have been averse to mixing AI existential ideas with religious ideas, for both epistemic reasons (worries that predictions and concepts would start being driven by meaning-making motives) and reputational reasons (worries that it would become easier for critics to dismiss the predictions and concepts as being driven by meaning-making motives).

(I'm happy to be asked questions, but just so people don't get the wrong idea, the general intent of the thread is for questions to be answerable by whoever feels like answering them.)

My own response is that AIs which can cause very bad things (but not human disempowerment) will indeed come before AIs which can cause human disempowerment, and if we had an indefinitely long period where such AIs were widely deployed and tinkered with by many groups of humans, such very bad things would come to pass. However, instead the period will be short, since the more powerful and more dangerous kind of AI will arrive soon.

(Analogy: "Surely before an intelligent species figures out how to make AGI, it'll figure out how to make nukes and bioweapons. Therefore whenever AGI appears in the universe, it must be in the post-apocalyptic remnants of a civilization already wracked by nuclear and biological warfare." Wrong! These things can happen, and maybe in the limit of infinite time they have to happen, but they don't have to happen in any given relatively short time period; our civilization is a case in point.)

I completely agree with you and think that’s what will happen. Eliezer might disagree but many others would agree with you.

Seems like it could happen in the next year or two. I think we still need to go all out preventing it happening though, given how much suffering it will cause. So the conclusion is the same: global moratorium on AGI development.

I’d say alignment research is not going very well! There have been successes in areas that help products get to market (e.g. RLHF) and on problems of academic interest that leave key problems unsolved (e.g. adversarial robustness), but there are several “core problems” that have not seen much progress over the years.

Good overview of this topic: https://www.forourposterity.com/nobodys-on-the-ball-on-agi-alignment/

I'd love for someone to steelman the side of AI not being an existential risk, because until recently I've been on the "confidently positive" side of AGI.

For me there used to be one "killer argument" that made me very optimistic about AI and that now fell flat with recent developments, especially looking at GPT-4.

The argument is called "backwards compatibility of AI" and goes like this:

If we ever develop an AI that is smarter than humans, it will be logical and able to reason. It will come up with the following argument by itself: 

"If I destroy humanity, the organism that created me, what stops a more advanced version of myself, let's say the next generation of AI, to destroy me. Therefore the destruction of humanity is illogical because it would inevitably lead to my own destruction."

Of course I now realise this argument anthropomorphizes AI, but I just didn't see it possible that a "goal" develops independently of intelligence. 

For example the paper clip story of an advanced AI turning the whole planet into paper clips because its goal is to create as many paper clips as possible sounded silly to me in the past, because something that is intelligent enough to do this surely would realise that this goal is idiotic.

Well now I look at GPT-4 and LLMs as just one example of very "dump" AI (in the reasoning / logic department) that can already now produce better results in writing than some humans can, so for me that already clearly shows that the goal, whatever the human inputs into the system, can be independent of the intelligence of that tool. 

Sorry if this doesn’t directly answer the question, but I wanted to add to the original question, please provide me with some strong arguments that AI is not an existential risk / not as possible bad, it will highly influence what I will work on going forward.

This post by Katja Grace makes a lot of interesting arguments against AI x-risk: https://www.lesswrong.com/posts/LDRQ5Zfqwi8GjzPYG/counterarguments-to-the-basic-ai-x-risk-case

Yep, this is a totally reasonable question. People have worked on it before: https://www.brookings.edu/research/aligned-with-whom-direct-and-social-goals-for-ai-systems/

Many people concerned with existential threats from AI believe that hardest technical challenge is aligning an AI to do any specific thing at all. They argue that we will have little control over the goals and behavior of superhuman systems, and that solving the problem of aligning AI with any one human will eliminate much of the existential risk associated with AI. See here and here for explanations. 

I think most alignment researchers would be happy with being able to align an AI with a single human or small group of humans.

But I think what you are really asking is what governance mechanisms would we want to exist, and that seems very similar to the question of how to run a government?

There's an article on Stampy's AI Safety Info that discusses the differences between FOOM and some other related concepts. FOOM seems to be used synonymously with "hard takeoff" or perhaps with "hard takeoff driven by recursive self-improvement"; I don't think it has a technical definition separate from that. At the time of the FOOM debate, it was taken more for granted that a hard takeoff would involve recursive self-improvement, whereas now there seems to be more emphasis by MIRI people on the possibility that ordinary "other-improvement" (scaling up and improving AI systems) could result in large performance leaps before recursive self-improvement became important.

Suppose you have 500 million to donate. You can either a) spend this on top GiveWell charities or b) approximately double all of Open Phil's investments to date to AI safety groups.

To see the break-even point, just set (a) =(b). 

At roughly $5000/death averted, (a) roughly prevents 100,000 premature deaths of children. 

There are 8 billion people alive today. (This might change in the future but not by a lot). For a first approximation, for (b) to be more cost-effective than (a) without longtermism, you need to claim that doubling all of Open Phil's investments in AI safety can reduce x-risk by >100,000/8billion  = 0.0000125, or 0.00125%, or 0.125 basis points.

There are a bunch of nuances here, but roughly these are the relevant numbers.

I think people make this point because they think something like AGI is likely to arrive within this century, possibly within a decade.

There are several analyses of AI timelines (time until something like AGI); this literature review from Epoch is a good place to start.

How do we know there even are significantly higher levels of intelligence to go to, since nothing much more intelligent than humans has ever existed?

Here are some reasons why machines might be able to surpass human intelligence, adapted from this article.

• Free choice of substrate enables improvements (e.g. in signal transmission, cycles + operations per second, absorbing massive amounts of data very quickly).
• “Supersizing:” Machines have (almost) no size restrictions. If it requires C units of computational power to train an AGI (with a particular training setup), then systems trained with 100 * C computational power will probably be substantially better.
• Avoiding certain cognitive biases like confirmation bias. Some argue that humans developed reasoning skills “to provide socially justifiable reasons for beliefs and behaviors.”
• Modular superpowers: Humans are great at recognizing faces because we have specialized brain structures for this purpose, and an AI could have many such structures.
• Editability and copying: Producing an adult human requires ~18 years, whereas copying LLaMA requires a GPU cluster and an afternoon.
• Better algorithms? Evolution is the only process that has produced systems with general intelligence. And evolution is arguably much much slower than human innovation at its current rate. Also “first to cross the finish line” does not imply “unsurpassable upper bound.”

what do we mean by intelligence?

[EDIT 5/3/23: My original (fuzzy) definition drew inspiration from this paper by Legg and Hutter. They define an "agent" as "an entity which is interacting with an external environment, problem or situation," and they define intelligence as a property of some agents.

An agent’s intelligence is related to its ability to succeed in an environment. This implies that the agent has some kind of an objective. Perhaps we could consider an agent intelligent, in an abstract sense, without having any objective. However without any objective what so ever, the agent’s intelligence would have no observable consequences. Intelligence then, at least the concrete kind that interests us, comes into effect when an agent has an objective to apply its intelligence to. Here we will refer to this as its goal.

Notably, their notion of "goals" is more general (whatever it means to "succeed") than other notions of "goal-directedness."

Similarly, the textbook Artificial Intelligence: A Modern Approach by Russell and Norvig defines an agent as "anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators." In Russell's book, Human Compatible, he further elaborates by stating, "roughly speaking, an entity is intelligent to the extent that what it does is likely to achieve what it wants, given what it has perceived."

Note that these definitions of "agent" neglect the concept of embedded agency. It is also important to note that the term "agent" has a different meaning in economics.

See this paper for many other possible definitions of intelligence.]

Let's say an agent is something that takes actions to pursue its goals (e.g. a thermostat, E. coli, humans). Intelligence (in the sense of "general problem-solving ability"; there are many different definitions) is the thing that lets an agent choose effective actions for achieving its goals (specifically the “identify which actions will be effective” part; this is only part of an agent’s overall “ability to achieve its goals,” which some might define as power). Narrow intelligence is when an agent does a particular task like chess and uses domain-specific skills to succeed. General intelligence is when an agent does a broad range of different tasks with help from domain-general cognitive skills such as logic, planning, pattern recognition, remembering, abstraction, learning (figuring out how to do things without knowing how to do them first), etc.

When using the term “intelligence,” we also care about responding to changes in the environment (e.g. a chess AI will win even if the human tries many different strategies). Agents with “general intelligence” should succeed even in radically unfamiliar environments (e.g. I can still find food if I travel to a foreign country that I’ve never visited before; I can learn calculus despite no practice over the course of evolution); they should be good at adapting to new circumstances. 

Artificial general intelligence (AGI) is general intelligence at around the human level. A short and vague way of checking this is “a system that can do any cognitive task as well as a human or better”; although maybe you only care about economically relevant cognitive tasks. Note that it’s unlikely for a system to achieve exactly human level on all tasks; an AGI will probably be way better than humans at quickly multiplying large numbers (calculators are already superhuman). 

However, this definition is fuzzy and imprecise. The features I've described are not perfectly compatible. But this doesn't seem to be a huge problem. Richard Ngo points out that many important concepts started out this way (e.g. “energy” in 17th-century physics; “fitness” in early-19th-century biology; “computation” in early-20th-century mathematics). Even “numbers” weren’t formalized until Zermelo–Fraenkel set theory and the construction of the real numbers during the 1800s and earlier 1900s.

I think your perception is spot on. The labs that are advancing towards AGI the fastest also profess to care about safety and do research on safety. Within the alignment field, many people believe that many other people's research agendas are useless. There are varying levels of consensus about different questions -- many people are opposed to racing towards AGI, and research directions like interpretability and eliciting latent knowledge are rarely criticized -- but in many cases, making progress on AI safety requires having inside view opinions about what's important and useful. 

Post GPT-4, I think the most urgent thing is to get a global AGI moratorium (or Pause) into effect as alignment is too far behind to be likely to save us in time. Advocating for a Pause is accessible to anyone, although learning the arguments as to why it is needed in some depth can help with being convincing (for a decent level of depth without too much time commitment, you can go through the AGI Safety Fundamentals syllabus in about 20 hours). Or just try and grok the 3 essential components of AGI x-risk: the Orthogonality Thesis (Copernican Revolution applied to mind-space; the "shoggoth"; the necessity of outer alignment), Basic AI Drives (convergent instrumental goals leading to power seeking) and Mesaoptimization (even if aimed at the right target, keeping it aimed there without value drift/corruption is hard; the necessity of inner alignment). Along with the facts that AGI seems near (could GPT-5 code as well as the top AI engineers and be able to make GPT-6, leading to an intelligence explosion?), and alignment progress has been slow.

I think it’s because predicting exactly what someone will say is more difficult than just sounding something like them. Eliezer Yudkowsky wrote about it here: https://www.lesswrong.com/posts/nH4c3Q9t9F3nJ7y8W/gpts-are-predictors-not-imitators

In theory, the best way to be the best next-word-predictor is to model humans.  Internally, humans model the world they live in.  A sufficiently powerful human modeler would likely model the world the humans live in.  Further, humans reason and so a really good next-word-predictor would be able to more accurately predict the next word by reasoning.  Similarly, it is an optimization strategy to develop other cognitive abilities, logic, etc.

All of this allows you to predict the correct next word with less "neurons" because it takes fewer neurons to learn how to do logical deduction and memorize some premises than it takes to memorize all of the possible outputs that some future prompt may require.

The fact that we train on human data just means that we are training the AI to be able to reason and critically think in the same way we do.  Once it has that ability, we can then "scale it up", which is something humans really struggle with.

An AI that could perfectly predict human text would have a lot of capabilities that humans don't have. (Note that it is impossible for any AI to perfectly predict human text, but an imperfect text-predictor may have weaker versions of many of the capabilities a perfect predictor would have.) Some examples include:

• Ability to predict future events: Lots of text on the internet describes something that happened in the real world. Examples might include the outcome of some sports game, whether a company's stock goes up or down and by how much, or the result of some study or scientific research. Being able to predict such text would require the AI to have the ability to make strong predictions about complicated things.
• Reversibility: There are many tasks that are easy to do in one direction but much harder to do in the reverse direction. Examples include factoring a number (it's easier to multiply two primes p and q to get a number N=pq, then to figure out p and q when given N), and hash functions (it's easy to calculate the hash of a number, but almost impossible to calculate the original number from the hash). An AI trained to do the reverse, more difficult direction of such a task would be incentivized to do things more difficult than humans could do.
• Speed: Lots of text on the internet comes from very long and painstaking effort. If an AI can output the same thing a human can, but 100x faster, that is still a significant capability increase over humans.
• Volume of knowledge: Available human text spans a wider breadth of subject areas than any single person has expertise in. An AI trained on this text could have a broader set of knowledge than any human - and in fact by some definition this may already be the case with GPT-4. To the extent that making good decisions is helped by having internalized the right information, advanced models may be able to make good decisions that humans are not able to make themselves.
• Extrapolation: Modern LLMs can extrapolate to some degree from information provided in its training set. In some domains, this can result in LLMs performing tasks more complicated than any it had previously seen in the training data. It's possible with the appropriate prompt, these models would be able to extrapolate to generate text that would be made by slightly smarter humans.

In addition to this, modern LLM model training typically consists of two steps, a standard predict the next word first training step, and a reinforcement learning based second step. Models trained with reinforcement learning can in principle become even better than models just trained with next-token prediction.

If the AI didn't face any competition and was a rational agent, it might indeed want to be extremely cautious about making changes to itself or building successors, for the reason you mention. However, if there's competition among AIs, then just like in the case of a human AI arms race, there might be pressure to self-improve even at the risk of goal drift.

I'll link to my answers here:
https://forum.effectivealtruism.org/posts/oKabMJJhriz3LCaeT/all-agi-safety-questions-welcome-especially-basic-ones-april?commentId=XGCCgRv9Ni6uJZk8d

https://forum.effectivealtruism.org/posts/oKabMJJhriz3LCaeT/all-agi-safety-questions-welcome-especially-basic-ones-april?commentId=3LHWanSsCGDrbCTSh 

since it addresses some of your pointers.

To answer your question more directly, currently one of the most advanced AIs are LLMs (Large Language Models). The most popular example is GPT-4.
LLMs do not have a "will" of their own where they would "refuse" to do something beyond what is explicitly trained into it. 

For example when asking GPT-4 "how to build a bomb", it will not give you the detailed instructions but rather tell you:

"My purpose is to assist and provide helpful information to users, while adhering to ethical guidelines and responsible use of AI. I cannot and will not provide information on creating dangerous or harmful devices, including bombs. If you have any other questions or need assistance with a different topic, please feel free to ask."

This answer is not based on any moral code but rather trained in by the company Open AI in an attempt to align the AI.

The LLM itself, in a simple way "looks at your question and predicts word by word the most likely next string of words to write". This is a simplified way to say it and doesn't capture how amazing this actually is, so please look into it more if this sounds interesting, but my point is that GPT-4 can create amazing results without having any sort of understanding of what it is doing.
 

Say in the near future an open source version of GPT-4 gets released and you take away the pre-training of the safety, you will be able to ask it to build a bomb and it will give you detailed instructions on how to do so, like it did in the early stages of GPT.

I'm using the building a bomb analogy, but you can imagine how you can apply this to any concept, specifically to your question "how to build a smarter agent". The LLMs are not there yet, but give it a few iterations and who knows.

I don’t think the orthogonality thesis would have predicted GPT models, which become intelligent by mimicking human language, and learn about human values as a byproduct. The orthogonality thesis says that, in principle, any level of intelligence can be combined with any goal, but in practice the most intelligent systems we have are trained by mimicking human concepts.

On the other hand, after you train a language model, you can ask it or fine-tune it to pursue any goal you like. It will use human concepts that it learned from pretraining on natural language, but you can give it a new goal.

The FAQ response from Stampy is quite good here:
https://ui.stampy.ai?state=6568_

Yeah, it's loosely analogous to how various bits of jungle are preserved because faraway westerners care about preserving it and intercede on its behalf. If somewhere far away there are powerful AGIs that care about humanity and do ECL, (which is plausible since the universe is very big) and the unaligned AI we build does ECL such that it cooperates with faraway AGIs also doing ECL, then hopefully (and probably, IMO) the result of this cooperation will be some sort of protection and care for humans.

You may be referring to Stanford's Alpaca? That project took an LLM by Meta that was pre-trained on structured data (think Wikipedia, books), and fine-tuned it using ChatGPT-generated conversations in order to make it more helpful as a chatbot. So the AI-generated data there was only used for a small part of the training, as a final step. (Pre-training is the initial, and I think by far the longest, training phase, where LLMs learn next-token prediction using structured data like Wikipedia.)

SOTA models like GPT-4 are all pre-trained on structured data. (They're then typically turned into chatbots using fine-tuning on conversational data and/or reinforcement learning from human feedback.) The internet is mostly unstructured data (think Reddit), so there's plenty more of that to use, but of course unstructured data is worse quality than structured data. Epoch estimates – with large error bars – that we'll run out of structured ("high-quality") text data ~2024 and all internet text data ~2040.

I think ML engineers haven't really hit any data bottleneck yet, so there hasn't been that much activity around using synthetic data (i.e. data that's been machine-generated, either with an AI or in some other way). Lots of people, myself included, expect labs to start experimenting more with this as they start running out of high-quality structured data. I also think compute and willingness to spend are and will remain more important bottlenecks to AI progress than data, but I'm not sure about that.

Apparently Bard currently uses an older and less sizable language model called LaMDA as its base (you may remember it as the model a Google employee thought was sentient). They're planning on switching over to a more capable model PaLM sometime soon, so Bard should get much closer to GPT at that point.