Why is [robustly pointing at goals] so difficult? Is there an argument that it is impossible?

Well, I hope it's not impossible!  If it is, we're in a pretty bad spot.  But it's definitely true that we don't know how to do it, despite lots of hard work over the last 30+ years.  To really get why this should be, you have to understand how AI training works in a somewhat low-level way.

Suppose we want an image classifier -- something that'll tell us whether a picture has a sheep in it, let's say.  Schematically, here's how we build one:

1. Start with a list of a few million random numbers.  These are our parameters.
2. Find a bunch of images, some with sheep in them and some without (we know which is which because humans labeled them manually).  This is our training data.
3. Pick some images from the training data, multiply them with the parameters in various ways, and interpret the result as a confidence, between 0 and 1, of whether each image has a sheep.
4. Probably it did terribly!  Random numbers don't know anything about sheep.
5. So, we make some small random-ish changes to the parameters and see if that helps.
   1. For example, we might say "we changed this parameter from 0.5 to 0.6 and overall accuracy went from 51.2% to 51.22%, next time we'll go to 0.7 and see if that keeps helping or if it's too high or what."
6. Repeat step 5, a lot.
7. Eventually you get to where the parameters do a good job predicting whether the images have a sheep or not, so you stop.

I'm leaving out some mathematical details, but nothing that changes the overall picture.

All modern AI training works basically this way: start with random numbers, use them to do some task, evaluate their performance, and then tweak the numbers in a way that seems to point toward better performance.

Crucially, we never know why a change to particular parameter is good, just that it is.  Similarly, we never know what the AI is "really" trying to do, just that whatever it's doing helps it do the task -- to classify the images in our training set, for example.  But that doesn't mean that it's doing what we want.  For example, maybe all the pictures of sheep are in big grassy fields, while the non-sheep pictures tend to have more trees, and so what we actually trained was an "are there a lot of trees?" classifier.  This kind of thing happens all the time in machine learning applications.  When people talk about "generalizing out of distribution", this is what they mean: the AI was trained on some data, but will it still perform the way we'd want on other, different data?  Often the answer is no. 

So that's the first big difficulty with setting terminal goals: we can't define the AI's goals directly, we just show it a bunch of examples of the thing we want and hope it learns what they all have in common.  Even after we're done, we have no way to find out what patterns it really found except by experiment, which with superhuman AIs is very dangerous.  There are other difficulties but this post is already rather long.

Part of your question here seems to be, "If we can design a system that understands goals written in natural language, won't it be very unlikely to deviate from what we really wanted when we wrote the goal?" Regarding that point, I'm not an expert, but I'll point to some discussion by experts.

There are, as you may have seen, lists of examples where real AI systems have done things completely different from what their designers were intending. For example, this talk, in the section on Goodhart's law, has a link to such a list. But from what I can tell, those examples never involve the designers specifying goals in natural language. (I'm guessing that specifying goals that way hasn't seemed even faintly possible until recently, so nobody's really tried it?)

Here's a recent paper by academic philosophers that seems supportive of your question. The authors argue that AGI systems that involve large language models would be safer than alternative systems precisely because they could receive goals written in natural language. (See especially the two sections titled "reward misspecification" -- though note also the last paragraph, where they suggest it might be a better idea to avoid goal-directed AI altogether.) If you want more details on whether that suggestion is correct, you might keep an eye on reactions to this paper. There are some comments on the LessWrong post, and I see the paper was submitted for a contest.

I think you're on to something and some related thoughts are a significant part of my research agenda. Here are some references you might find useful (heavily biased towards my own thinking on the subject), numbered by paragraph in your post:

1. There's a lot of cumulated evidence of significant overlap between LM and human linguistic representations, scaling laws of this phenomenon seem favorable and LM embeddings have also been used as a model of shared linguistic space for transmitting thoughts during communication. I interpret this as suggesting outer alignment will likely be solved by default for LMs. 
2. I think I disagree quite strongly that "We don't know how to get an AI system's goals to robustly 'point at' objects like 'the American people' ... [or even] simpler physical systems.", e.g. I suspect many alignment-relevant concepts (like 'Helpful, Harmless, Honest') are abstract and groundable in language, see e.g. Language is more abstract than you think, or, why aren't languages more iconic?. Also, the previous point (brain-LM comparisons), as well as LM performance, suggest the linguistic grounding is probably already happening to a significant degree.
3. Robustness here seems hard, see e.g. these references on shortcuts in in-context learning (ICL) / prompting: https://arxiv.org/abs/2303.03846 https://arxiv.org/abs/2305.17256 https://arxiv.org/abs/2305.13299 https://arxiv.org/abs/2305.14950 https://arxiv.org/abs/2305.19148. An easier / more robust target might be something like 'be helpful'. Though I agree in general ICL as Bayesian inference (see e.g. http://ai.stanford.edu/blog/understanding-incontext/ and follow the citation trail, there are a lot of recent related works) suggests that the longer the prompt, the more likely it would be to 'locate the task'.

This comment will focus on the specific approaches you set out, rather than the high level question, although I'm also interested in seeing comments from others on how difficult it is to solve alignment, and why.

The approach you've set out resembles Coherent Extrapolated Volition (CEV),  which was described earlier by Bostrom. I'm not sure what the consensus is on CEV, but here's a few thoughts which I have in my head from when I thought about CEV (several years ago now).

• How do we choose the correct philosophers and intellectuals -- e.g. would we want Nietsche or Wagner to be on the list of intellectuals, given the (arguable) links to the Nazis? 
• How do we extrapolate? (i.e. how do you determine whether the list of intellectuals would want the action to happen?) 
  • For example, Plato was arguably in favour of dictatorships and preferred them over democracies, but recent history seems to suggest that democracies have fared better than dictatorships -- should we extrapolate that Plato would prefer democracies if he lived today? How do we know?
  • Another example, perhaps a bit closer to home: some philosophers might argue that under some forms of utilitarianism, the ends justify the means, and it is appropriate to steal resources in order to fund activities which are in the best long-term interests of humanity. Even if those philosophers say they don't believe that, they might just be pandering to expectations from society, and the AI might extrapolate that they would say that if unfettered.

In other words, I don't think this does clearly guarantee us against power-seeking behaviour.

The goal you specify in the prompt is not the goal that the AI is acting on when it responds.  Consider: if someone tells you, "Your goal is now [x]", does that change your (terminal) goals?  No, because those don't come from other people telling you things (or other environmental inputs)^[1].

Understanding a goal that's been put into writing, and having that goal, are two very different things.

1. ^{^}

   This is a bit of an exaggeration, because humans don't generally have very coherent goals, and will "discover" new goals or refine existing ones as they learn new things.  But I think it's basically correct to say that there's no straightforward relationship between telling a human to have a goal, and them having it, especially for adults (i.e. a trained model).

I'd suggest this thread (and the linked LW post) as a good overview of the arguments. You could also take a look at the relevant section of the Intro to EA handbook or this post.

In general, I think you'll probably find that you'll get a better response on the forum if you spend some time engaging with the intro materials and come back with specific questions or arguments. 

Maybe frame it more as if you're talking to a child. Yes you can tell the child to follow something but how are you certain that it will do it?

Similarly, how can we trust the AI to actually follow the prompt? To trust it we would fundamentally have to understand the AI or safeguard against problems if we don't understand it. The question then becomes how your prompt is represented in machine language, which is very hard to answer.

To reiterate, ask yourself, how do you know that the AI will do what you say?

I found this post to be astute and lucid. Thanks for writing it.