I upvoted this because AI-related advocacy has become a recent focus of mine. My background is from organising climate protests, and I think EAs have a bit of a blindspot when it comes to valuing advocacy. So it's good to have this discussion. However, I do disagree on a few points.

1. Just Ask: In broad strokes, I think people tend to overestimate exactly how unreasonable and persistent initial objections will be. My simplest rebuttal would be: How do you know these advocates would even disagree with your approach? An approach I'm considering now is to find a decent AI Governance policy proposal, present it to the advocates explaining how it solves their problem and see who says yes. If half of them say no, you work with the other half. Before assuming the "neo-Luddites" won't listen to reason, shouldn't you ... ask? Present them with options? I don't see why it's not at least worth reaching out to potential allies, and I don't see why it's an irredeemable sin to be angry at something with no clear solutions, when no one has presented a solution. It's perhaps ironic the assumptions given here.

2. Counterfactuals I think by most estimates, anti-AI advocacy only grows from here. Having a lot of structurally unemployed angry people is historically a recipe for trouble. You then have to consider that reactionary responses will happen regardless of whether "we align with them". If they are as persistently unreasonable as you say they are, they will force bad policy regardless. They will influence mainstream discourse towards their views, and be loud enough to crowd out our "more reasonable" views. I just think it makes a lot of sense to engage these groups early on, and make an earnest effort to make our case. Because the counterfactual is that they get bad policies passed without our input.

3. False dichotomy of advocates and researchers I speak more generally here. In my time in climate risk, everyone had an odd fixation on separating climate advocates and researchers.^[1] I don't think this split was helpful for epistemics or strategy overall. Because then you had scientists who had all the solutions and epistemics that the public/policymakers generally ignored out of lack of engagement, and the advocates who started latching onto poorly-informed and counterproductive radical agendas, and were constantly rebutted with "why are we listening to you clueless youngsters and not the scientists (who we ignore anyway)". It was just a constant headache to have two subgroups needlessly divide themselves while the clock ran down. Like sure, the advocates were ... not the most epistemically rigorous. And the scientists generally struggled to put across their concerns. But I'd greatly prefer if everyone valued more communication/coordination, and not less.

And for my sanity's sake, I'd like the AI risk community to not repeat this dynamic.

1. ^{^}

   I suspect most of this dichotomy was not made in good faith, but simply by people uncomfortable with the premise of anthropogenic climate change and throwing out fallacies to discredit any arguments they're confronted with in their daily lives.

...their regulations will probably not, except by coincidence, be the type of regulations we should try to install.

A priori, I'd expect a randomly formulated AI regulation to be about 50% likely to be an improvement on the status quo, since the status quo wasn't selected for being good for alignment.

Adopting the wrong AI regulations could lock us into a suboptimal regime that may be difficult or impossible to leave.

I don't see good arguments supporting this point. I tend to think the opposite -- building a coalition to pass a regulation now makes it easier to pass other regulations later.

arbitrary data restrictions risk preventing researchers from having access to good data that might help with alignment

OpenAI claims that ChatGPT is an "alignment advance". I think this is way off -- the approach they're using just isn't good enough in principle. Incrementally improving on ChatGPT's "alignment" the way OpenAI is doing leads to disaster, IMO. You don't write the code for a nuclear reactor through trial and error.

If an alignment scheme doesn't work with arbitrary data restrictions, it's probably not good enough in principle. Even all the data on the internet is "arbitrarily restricted" relative to all the data that exists or could exist. If my alignment scheme fails with public domain data only, why shouldn't it also fail with all the data on the internet? (And if an alignment scheme works on public domain data only, it should be possible to add in more data to boost performance before using the AI for something mission-critical.)

I think a better argument for your conclusion is that incentivizing researchers to move away from big data approaches might make AI research more accessible and harder to control. Legal restrictions also favor open source, which has the same effect. We don't want cutting-edge AI to become something that exists on the margins, like using bittorrent for piracy.

I suspect the right compromise is for AI art companies to pay a license fee to any artist in their training data who signs up to be paid. We want to reduce company profits and decrease AI progress incentives without pushing things towards open source a bunch. I'd take the same approach for language models like GPT. I think our interests are actually fairly well-aligned with artists here, in the sense that the approach which allocates profits away from AI companies and towards artists is probably also the best approach for humanity's long-run future.

A friend in AI Governance just shared this post with me.

I was blunt in my response, which I will share below:

~ ~ ~

Two cruxes for this post:

1. Is aligning AGI to be long-term safe even slightly possible – practically given default AI scaled training and deployment trends and complexity of the problem (see Yudkowsky’s list of AGI lethalities) or theoretically given strict controllability limits (Yampolskiy) and uncontrollable substrate-needs convergence (Landry).

If clearly, pre-aligning AGI to not cause a mass extinction is not even slightly possible, then IMO splitting hairs about “access to good data that might help with alignment” is counterproductive.

2. Is a “richer technological world” worth the extent to which corporations are going to automate away our ability to make our own choices (starting with our own data), the increasing destabilisation of society, and the toxic environmental effects of automating technological growth?

These are essentially rhetorical questions, but covers the points I would ask someone who proposes desisting from collaborating with other groups who notice related harms and risks of corporations scaling AI.

To be honest, the reasoning in this post seems rather motivated without examination of underlying premises.

These sentences particularly:

“A world that restricts compute will end up with different AGI than a world that restricts data. While some constraints are out of our control — such as the difficulty of finding certain algorithms — other constraints aren't. Therefore, it's critical that we craft these constraints carefully, to ensure the trajectory of AI development goes well. Passing subpar regulations now — the type of regulations not explicitly designed to provide favorable differential technological progress — might lock us into bad regime.”

It assumes AGI is inevitable, and therefore we should be picky about how we constrain developments towards AGI.

It also implicitly assumes that continued corporate scaling of AI counts as positive “progress” – at least for the kind of world they imagine would result and want to live in.

The tone also comes across as uncharitable. As if they are talking down at others they have not spent time trying to listen carefully to, take the perspective of, and paraphrase back their reasoning to (at least nothing is written about/from those attempts in the post).

Frankly, we cannot be held back by motivated techno-utopian arguments from taking collective action against exponentially increasing harms and risks (in extents of the scale and local impacts). We need to work with other groups to make traction.

~ ~ ~

You wrote

Earlier this month, digital artists staged a mass protest against AI art on ArtStation. A few people are reportedly already getting together to hire a lobbyist to advocate more restrictive IP laws around AI generated content. And anecdotally, I've seen numerous large threads on Twitter in which people criticize the users and creators of AI art.

and

Personally, this sentiment disappoints me. While I sympathize with the artists who will lose their income, I'm not persuaded by the general argument. The value we could get from nearly free, personalized entertainment would be truly massive. In my opinion, it would be a shame if humanity never allowed that value to be unlocked, or restricted its proliferation severely.

and

it is not worth sacrificing a technologically richer world just to protect workers from losing their income.

Are you arguing from principle here?

Artists' (the workers') work is being imitated by the AI tools, so cost-effectively that an artist's contributions, once public, render the artists' continuing work unnecessary to produce work with their style.

Is the addition of technology T with capability C that removes need for worker W with job role R and capability C more important than loss of income I to worker W, for all T, C, W, R, and I?

Examples of capabilities could be:

• summarizing existing research work (for example, an AI Safety paper)
• collecting typical data used to make predictions (for example, cost and power of compute)
• monitoring new research work (for example, recent publications and their relationships, such as supporting, building on or contradicting)
• hypothesizing about preconditions for new developments (for example, conditions suitable for AGI development)
• developing new theories or models (for example, of AI Safety)
• testing new theories or models (for example, of AI Safety)

Loss of income could be:

• partial (for example, a reduction in grant money for AI Safety workers as those funds are diverted to automation projects with 10-20 year timelines)
• complete (for example, replacement of 50 AI Safety workers with 10 workers that rely on semi-automated research tools)

The money allocated to workers could be spent on technology instead.

Investments in specific technologies T1, T2 with capabilities C1, C2 can start with crowd-sourcing from workers W1, W2,..., Wk, and more formal automation and annotation projects targeting knowledge K developed by workers Wk+1, ..., Wn (for example, AI Safety researchers) who do not participate in the crowd-sourcing and automation effort but whose work is accessible.

You repeatedly referred to "we" as in:

True, if very powerful AI is coming very soon (<5 years from now), there might not be much else we can do except for aligning with vaguely friendly groups, and helping them pass poorly designed regulations.

However, a consequence of automation technology is that it removes the political power (both money and responsibility) that accrued to the workers that it replaces. For example, any worker in the field of AI Safety, to the extent that her job depends on her productivity and cost-effectiveness, will lose both her income and status as the field progresses to include automation technology that can replace her capabilities. Even ad hoc automation methods (for example, writing software that monitors cost and power of compute using web-scraping and publicly available data) remove a bit of that status. In that way, the AI Safety researcher loses status among her peers and her influence on policy that her peers direct. The only power left to the researcher is as an ordinary voter in a democracy.

Dividing up and replacing the responsibilities for the capabilities Ci of an individual W1 can help an ad hoc approach involving technologies Ti corresponding to the capabilities of that worker. Reducing the association of the role with the status can dissolve the role and sometimes the worker's job who held that role. The role itself can disappear from the marketplace, along with the interests that it represents. For example, although artists have invested many years in their own talents, skills, and style, within a year they lost their status and income to some new AI software. I think artists have cause to defend their work from AI. The artist role won't disappear from the world of human employment entirely but the future of the role has been drastically reduced and has permanently lost a lot of what gave it social significance and financial attractiveness, unless the neo-luddites can defend paid employment in art from AI.

Something similar can happen to AI Safety researchers, but will anyone object? AI Safety researcher worker capabilities and roles could be divided and dissolved into larger job roles held by fewer people with different titles, responsibilities, and allegiances over time as the output of the field is turned into a small, targeted knowledge-base and suite of tools for various purposes.

If you are in fact arguing from principle, then you have an opportunity to streamline the process of AI safety research work through efforts such as:

• collecting AI Safety research work on an ongoing basis as it appears in different venues and making it publicly accessible
• annotating the research work to speed up queries for common questions such as:
  • what are controversies in the field, that is, who disagrees with whom about what and why?
  • what is the timeline of development of research work?
  • what literature address specific research questions (for example, on compute developments, alternative technologies, alignment approaches, specific hypotheses in the field, prediction timelines)?
  • what are summaries of current work?
• paying for public hosting of AI Safety information of this type as well as ad hoc tools (for example, the compute power tracker)

I'm sure you could come up with better ideas to remove AI Safety worker grant money from those workers, and commensurately benefit the cost-effectiveness of AI Safety research. I've read repeatedly that the field needs workers and timely answers, automation seems like a requirement or alternative to reduce the financial and time constraints on the field but also to serve its purpose effectively.

While artists could complain that AI art does a disservice to their craft and reduces the quality of art produced, I think the tools imitating those artists have developed to the point that they serve the purpose and artists know it and so does the marketplace. If AI Safety researchers are in a position to hold their jobs a little while longer, then they can assist the automation effort to end the role of AI Safety researchers and move on to other work that much sooner! I see no reason to hold you back from applying the principle that you seem to hold, though I don't hold it myself.

AI Safety research is a field that will hopefully succeed quickly and end the need for itself within a few decades. It's workers can move on, presumably to newer and better things. New researchers in the field can participate in automation efforts and then find work in related fields, either in software automation elsewhere or other areas such as service work for which consumers still prefer a human being. Supposedly the rapid deployment of AGI in business will grow our economies relentlessly and at a huge pace, so there should be employment opportunities available (or free money from somewhere).

If any workers have a reason to avoid neo-ludditism, it would have to be AI Safety researchers, given their belief in a future of wealth, opportunity, and leisure that AI help produce. Their own unemployment would be just a blip of however long before the future they helped manifest rescues them. Or they can always find other work, right? After all they work on the very technology depriving others of work. A perfectly self-interested perspective from which to decide whether neo-ludditism is a good idea for themselves.

EDIT: sorry, I spent an hour editing this to convey my own sense of optimism and include a level of detail suitable for communicating the subtle nuances I felt deserved inclusion in a custom-crafted post of this sort. I suppose chatGPT could have done better? Or perhaps a text processing tool and some text templates would have sped this up. Hopefully you find these comments edifying in some way.

I agree, and I do not think that slowing down AI or speeding it up is desirable, for reasons related to Rohin Shah's comment on LW about why slowing down AI progress is not desirable:

1. It makes it easier for a future misaligned AI to take over by increasing overhangs, both via compute progress and algorithmic efficiency progress. (This is basically the same sort of argument as "Every 18 months, the minimum IQ necessary to destroy the world drops by one point.")

2. Such strategies are likely to disproportionately penalize safety-conscious actors.

(As a concrete example of (2), if you build public support, maybe the public calls for compute restrictions on AGI companies and this ends up binding the companies with AGI safety teams but not the various AI companies that are skeptical of “AGI” and “AI x-risk” and say they are just building powerful AI tools without calling it AGI.)

For me personally there's a third reason, which is that (to first approximation) I have a limited amount of resources and it seems better to spend that on the "use good alignment techniques" plan rather than the "try to not build AGI" plan. But that's specific to me.

While it's understandable to want to take action and implement some form of regulation in the face of rapidly advancing technology, it's crucial to ensure that these regulations are effective and aligned with our ultimate objectives.

It's possible that regulations proposed by neo-Luddites could have unintended consequences or even be counterproductive to our goals. For example, they may focus on slowing down AI progress in general, without necessarily addressing specific concerns about AI x-risk. Doing so could drive cutting-edge AI research into the black market or autocratic countries. It's important to carefully evaluate the motivations and objectives behind different regulatory proposals and ensure that they don't end up doing more harm than good. 

Personally, I'd rather have a world with 200 mostly-positively-aligned research organizations than a world where only autocratic regimes and experienced coding teams - that are willing to disregard the law - can push the frontiers of AI.

While I sympathize with the artists who will lose their income, I'm not persuaded by the general argument. The value we could get from nearly free, personalized entertainment would be truly massive. In my opinion, it would be a shame if humanity never allowed that value to be unlocked, or restricted its proliferation severely.

This seems like an area in which utilitarianism should be bounded by deontology.^[1] The reasoning here seems to be roughly "the value of this scenario is so high that I am okay with the harm it causes."

There are options other than artists loose their income by having their work pirated/stolen, copied, and altered, and humanity never unlocks that value. For example, by paying artists for their work.

1. ^{^}

   Vaguely parallel to how democracy should be bounded by liberalism: 9 people of group A voting to remove the rights of 1 group B person is democratic, which is the simplistic example of how the liberalism of protecting rights is important.