Elias -- good post, and it's worth being aware of these potential downsides.

To me, the most significant of these is the risk of legitimate AI safety efforts being co-opted into 'AI safety-washing' by companies such an Anthropic -- which systematically recruit naive young EAs to do their 'safety work', but which ends up being little more than corporate virtue-signaling. 

Joshua -- an important and thought-provoking piece.

Do you have a sense of which kinds of crimes are most likely to involve AI agents hiring human tasker or 'innocent agents'? 

My hunch is that the most typical situation might involve a human giving a vague goal to an AI agent (e.g. 'figure out how to increase the amount of bitcoin in my crypto account'), and the agent developing some strategy that happens to be illegal, and (possibly) hiring some human taskers to help. 

Rather than, the AI agent going off on its own, running around and committing crimes, just because 'instrumental convergence' principle says that amassing money and power is likely to be useful.

If the typical situation (vague human prompts --> AI hatching felonious plans --> using human taskers) is valid, then the human originating the 'felony prompt' may have strong incentives to create plausible deniability. 

In any case, I think you're correct that the legal & criminal justice systems need to start thinking this through ASAP.

Hi Tristan -- fair points.

Indeed, my post was somewhat inspired by my wife Diana Fleischman's discussion back in 2013 of whether it's OK for ethical vegans to eat shellfish, given the uncertainty about whether their little ganglia carry any sentience; see link here.

My hunch as an evolutionary psychologist is that, given the extremely strong selection pressures on shellfish to resist being eaten by starfish, and their apparent use of all possible muscular effort and endurance to keep their shells closed when being attacked by starfish, if shellfish are sentient about anything, they're most likely to be sentient about resisting starfish attacks, and being motivated to treat them as a negative experience.

TBH, I may have also been biased against starfish by the scene in 'Suicide Squad' (2021) of the superheroes fighting 'Starro the Conqueror', the giant alien kaiju starfish.

Thanks, appreciate it! 

I've always loved the back-and-forth give-and-take of EA discourse, and the quantification mindset.

Aaron - thanks for sharing a poetic and moving story.

There is a dark downside to this narrative.

Starfish are predators that eat mostly shellfish (mussels, clams, oysters) -- typically 3-5 times their body weight per day, meaning they're consuming 2-8 shellfish a day. They use suction-cupped feet to pull the shell apart until the shellfish tires out, and a tiny gap opens. Then the starfish extrude their 'cardiac stomach' out of their mouth, into the shell, and uses various enzymes (proteases, lipases, and amylases) to digest the shellfish's soft body, which can take a couple of hours. I imagine this is not a happy time for the shellfish, insofar as they might be sentient.

Starfish live 5-30 years, averaging maybe 10 years. So, during one lifetime -- assuming 10 year lifespan x 365 days/year x 5 shellfish/day -- one starfish may be eating about 18,000 shellfish. Each of which dies after a long, exhausting struggle to keep its shell closed, and then being gradually dissolved by enzymes over a couple of hours. 

If shellfish are sentient, that's a significant amount of suffering each starfish may be imposing.

So, for each starfish a girl 'saves' (and assuming she's saving it about halfway through its life), we may be condemning about 9,000 shellfish to an prolonged, excruciating death. 

Is the starfish-saving girl a hero, or a villain? 

Does it depend on whether a starfish is 9,000 times more sentient than a shellfish? 

I don't know. But. when analyzing how interventions affect total 'wild animal suffering', in complex ecosystems, we have to be careful about which victims we may be overlooking.

Sophie -- thanks for your reply. I agree. And I strongly upvoted your more recent post that you mentioned, which is excellent.

Sophie - thanks for an excellent, important, and timely article. 

My main concern here is that a huge influx of 'technical AI safety research' money from beneficiaries of an Anthropic IPO would be very biased in favor of pro-AI corporate safety-washing, rather than anti-AI activism. 

Funders dispersing post-IPO Anthropic founder/employee funds would be under heavy pressure to buy into what we might call the 'Anthropic Model of AI safety', which seems to have a few assumptions that are often implicit, but that are IMHO very misguided and very dangerous, e.g. the assumptions that

1. ASI (Artificial Superintelligence) alignment is a solvable problem, at least in principle -- and indeed, is likely so be solved within a few years, if we can just throw enough talent and money at the problem;
2. ASI alignment is more likely to be solved the more we advanced AI capabilities, because that gives us more insight into how advanced AI systems work (rather than, by example, funding more research on the game theory issues around ASI vs human goals, incentives, and payoffs, or more research on characterizing the nature and diversity of human morals, values, and goals that ASI would be trying to align with);
3. extinction risks (e.g. as indexed by p(doom)) can be reduced most effectively by accelerating 'technical AI safety research', rather than by anti-AI activism, by severe gov't regulations and global treaters, and by moral stigmatization of AI founders, researchers, and devs.

If the vast majority of AI safety work and advocacy comes from post-IPO Anthropic money, the likely result is that the entire EA focus on AI safety will get warped into chasing 'technical AI safety' jobs and money, rather than fighting the AI industry at the grassroots political level, or policy level, or public moral level. The people like me who see Anthropic as just as reckless and evil as OpenAI, Google, xAI, or DeepSeek, will get squeezed out. People who think that anyone participating in the 'AI arms race' is basically endangering our kids for their own greed and hubris will get silenced.  

This post-IPO Anthropic money would not get spent on promoting a bipartisan political consensus within the US to shut down all further ASI development. It would not get spent on raising public awareness of AI risks in China, among CCP leaders, influencers, and citizens. Rather, it would get spent on hiring armies of 'technical AI safety' grant-recipients to work within or alongside AI companies. 

Those 'AI safety researchers' will not rock the boat of the AI industry. They will not advocate for the kinds of pauses or shutdowns advocated by Pause AI, Stop AI, or Control AI. They will be easily co-opted by trillion-dollar AI companies into being their safety-washing minions, their PR representatives, and their reassurances to the public that 'the AI industry is taking safety very seriously indeed', while it races ahead towards ASI.

That's my concern. If the 'Anthropic Model of AI safety' is simply wrong in major ways -- e.g. if 'ASI alignment' is not a solvable problem, if ASI alignment isn't best solved by advancing AI capabilities, and if 'technical AI safety research' actually increases p(doom) (e.g. as safety-washing that gives misleading comfort to legislators, citizens, and other nations), then this Anthropic IPO, and resulting distortions of AI safety efforts, could be a disastrous development.

You do realize, I hope, that this all sounds wildly speculative to anyone who works in biomedical research? 

It builds assumption on top of assumption. 

Basically you're saying 'trust us, ASI can do ANYTHING it needs to do to gather ALL the data it needs, by any means necessary, to solve all diseases quickly, reliably, with no side-effects, no tradeoffs, and no catastrophic tragedies that would turn public opinion against the whole enterprise'. 

That is not a compelling argument to me at all, and I think its implausibility undercuts the common talking point among e/accs and pro-AI lobbyists that 'ASI would cure death quickly and easily'

I'm extremely skeptical that it would be possible to 'simulate biology from first principles' in any computationally feasible way, given the hierarchical complexity of biology across huge range of scales -- both spatial (from biomolecules to organelles to cells, tissues, organs, and organism) and temporal (from femtoseconds to decades). We just don't have any 'first principles' in biology that are analogous to physical laws that could be used to simulate planet formation or weather. 

We also don't have the data required to 'run trillions of lab-on-a-chip experiments'.

And I cannot imagine any situations in which 'nano bots that could remove cancer cells' could be deployed in living humans without the first several thousands patients dying in surprising and gruesome ways. 

I just don't see any of these suggestions connecting to the reality of human biomedical research.