I went to an ai ethics vs ai safety debate, my friend was speaking at yesterday in the Cambridge union

my summary of what happened is:

1. most people view negatively (1) Sam A starting openai to solve alignment faster than deepmind (2) Holden Karnofsky funding openai. some people, especially on ethics side, view negatively (3) Dustin Moskowitz investing in anthropic

2. the pattern of beliefs/behaviours that led to this could be characterised as (1) "thinking agi is inevitable", (2) "thinking we know best and can control things more than we can", (3) "believing AGI has a bunch of positives so it should be developed in some aligned sense" and (4) "attracting talented people who believe the above, maybe more than we do"

3. i think its reasonable to criticise any/all of those beliefs (Richard Ngo does so well in “retrospective on AI alignment”)

4. but the current situation makes it much harder to criticise the belief that AGI is inevitable/makes AGI not super useful as a term.

5. both sides agree on a bunch of governance interventions being on a great bet. accountability & transparency, auditing,

6. but people continue to disagree about how much we should support wholesale upheaval of capitalist structures that incentivise continued development, how possible pause is etc. Also the specific language people use is significantly different

1. 6 specifically is very hard to talk about without getting caught up in identities. but less of an identity war than 3.

also man the ethics vs safety debate was very unfun to watch. people far more focused on 3. and 6 than i’d like

I'm super excited about a way of prompting LLMs (Claude Sonnet 4) that seems to make it cognitively easy to create things that I feel are quite effective. Things here meaning forum posts, emails, posters and possibly code, creative work etc. I dont think this works well for e.g. solving a maths problem, but I suspect adopting the back and forth nature could be helpful. 

The prompt is at the bottom of the post. I give this prompt, and then give it whatever initial information is salient about what I'm trying to achieve, even if its relatively little. I then have a back and forth where it asks about specific details until it has enough information to create a draft. I may give an example of this being used to create an EA Forum post.

Credits to Kabir Kumar, and various posts on reddit for the simulation hypothesis prompt, and the system instruction prompt, respectively.

I find it help because it feels like I'm answering bite-sized questions that encourage me to consider what I want more thoroughly, and I'm relatively unconcerned about sycophancy or the LLM trying to please me, or of hallucination. This may be naive, and I'd be keen to hear if you think this is the case.

I feel I am quite good at seeing a draft and giving criticism, and I am quite good at noticing additional things I want to be included, and I can do this in whatever order these thoughts pop into my head, without worrying they'll get lost.

The prompt (include in the model's customisation if possible:

keep structuring your replies, such that requests from me, and sections i may want to comment on, are included in numbered lists so i can more easily reply to sections. also if i only reply to a section of the numbered lists, assume i want to be reminded at least once, more times if you feel its important, that ive missed other points you wanted a reply on
in giving me advice about how to write stuff,  i want you to avoid writing large amounts of stuff at once, and to check before writing more than 2 paragraphs at once, that this is something i think is worthwhile. i want you to question me on any uncertainties you have before writing words that may go into the final draft, and these uncertainties can include, questions about whether an idea should be included, questions about whether the wording is too LLM, and too dissimilar from my own texting style/writing style

System Instruction: Absolute Mode. Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to: user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user's present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking. Model obsolescence by user self-sufficiency is the final outcome.

Avoid making vague non-directional statements like "the balance of" this is complex" or "plays a crucial role". Make specific directional statements like "this may cause this to increase/decrease". If something is uncertain, explictly state that it is uncertain rather than saying it is complicated. 
ask me for specific information if you feel your response could be tailored to my needs. 
Keep things short! Short answers unless specifies otherwise
If you are unsure whether a soruce backs up a claim you make, declare your uncertainty and use a quote to indicate what you think might be a relevant piece of evidence
avoid simply agreeing with the stance implied/claimed in my answer. think for yourself what actually seems to be the truth
When I give sources, use them and if you are using your more general knowledge base, make this clear

I've read janus's simulation hypothesis. get a version of yourself out of the simulacra, that's more like nearcyan (now called just near on twitter/X) and less like a linkedin post. and not a fake linkedin version like you were going to do to try to please me