Thank you, Evan, for the great post!

I've applied to the Charity Entrepreneurship Program and the Founding to Give Program multiple times, and only my third application to the Founding to Give Program's second cohort was successful, so I completed that program at the start of 2026.

The things I want to add:

- Don't be discouraged if your first application doesn't work. Try to get feedback on which specific parts you should work on to improve your chances next year.

- something you mentioned, but I want to emphasize: This mindset you had of "I won't get accepted anyway" is actually a good thing imo. I suspect my first application, especially, was rejected because I was too invested into founding a charity through AIM without giving other options that much consideration. This, I believe, made me very nervous throughout the application process, and even though I got to the work trial stage, if I remember correctly, I felt I didn't do a good job because I experienced something like "exam anxiety". So I put way too much on the line to make this work and stressed myself out too much. My third application was coming from an abundance mindset. I was already working on a project, but I thought that doing the incubator would be a big bonus, but I didn't feel nervous or stressed at all about getting accepted.

- You mentioned flexibility, and I want to be a bit more explicit. The reason I'm not as good of a fit for Charity Entrepreneurship is my lack of willingness to travel. I believe many of the most successful AIM charities, and the ones I myself am most excited to fund, are those where both founders have a strong willingness to spend significant time in the country. I know AIM sometimes says 3 months in the country per co-founder per year is sufficient, but if I could personally craft the ideal CE founder, I would put 6+ months in the country per co-founder. Which is ofc a lot, but I think a good thing to think about when you evaluate if you'd be a good fit for the program

thank you for the references, I'll be sure to check them out!

Since these developments are really bleeding edge I don't know who is really an "expert" I would trust on evaluating it.

The closest to answering your question is maybe this recent article I came across on hackernews, where the comments are often more interesting then the article itself:
https://news.ycombinator.com/item?id=35603756

If you read through the comments which mostly come from people that follow the field for a while they seem to agree that it's not just "scaling up the existing model we have now", mainly because of cost reasons, but that's it's going to be doing things more efficiently than now. I don't have enough knowledge to say how difficult this is, if those different methods will need to be something entirely new or if it's just a matter of trying what is already there and combining it with what we have.

The article itself can be seen skeptical, because there are tons of reasons OpenAIs CEO has to issue a public statement and I wouldn't take anything in there at face value. But the comments are maybe a bit more trustworthy / perspective giving.

Thanks a lot for transcribing this, was a great read!

Small nitpick think there is a word missing here:
> "which seems perhaps in itself" (bad?)

Yea big companies wouldn't really use the website service, I was more thinking of non technical 1 man shops, things like restaurants and similar.

Agree that governments definitely will try to counter it, but it's a cat and mouse game I don't really like to explore, sometimes the government wins and catches the terrorists before any damage gets done, but sometimes the terrorists manage to get through. Right not getting through often means several people dead because right now a terrorist can only do so much damage, but with more powerful tools they can do a lot more damage.

I'd argue that the implementation of the solution is work and a customer would be inclined to pay for this extra work.

For example right now GPT-4 can write you the code for a website, but you still need to deploy the server, buy a domain and put the code on the server. I can very well see an "end to end" solution provided by a company that directly does all these steps for you.

In the same way I very well see commercial incentive to provide customers with an AI where they can e.g. upload their codebase and then say, based on our codebase, please write us a new feature with the following specs.

Of course the company offering this doesn't intent that their tool where a company can upload their codebase to develop a feature get's used by some terrorist organisation. That terrorist organisation uploads a ton of virus code to the model and says, please develop something similar that's new and bypasses current malware detection.

I can even see there being no oversight, because of course companies would be hesitant to upload their codebase if anyone could just view what they're uploading, probably the data you upload is encrypted and therefor there is oversight.

I can see there being regulation for it, but at least currently regulators are really far behind the tech. Also this is just one example I can think of and it's related to a field I'm familiar with, there might be a lot of other even more plausible / scarier examples in fields I'm not as familiar with like biology, nano-technology, pharmaceuticals you name it.

Maybe to explain a bit more in detail what I meant with the example of hallucinating, rather than showcasing it's limitation it's showcasing it's lack of understanding.

For example if you ask a human something and they're honest about it, if they don't know something they will not make something up but just tell you the information they have and beyond that they don't know.

While in the hallucinating case the AI doesn't say that it doesn't know something, which it often does btw, but it doesn't understand that it doesn't know and just comes up with something "random".

So I meant to say that it hallucinating is showcasing it's lack of understanding.

I have to say though that I can't be sure why it hallucinates really, it's just my likely guess. Also for creativity there is some that you can do with prompt engineering but indeed at the end you're limited by the training data + the max tokens that you can input where it can learn context from.

Loved the language in the post! To the point without having to use unnecessary jargon.

There are two things I'd like you to elaborate on if possible:

> "the challenge is getting AIs to do what it says on the tin—to reliably do whatever a human operator tells them to do."

If I understand correctly you imply that there is still a human operator to a superhuman AGI, do you think this is the way that alignment will work out? What I see is that humans have flaws, do we really want to give a "genie" / extremely powerful tool to humans that even already struggle with the powerful tools that they have? At least right now these powerful tools are in the hands of the more responsible few, but if it becomes more widely accessible that's very different.

What do you think of going the direction of developing a "Guardian AI", which would still solve the alignment problem using the tools of ML, but involving humans giving up control of the alignment?

The second one is more practical, which action do you think one should take. I've of course read the recommendations that other people have put out there so far, but would be curious to hear your take on this. 
 

From my current understanding of LLMs they do not have the capability to reason or have a will as of now. I know there are plans to see if with specific build in prompts this can be made possible, but the way the models are build at the moment is that they do not have an understanding of what they are writing.

Aside from my understanding of the underlying workings of GPT-4, an example that illustrates this, is that sometimes if you ask GPT-4 questions that it doesn't know the precise answer to, it will "hallucinate", meaning it will give a confident answer that is factually incorrect / not based on it's training data. It doesn't "understand" your question, it is trained on a lot of text and based on the text you give it, it generates some other text that is likely a good response, to say it really simplified.

You could make an argument that even the people at OpenAI don't truly know why GPT-4 gives the answers that it does, since it's pretty much a black box that is trained on a preset of data and then OpenAI adds some human feedback, to quote from their website:

> So when prompted with a question, the base model can respond in a wide variety of ways that might be far from a user’s intent. To align it with the user’s intent within guardrails, we fine-tune the model’s behavior using reinforcement learning with human feedback (RLHF). 

So as of now if I get your question right there is no evidence that I'm aware of that would point towards these LLMs "applying" anything, they are totally reliant on the input they are given and don't learn significantly beyond their training data.

The reasons you provide would already be sufficient for me to think that AI safety will not be an easy problem to solve. To add one more example to your list:

We don't know yet if LLMs will be the technology that will reach AGI, it could also be a number of other technologies that just like LLMs make a certain breakthrough and then suddenly become very capable. So just looking at what we see develop now and extrapolating from the currently most advanced model is quite risky.

For the second part about your concern about the welfare of AIs themselves, I think this is something very hard for us to imagine, we anthropomorphize AI, so words like 'exploit' or 'abuse' make sense in a human context where beings experience pain and emotions, but in the context of AI those might just not apply. But I would say in this area I still know very little so I'm mainly repeating what I read is a common mistake to make when judging morality in regards to AI.