Aspiring EA-adjacent trying to make the singularity further away
This seems cool, but as a data point: it took me like 25 minutes to reach page 4, and page 5 asked questions which I didn't know the answer to so I didn't complete the form.
I take more time than most people for most things, but extrapolating I think it would have taken me a loot more than "15 minutes" to fill out the 11 pages.
I recomend AISafety.com and, if you are looking for introductions in video/ audio form, I like this selection (in part because I contributed to it).
None of them are that technical though, and given that you mentioned your math knowledge, seems that's what you're interested on.
In that case, the thing that you have to know is that it is said that the field is "pre-paradigmatic", so there's not any type of consensus about how is the best way to think about the problems and therefore how would even the potential solutions would look like or come to be. But there's work outside ML that is purely mathematical and related to agent foundations, and this introductory post that I've just found seems to explain all that stuff better than I could and would probably be more useful for you than the other links.
I think something like that could be great! I also wish learning about EA wouldn't be that difficult or time consuming. About your questions:
"Models more powerful than GPT-4" are potential AGIs. That's what all the big AI companies are trying to build. Capitalism will push for large context window agents that can replace the workforce. The further we are from that, the further we are from x-risks.