By Scott Garrabrant et al:

• Logical induction
• Cartesian frames
• Finite factored sets

By John Wentworth

By myself:

• Infra-Bayesianism (collaboration with Alexander Appel)
  • In particular, infra-Bayesian physicalism
• RL with imperceptible rewards
• Quantilization (building on work by Jessica Taylor)
• Forecasting using incomplete models (related to logical induction and infra-Bayesianism)
• Various topics that don't have sufficiently good articles yet, such as 1 2 3

(Told this to Jenny in person, but posting for the benefit of others)

AI safety is a young, pre-paradigmatic area of research without a universally accepted mathematical formalism, so if you're after cool math, my suggestion is to learn the basics of one or two well-established fields that are mathematically mature and have a decent chance of being relevant to AI safety.

In particular, I think Learning Theory and Causality are areas with plenty of Aesthetic Math™.

Learning theory

Statistical learning theory is the mathematical study of inductive reasoning—how can we make generalizations from past observations to future observations? It's an entire mathematically rich field devoted to formalizing Occam's razor.

Computational learning theory imposes the further restriction that learning algorithms be computationally efficient. It has rich connections to other parts of theoretical computer science (for example, there is a duality between computational learning theory and cryptography—positive results for one translate to negative results for the other!) And there are many fun problems of a combinatorial puzzle flavor.

Most of learning theory assumes that observations are drawn i.i.d. from a distribution. Online Learning asks what happens if we eliminate this assumption. Incredibly, it can be shown that inductive reasoning can be successful even when observations are handcrafted by an adversary. The key is to measure success in relative rather than absolute terms: how did you perform in comparison to the best member of a pre-specified class of predictors? There are beautiful connections to convex analysis.

Readings:

• An Introduction to Computational Learning Theory, by Kearns & Vazirani. A well-written, approachable introductory textbook with fun (to me) exercises. It's somewhat outdated, but still the best learning theory text for a beginner imo.
• Understanding Machine Learning: From Theory to Algorithms, by Shalev-Shwartz & Ben-David. A more up-to-date and comprehensive textbook on learning theory.
• Online Learning & Online Convex Optimization by Shalev-Shwartz. An excellent unified treatment of the mathematics of online learning.

Causality

I don't know this area as well, but the material I have learned has been mathematically beautiful. In particular, I suggest learning about Judea Pearl's theory of causality, which has been very influential in computer science, statistics, and some of the natural and social sciences. (There are a few competing formalisms for causality, but Pearl's is the most mathematically beautiful as far as I can tell.) Pearl's theory generalizes the classical theory of probability to allow for reasoning about cause and effect, using a framework that involves manipulations of directed acyclic graphs.

Reading: Causality, by Pearl.

A handful of ideas (things that tickle my aesthetic) from an ex-topologist:

https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign

https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy/

https://ai-alignment.com/corrigibility-3039e668638 (and other things from https://ai-alignment.com/ )

I'm did a pure maths undergrad and recently switched to doing mechanistic interpretability work - my day job isn't exactly doing maths, but I find it has a strong aesthetic appeal in a similar way. My job is not to train an ML model (with all the mess and frustration that involves), it's to take a model someone else has trained, and try to rigorously understand what is going on with it. I want to take some behaviour I know it's capable of and understand how it does that, and ideally try to decompile the operations it's running into something human understandable. And, fundamentally, a neural network is just a stack of matrix multiplications. So I'm trying to build tools and lenses for analysing this stack of matrices, and converting it into something understandable. Day-to-day, this looks like having ideas for experiments, writing code and running them, getting feedback and iterating, but I've found a handful of times where having good intuitions around linear algebra, or how gradients work, and spending some time working through algebra has been really useful and clarifying. 

If you're interested in learning more, Zoom In is a good overview of a particular agenda for mechanistic interpretability in vision models (which I personally find super inspiring!), and my team wrote a pretty mathsy paper giving a framework to breakdown and understand small, attention-only transformers (I expect the paper to only make sense after reading an overview of autoregressive transformers like this one). If you're interested in working on this, there are currently teams at Anthropic, Redwood Research, DeepMind and Conjecture doing work along these lines!

Not sure what mathematically interests you, but you should probably check out Vanessa Kosoy's learning-theoretic research agenda (she is hiring mathematicians!). Also, the Topos Institute are doing many interesting things in AI safety and other things (I'm personally particularly interested in their compositionality/modeling work, which seems very cool to me).

A couple of unasked-for pieces of advice that may be relevant (would be for my past self who was sort of in a similar position):

1. Sadly, many times we should expect tradeoffs between impact and interest, where to actually implement innovations requires doing hard manual work. Especially in academic fields, where the impactful uninteresting work is more neglected. 
2. Our interests change quite a bit over time, and it's usually hard to predict how it might change. That said, for many people they find stuff more interesting the more competence they feel at it and the more they care about the problem they try to solve or about the product they intend to deliver. 

Here are some of mine to add to Vanessa's list.

One on imitation learning. [Currently an "accept with minor revisions" at JMLR]

One on conservatism in RL. A special case of Vanessa's infra-Bayesianism. [COLT 2020]

One on containment and myopia. [IEEE]

Some mathy AI safety pieces or other related material off the top of my head (in no particular order, and definitely not comprehensive nor weighted toward impact or influence):

• The Speed + Simplicity Prior is probably anti-deceptive
• Prediction can be Outer Aligned at Optimum
• Reinforcement Learning in Newcomblike Environments
• Commitment games with conditional information revelation
• Chris Olah's older pieces on neural networks (under 'Neural Networks (General)' and below)

In outer alignment one can write down a correspondence between ML training schemes that learn from human feedback and complexity classes related to interactive proof schemes.  If we model the human as a (choosable) polynomial time algorithm, then

1. Debate and amplification get to PSPACE, and more generally $n$-step debate gets to ${\Sigma }_{n}P$.
2. Cross-examination gets to NEXP.
3. If one allows opaque pointers, there are schemes that go further: market making gets to R.

Moreover, we informally have constraints on which schemes are practical based on properties of their complexity class analogues.  In particular, interactive proofs schemes are only interesting if they relativize: we also have $IP=PSPACE$ and thus a single prover gets to PSPACE given an arbitrary polynomial time verifier, but w.r.t. a typical oracle $I{P}^O<PSPAC{E}^O$.  My sense is there are further obstacles that can be found: my intuition is that "market making = R" isn't the right theorem once obstacles are taken into account, but don't have a formalized model of this intuition.

The reason this type of intuition is useful is humans are unreliable, and schemes that reach high complexity class analogies should (everything else equal) give more help to the humans in noticing problems with ML systems.

I think there's quite a bit of useful work that can be done pushing this type of reasoning further, but (full disclosure) it isn't of the "solve a fully formalized problem" sort.  Two examples:

1. As mentioned above, I find "market making = R" unlikely to the right result.  But this doesn't mean that market making isn't an interesting scheme: there are connections between market making and Paul Christiano's learning the prior scheme.  As previously formalized, market making misses a practical limitation on the available human data (the $n$-way assumption in that link), so there may be work to do to reformalize it into a more limited complexity class in a more useful way.

2. Two-player debate is only one of many possible schemes using self-play to train systems, and in particular one could try to shift to $n$-player schemes in order to reduce playing-for-variance strategies where a behind player goes for risky lies in order to possibly win.  But the "polynomial time judge" model can't model this situation, as there is no variance when trying to convince a deterministic algorithm.  As a result, there is a need for more precise formalization that can pick up the difference between self-play schemes that are more or less robust to human error, possibly related to CRMDPs.

You might be interested in this great intro sequence to embedded agency. There's also corrigibility and MIRI's other work on agent foundations.

Also, coherence arguments and consequentialist cognition.

AI safety is a young field; for most open problems we don't yet know of a way to crisply state them in a way that can be resolved mathematically. So if you enjoy taking messy questions and turning them into neat math you'll probably find much to work on.

ETA: oh and of course ELK.