Related to open relation extraction, you might find the ROME paper interesting: Locating and Editing Factual Associations in GPT:

Knowing differs from saying: uttering words by rote is different from knowing a fact, because knowledge of a fact generalizes across contexts. In this project, we show that factual knowledge within GPT also corresponds to a localized computation that can be directly edited. For example, we can make a small change to a small set of the weights of GPT-J to teach it the counterfactual "Eiffel Tower is located in the city of Rome." Rather than merely regurgitating the new sentence, it will generalize that specific counterfactual knowledge and apply it in very different linguistic contexts.

Yeah your first point is probably true, 100 may be unreasonable even as a lower bound (in the rightmost column). I should change it. 

--

Following  your second point, I changed:

Upon entering the field you may receive sufficiently strong indications that you will not be able to be a part of the most efficacious fraction of AI safety researchers. 

to

Upon entering the field (or just on reviewing your own personal fit) you may receive sufficiently strong indications that you will not be able to be a part of the most efficacious fraction of AI safety researchers. 

Important point. I changed 

... AI safety research seems unlikely to have strong enough negative unexpected consequences to outweigh the positive ones in expectation.

to

... Still, it's possible that there will be a strong enough flow of negative (unforseen) consequences to outweigh the positives. We should take these seriously, and try to make them less unforseen so we can correct for them, or at least have more accurate expected-value estimates. But given what's at stake, they would need to be pretty darn negative to pull down the expected values enough to outweigh a non-trivial risk of extinction.

Thanks!  Means a lot :) 
(I promise this is not my alt account flattering myself)

I'll be attending MLAB2 in Berkeley this August so hopefully I'll meet some people there.

Yes, that is true. I'm sure those other careers are also tremendously valuable. Frankly I have no idea if they're more or less valuable than direct AI safety work. I wasn't making any attempt to compare them (though doing so would be useful). My main counterfactual was a regular career in academia or something, and I chose to look at AI safety because I think I might have good personal fit and I saw opportunities to get into that area. 

This is a good point I hadn't considered. I've added a few rows calculating a marginal correction-factor to the google sheet  and I'll update the table if you think they're sensible. 

The new correction factor is based on integrating an exponentially decaying function from N_researchers to N_researchers+1, with the decay rate set by a question about the effect of halving the size of the AI alignment community. Make sure to expand the hidden rows in the doc if you want to see the calculations.