A method I've found useful for generating lots of ideas is to assume that reductio ad absurdum is not valid. This might be useful here too, for slightly different reasons.
I really like this. In fact, I would take it a step further. I believe we should expand the multi-armed bandit model to cover exploring areas like:
Philosophy, particularly ethics. It would be nice to know whether Hedonic or Preference utilitarianism is correct without having to compute all of our Coherent Extrapolated Volition. Perhaps a few people doing such narrow, targeted research could make headway in our lifetimes with university funding rather than EA money. This seems likely to make a large impact in what EAs fund for generations to come.
Neuroscience, especially regarding qualia and types of thought with moral concern. Principia Qualia is an EA attempt to solve this. This could resolve the questions which divide EA between animal and human charities.
Finding new big ideas. There are already people working on big projects like x-risks, s-risk, space colonization/industrialization, curing aging, cryogenic freezing, simulating neurons digitally, brain-computer interfaces, AGI, nanotechnology, self-replicating machines, etc. Most of these will likely fail, but perhaps a few will succeed if we're not just deluding ourselves on all counts. Are there entirely new fields which no one has thought of yet? I suspect the answer is yes. Having a better understanding of our own utility function would narrow the search space of valuable ideas significantly, but I think we likely can make headway based on existing philosophy. Robin Hanson made some suggestions just a few days ago.
Improving EA thought, mental-tools, physical tools, methodologies, and other ways of exploring/exploiting more efficiently. (This was mentioned in the OP, but I wanted to highlight it.)
Resources. In Situ Resource Utilization is necessary for any colony of any size. (It could also dramatically reduces transportation costs for a smaller colony.) Unfortunately, "magnificent desolation" sums it up pretty well. Yes, you can make lunar concrete, any yes, they found ice in the permanently shadowed craters of the loner poles, and yes, there is silica in the sand, just like all sand on earth. But, that's about it. I'm all for unmanned mining, but humans would be putting the cart before the horse.
The moon is tidally locked to the earth, meaning that the same face is always facing us. As a result, it rotates exactly once a month. That's a really long day. If using solar power, that means needing enough batteries to last 14 days. For colonization, it means crops cannot be grown using sunlight, necessitating enormous amounts of solar energy. Yes, the peaks of eternal light on the poles are in continuously in sunlight, but they don't have a ton of surface area, and would need extremely tall, rotating solar panels to take advantage of the solar power. So, the growth of any colony would be extremely limited.
On the resource side, Mars has enormous icecaps, large glaciers, and small concentrations of chemically bound water in the soil even near the equator. It even occasionally has liquid water on the surface, in areas where the salt concentration is high enough that frost can form a liquid brine. This provides a ready source of both H2 and O2. Mars also has CO2, Ar, and N2 in the atmosphere. The moon, however is lacking in carbon, which makes it difficult to grow food, and it doesn't have any inert gas handy for a breathable atmosphere. (You don't want to breathe 100% O2.) N2 is also a necessary component in fertilizers.
With respect to power and crop growth, Mars has a day that lasts 24 hours, 39 minutes, which makes solar power viable without massive batteries to store the electricity for 2 weeks. The downside is that Mars is further from the sun. Wind isn't strong enough to be useful, and geothermal may or may not be possible, so we're stuck with mediocre solar. A small nuclear reactor like what is used on nuclear submarines would be helpful to power growing colonies on either the moon or mars, but politically difficult.
So, if you want to maximize the probability that a self-sustaining colony is built in space, I would concentrate on tech applicable to mars. Particularly, rather than looking at all the components needed by such a colony, I would try and figure out a different question. What is the minimum amount (number, mass, etc.) of machines necessary to produce most of the heaviest components of such a colony?
There are already plenty of people in the space community, but as far as I can tell almost all of them are bikeshedding. If you want to make a difference there, I strongly recommend finding a clever, minimalist solution to the problem of bootstrapping a martian (or lunar) industrial base.
I finally got over the trivial inconvenience of creating an account just to upvote this.
This sort of thing seems like precisely the right approach from an explore/exploit (multi-armed bandit) perspective. In fact, while most criticism of EA seems to center around the "effective" part, it seems like the "altruism" part may be the weaker target. (What if the specific subset of goals EAs look to maximize aren't what we'd endorse upon reflection?)
So, this sort of research could lead to entirely new branches of EA, which wouldn't exist counterfactually. It could also help EAs make informed decisions reguarding how to weight different types of suffering (like QALYs and DALYs), or human suffering against animal suffering, or X units of happiness against Y units of suffering, or even between entirely different types of qualia.
However, even more broadly, I don't see qualia themselves as being the only thing of value. It seems like they are the reward/punishment signals that prod us into developing certain values and not others. As a preference utilitarian and not a hedonistic utilitarian, it's those values I'm most interested in, and not qualia per se, unless on reflection we decide to place terminal value on the qualia themselves.
That said, understanding qualia seems like at least a first step in understanding and improving value generation, if it occurs via reinforcement learning. And, even if we only value some nonzero fraction of qulia upon reflection, tiling the universe with that is still hugely preferable to tiling the universe with paperclips or whatever. So, it could offer a plan B for some existential threats, with flexible timing in the event of an arms race.
A method I've found useful for generating lots of ideas is to assume that reductio ad absurdum is not valid. This might be useful here too, for slightly different reasons.
I really like this. In fact, I would take it a step further. I believe we should expand the multi-armed bandit model to cover exploring areas like:
Philosophy, particularly ethics. It would be nice to know whether Hedonic or Preference utilitarianism is correct without having to compute all of our Coherent Extrapolated Volition. Perhaps a few people doing such narrow, targeted research could make headway in our lifetimes with university funding rather than EA money. This seems likely to make a large impact in what EAs fund for generations to come.
Neuroscience, especially regarding qualia and types of thought with moral concern. Principia Qualia is an EA attempt to solve this. This could resolve the questions which divide EA between animal and human charities.
Finding new big ideas. There are already people working on big projects like x-risks, s-risk, space colonization/industrialization, curing aging, cryogenic freezing, simulating neurons digitally, brain-computer interfaces, AGI, nanotechnology, self-replicating machines, etc. Most of these will likely fail, but perhaps a few will succeed if we're not just deluding ourselves on all counts. Are there entirely new fields which no one has thought of yet? I suspect the answer is yes. Having a better understanding of our own utility function would narrow the search space of valuable ideas significantly, but I think we likely can make headway based on existing philosophy. Robin Hanson made some suggestions just a few days ago.
Improving EA thought, mental-tools, physical tools, methodologies, and other ways of exploring/exploiting more efficiently. (This was mentioned in the OP, but I wanted to highlight it.)
No, but maybe a Mars colony.
The moon has a couple issues:
Resources. In Situ Resource Utilization is necessary for any colony of any size. (It could also dramatically reduces transportation costs for a smaller colony.) Unfortunately, "magnificent desolation" sums it up pretty well. Yes, you can make lunar concrete, any yes, they found ice in the permanently shadowed craters of the loner poles, and yes, there is silica in the sand, just like all sand on earth. But, that's about it. I'm all for unmanned mining, but humans would be putting the cart before the horse.
The moon is tidally locked to the earth, meaning that the same face is always facing us. As a result, it rotates exactly once a month. That's a really long day. If using solar power, that means needing enough batteries to last 14 days. For colonization, it means crops cannot be grown using sunlight, necessitating enormous amounts of solar energy. Yes, the peaks of eternal light on the poles are in continuously in sunlight, but they don't have a ton of surface area, and would need extremely tall, rotating solar panels to take advantage of the solar power. So, the growth of any colony would be extremely limited.
On the resource side, Mars has enormous icecaps, large glaciers, and small concentrations of chemically bound water in the soil even near the equator. It even occasionally has liquid water on the surface, in areas where the salt concentration is high enough that frost can form a liquid brine. This provides a ready source of both H2 and O2. Mars also has CO2, Ar, and N2 in the atmosphere. The moon, however is lacking in carbon, which makes it difficult to grow food, and it doesn't have any inert gas handy for a breathable atmosphere. (You don't want to breathe 100% O2.) N2 is also a necessary component in fertilizers.
With respect to power and crop growth, Mars has a day that lasts 24 hours, 39 minutes, which makes solar power viable without massive batteries to store the electricity for 2 weeks. The downside is that Mars is further from the sun. Wind isn't strong enough to be useful, and geothermal may or may not be possible, so we're stuck with mediocre solar. A small nuclear reactor like what is used on nuclear submarines would be helpful to power growing colonies on either the moon or mars, but politically difficult.
So, if you want to maximize the probability that a self-sustaining colony is built in space, I would concentrate on tech applicable to mars. Particularly, rather than looking at all the components needed by such a colony, I would try and figure out a different question. What is the minimum amount (number, mass, etc.) of machines necessary to produce most of the heaviest components of such a colony?
There are already plenty of people in the space community, but as far as I can tell almost all of them are bikeshedding. If you want to make a difference there, I strongly recommend finding a clever, minimalist solution to the problem of bootstrapping a martian (or lunar) industrial base.
I finally got over the trivial inconvenience of creating an account just to upvote this.
This sort of thing seems like precisely the right approach from an explore/exploit (multi-armed bandit) perspective. In fact, while most criticism of EA seems to center around the "effective" part, it seems like the "altruism" part may be the weaker target. (What if the specific subset of goals EAs look to maximize aren't what we'd endorse upon reflection?)
So, this sort of research could lead to entirely new branches of EA, which wouldn't exist counterfactually. It could also help EAs make informed decisions reguarding how to weight different types of suffering (like QALYs and DALYs), or human suffering against animal suffering, or X units of happiness against Y units of suffering, or even between entirely different types of qualia.
However, even more broadly, I don't see qualia themselves as being the only thing of value. It seems like they are the reward/punishment signals that prod us into developing certain values and not others. As a preference utilitarian and not a hedonistic utilitarian, it's those values I'm most interested in, and not qualia per se, unless on reflection we decide to place terminal value on the qualia themselves.
That said, understanding qualia seems like at least a first step in understanding and improving value generation, if it occurs via reinforcement learning. And, even if we only value some nonzero fraction of qulia upon reflection, tiling the universe with that is still hugely preferable to tiling the universe with paperclips or whatever. So, it could offer a plan B for some existential threats, with flexible timing in the event of an arms race.