Right, most functionalist theories are currently not physically constrained. But if it turns out that consciousness requires causal properties implemented by EM fields, then the functions used by the theory would become ones defined in part by the math of EM fields. Which could then turn out to be physically constrained in practice, if the relevant causal properties could only be implemented by EM fields. (Though it might still allow a sufficiently good computer simulation of a brain with EM fields to be conscious.)

But "there are multiple ways to physically realize any (Turing-level) computation, and multiple ways to interpret a physical realization as computation, and no privileged way to choose between them"

This argument feels somewhat unconvincing to me. Of course, there are situations where you can validly interpret a physical realization as multiple different computations. But I tend to agree with e.g. Scott Aaronson's argument (p. 22-25) that if you e.g. want to interpret a waterfall as computing a good chess move, then you probably need to include in your interpretation a component that calculates the chess move and which could do so even without making use of the waterfall. And then you get an objective way of checking whether the waterfall actually implements the chess algorithm:

... it seems overwhelmingly likely that any reduction algorithm would just solve the chess problem itself, without using the waterfall in an essential way at all! A bit more precisely, I conjecture that, given any chess-playing algorithm A that accesses a “waterfall oracle” W, there is an equally-good chess-playing algorithm A0 , with similar time and space requirements, that does not access W. If this conjecture holds, then it gives us a perfectly observer-independent way to formalize our intuition that the “semantics” of waterfalls have nothing to do with chess. [...]

Interestingly, the issue of “trivial” or “degenerate” reductions also arises within complexity theory, so it might be instructive to see how it is handled there. [...] Suppose we want to claim, for example, that a computation that plays chess is “equivalent” to some other computation that simulates a waterfall. Then our claim is only non-vacuous if it’s possible to exhibit the equivalence (i.e., give the reductions) within a model of computation that isn’t itself powerful enough to solve the chess or waterfall problems.

Likewise, I think that if a system is claimed to implement all of the functions of consciousness (whatever exactly they are) and produce the same behavior as that of a real conscious human... then I think that there's some real sense of "actually computing the behavior and conscious thoughts of this human" that you cannot replicate unless you actually run that specific computation. (I also agree with @MichaelStJules 's comment that the various functions performed inside an actual human seem generally much less open to interpretation than the kinds of toy functions mentioned in this post.)

Interesting post!

One thing that confused me was that there were sections where it felt like you suggested that X is an argument against functionalism, where my own inclination would be to just somewhat reinterpret functionalism in light of X. For example:

I’ve yet to see a satisfactory functionalist account of how binding can happen (and I’ve come to believe that it’s not even possible in principle). At the same time, possible solutions positing that binding happens, for example, at the level of the electromagnetic fields produced by neurons strike me as elegant, parsimonious, and rigorous.

If it turns out that the information in separate neurons is somehow bound together by electromagnetic fields (I'll admit that I didn't read the papers you linked, so don't understand what exactly this means), then why couldn't we have a functionalist theory that included electromagnetic fields as its own communication channel? If we currently think that neurons communicate mostly by electric and chemical messages, then it doesn't seem like a huge issue to revise that theory to say that the causal properties involved are achieved in part electromagnetically.

I think that some of the bits in that essay were too strong, in particular this line

All, or at least most, of education could be done via games that were as addictive and enjoyable as traditional games.

was probably wrong, for reasons Andy Matuschak outlines:

Games are designed first and foremost to be fun—or beautiful, or engrossing, or exhilarating. Games are an aesthetic medium, and (generally speaking) they compel our participation insofar as they compel us aesthetically. It’s true that in some games, players end up developing certain skills or understandings along the way. But that doesn’t mean we can make a great game that teaches anything. You’re seeing the survivors. These games’ designers tried and discarded dozens of gameplay ideas in search of something aesthetically compelling. Then, only after they’d satisfied the primary constraint of making something fun, or beautiful, or whatever, the designers figured out how to ensure people would learn what they need as they play. Most mechanisms are not fun. Good games come from a demanding selection process which works the other way around: first, find the fun. There’s no reason at all to believe that for any arbitrary abstract topic, one can always “find the fun” which implicitly teaches it.

On the other hand, in principle it still seems to me like you should be able to make games that significantly improve on current education. Even if an edugame wasn't as fun as a pure entertainment game, it could still be more fun than school. And people still watch documentaries because they value learning, even though documentaries can't compete with most movies and TV shows on pure entertainment value.

But then again, for some reason DragonBox seems to have been an exception rather than the rule. Even the company that made it mostly just made games for teaching simpler concepts to younger kids afterward, rather than moving on to teaching more complicated concepts. The fact that I haven't really heard of even reasonably-decent edugames coming out in the 11 years since that post seems like strong empirical evidence against its thesis, though I don't really understand the reason for that.

Thanks for noticing that! These days I would always use "their" in this kind of context, but I guess I didn't yet have that habit back in 2014. Edited.

(Upvoted.)

Some of the attempts within EA to solve this seem to be to push even more towards just being a professional network. I think that's dangerously wrong, because it doesn't remove the informal networks and their power. It just makes access to them harder, and people more desperate to get in.

Somewhat relevant counterpoint:

For everyone to have the opportunity to be involved in a given group and to participate in its activities the structure must be explicit, not implicit. The rules of decision-making must be open and available to everyone, and this can happen only if they are formalized. This is not to say that formalization of a structure of a group will destroy the informal structure. It usually doesn't. But it does hinder the informal structure from having predominant control and make available some means of attacking it if the people involved are not at least responsible to the needs of the group at large. [...]

... an elite refers to a small group of people who have power over a larger group of which they are part, usually without direct responsibility to that larger group, and often without their knowledge or consent. [...] Elites are nothing more, and nothing less, than groups of friends who also happen to participate in the same political activities. They would probably maintain their friendship whether or not they were involved in political activities; they would probably be involved in political activities whether or not they maintained their friendships. It is the coincidence of these two phenomena which creates elites in any group and makes them so difficult to break.

These friendship groups function as networks of communication outside any regular channels for such communication that may have been set up by a group. If no channels are set up, they function as the only networks of communication. [...] 

Some groups, depending on their size, may have more than one such informal communications network. [...] In a Structured group, two or more such friendship networks usually compete with each other for formal power. This is often the healthiest situation, as the other members are in a position to arbitrate between the two competitors for power and thus to make demands on those to whom they give their temporary allegiance.

gwern on /r/machinelearning:

There's no comparison to prior full-press Diplomacy agents, but if I'm reading the prior-work cites right, this is because basically none of them work - not only do they not beat humans, they apparently don't even always improve over themselves playing the game as if it was no-press Diplomacy (ie not using dialogue at all). That gives an idea how big a jump this is for full-press Diplomacy.

There's a commentated video by someone who plays as the only human in an otherwise all-Cicero game, which at least makes it seem like the dialogue is doing a lot.

Worth noting that this was "Blitz" Diplomacy with only five-minute negotiation rounds. Still very impressive though.

Some behavioral traits such as general intelligence show very high heritability – over 0.70 – in adults, which is about as heritable as human height.

I'm very confused about what numbers such as this mean in practice, since the most natural interpretation ("70% of the trait is genetically determined") is wrong, but there aren't very many clear explanations of what the correct interpretation is. When I tried asking this on LW, the top-voted answer was that it's a number that's mostly useful if you're doing animal breeding, but probably not useful for much else.

You mention a lot of heritability numbers, could you clarify what it is that we're intended to infer from them? (It seems to me that the main thing we can infer from heritability numbers is that if a trait has heritability above zero, then there's some genetic influence on it, but since you mention some traits having "very high" heritability, I presume that you find there to be some other information too.)

I'm not sure if there is any reason that should be strongly persuasive to a disinterested third party, at the moment. I think the current evidence is more on the level of "anecdotally, it seems like a lot of rationalists and EAs get something out of things like IFS". 

But given that one can try out a few sessions of a therapy and see whether you seem to be getting anything out of it, that seems to be okay?  Anecdotal evidence isn't enough to strongly show that one should definitely do a particular kind of therapy. But it can be enough to elevate a therapy to the level of things that might be worth giving a shot to see if anything comes out of it.