Mercial applications are now open, for anybody looking to build a project in AI safety! Do models think stealing their own weights is good? How likely is a war over Taiwan? Does publishing safety research teach models to evade it?
Mercial is a remote, part-time research fellowship: one month, a facilitator, and a real problem in technical or conceptual AI safety. There are 11 streams this round, covering agent collusion, interpretability, compute governance, alignment evals and more. It’s made to build the track record you need to get into further AI safety programmes.
Fellowship runs 1 November – 1 December. Streams and applications:https://mercial.uk/fellowship Applications close 28 October.
I think most AI safety bootcamps could be improved significantly by shifting focus off from coding. None of the researchers I know code, and I think the counterfactual activity of reading / thinking high-level about concepts is better for building context and conceptual thinking. As well as this, the technical AI safety pipeline serves more than just technical research roles (people transition into startups, grantmaking, other non-technical roles), and I think context building serves all of these roles better than coding. I think a default for a lot of bootcamps is to follow the ARENA curriculum, which is in parts outdated or marginally bad (eg I think learning about the specifics of how SAEs work, instead of the huggingface incident is clearly not optimal).
I feel like the risk from general-purpose robot swarms growing in size is somewhat contingent on some party already having a large amount of power. For example, if AGI came about it would probably automate a lot of physical labour with robots, but I kinda expect this would happen post-takeover.
Its true that it would probably make it easier for a totalitarian regime to be established, but I think that a significant component of this risk would come from diminishing the economic leverage of the general population's manual labour by displacing their jobs. Then the government has less incentive to look out for the average citizen, more corruption happens, and after time there is a dictatorship of sorts? And this kind of risk is shared by a lot of novel technologies, so this could diminish how bad the risk seems.
Mercial applications are now open, for anybody looking to build a project in AI safety!
Do models think stealing their own weights is good?
How likely is a war over Taiwan?
Does publishing safety research teach models to evade it?
Mercial is a remote, part-time research fellowship: one month, a facilitator, and a real problem in technical or conceptual AI safety. There are 11 streams this round, covering agent collusion, interpretability, compute governance, alignment evals and more. It’s made to build the track record you need to get into further AI safety programmes.
Fellowship runs 1 November – 1 December.
Streams and applications: https://mercial.uk/fellowship
Applications close 28 October.
I think most AI safety bootcamps could be improved significantly by shifting focus off from coding. None of the researchers I know code, and I think the counterfactual activity of reading / thinking high-level about concepts is better for building context and conceptual thinking. As well as this, the technical AI safety pipeline serves more than just technical research roles (people transition into startups, grantmaking, other non-technical roles), and I think context building serves all of these roles better than coding. I think a default for a lot of bootcamps is to follow the ARENA curriculum, which is in parts outdated or marginally bad (eg I think learning about the specifics of how SAEs work, instead of the huggingface incident is clearly not optimal).
How large of a risk do you propose this is?
I feel like the risk from general-purpose robot swarms growing in size is somewhat contingent on some party already having a large amount of power. For example, if AGI came about it would probably automate a lot of physical labour with robots, but I kinda expect this would happen post-takeover.
Its true that it would probably make it easier for a totalitarian regime to be established, but I think that a significant component of this risk would come from diminishing the economic leverage of the general population's manual labour by displacing their jobs. Then the government has less incentive to look out for the average citizen, more corruption happens, and after time there is a dictatorship of sorts? And this kind of risk is shared by a lot of novel technologies, so this could diminish how bad the risk seems.