Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm?
We’ve seen two large and extremely capable swarms from OpenAI in the last few months:
* 1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched...
Often folks hit us up because they are thinking of starting an incubator and want advice.
Typically their motivation is either that (a) they have a list of specific things they want built that no one is building, or (b) they think an ecosystem needs more new projects generally to absorb more talent and deploy more funding effectively.
Here are six questions we often ask prospective teams, to help them figure out what to do. If you're incubator-curious...
Summary:
First, I give several different angles on how I feel about reinforcement learning:
* Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffolding
* Recent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progress
* I’m worried things might get worse:...
This seems interesting, but I'm confused as to what the point of this is over "work at an EA org". It seems like most EA orgs, are bottlenecked a lot more on talent than money, and if you're doing high-talent work for an EA organization than your marginal hour is likely more valuable than $60/hr. I wonder what subset of the population would benefit substantially from this advice—it seems like earning to give and direct work cover most of the space that earning to volunteer might.
What kind of person is this advice targeted at, and why do you think that this is better than direct work for those people?
"If you're doing high-talent work for an EA organization than your marginal hour is likely more valuable than $60/hr." And yet, few EA orgs pay $60-$90 per hour for non-programming jobs? Seems reasonable to me that someone might want to tutor or program for cash while writing EA content on the side, rather than getting paid a low writer's wage to produce said content -- the latter has comparable income, less total work, but also less flexibility to work when you want. Plus if you're volunteering, you can more easily hop between organizations to always focus on the projects that seem highest-impact to you.
Another advantage of earning-to-volunteer is to build skills in a large market where you're sure of your long-term prospects. Versus jumping into an EA career might seem riskier, since you are putting more of your life's eggs into the EA basket.
I think this strategy might be suitable for some people, enabling them to have a nice (if busy) lifestyle while also having fun contributing a lot to EA. But of course I don't think EA should try to market it widely because it might come dangerously close to sounding like we are advocating for people to slack off on their day job and rip off their employers. That message would be terrible for EA's reputation.
Regarding bottleneck, some areas are massively funding constrained e.g. I'm not sure I could name a single mental EA mental health charity that is talent constrained. People who graduate with firsts from the world's top 3 or so neuroscience/psychology departments, with a ton of work experience, are passionate enough to be willing to volunteer 20Hs per week but one can't hire and train them up as there simply isn't enough funding.
This advice is targeted at: