You want to do something to help AI go well and are starting a project to make that happen. Should you create a nonprofit or a for-profit?
A lot of charitably-minded people naturally default to “nonprofit.” Their goal is to do good things for others, not make money for themselves. But we think that’s the wrong way to look at it.
A for-profit is just an alternative legal vehicle for doing good. When consumers pay for a product, there is surplus created; part goes to the consumer, and part goes to the company. So the basic distinction between nonprofit and for-profit isn’t “doing good” vs. “making money”: it’s that the good that for-profits are compensated for doing is limited to excludable goods for users who can pay.
If your mission is altruistic, unless your funding model is “impact certificates bought by God,” the structural incentives of your organization are never going to be perfectly aligned with your mission. There’s a natural tendency for the goals of your organization, as revealed by its actions, to drift toward its structural incentives. Nonprofits and for-profits face different sets of structural incentives.
So instead of asking, “do I want to do good or make money,” consider asking “what is the value I’m trying to create, and what funding sources and set of incentives best track that?”
A typical nonprofit vs. a typical for-profit
A typical nonprofit model goes like:
- raise money from donors
- spend the money
- repeat; continue raising donations on a regular basis to fund operations
A typical for-profit startup goes like:
- raise equity capital from investors
- spend the money to eventually generate revenue
- use that revenue to fund more spending
It’s worth noting these are simplifications. A nonprofit can charge customers and earn revenue (and doing this gets you some of the benefits of being a for-profit). But a nonprofit can’t legally raise equity capital.
Good things about for-profit models
1. It’s much easier for for-profits to get big.
A lot of the upside of founding an organization comes from the probability that it will end up being very big and influential.
This is much easier for for-profits, because their ability to raise equity capital and to make revenue and reinvest it in their operations allows for compounding growth.
2. For-profits have feedback loops and contact with reality that donor-funded nonprofits often lack
Donor-funded feedback loops are long (on the order of 3-12 months between fundraising cycles), and require a long inferential chain, with grantmakers often reasoning about what future people would want from an altruistic perspective.
In contrast, the feedback from revenue is short (it can be on the timescale of days or seconds), and comes from users noticing what they want personally. If you are forced to provide something that users want and will pay for, it may not be good in every way, but you’re at least making something that’s good on one dimension and getting the regular feedback to do so.
3. For-profit models allow the space to grow without grantmaker/trust bottlenecks.
The for-profit model comes with positive selection effects: companies grow if they are providing value to customers and shrink otherwise. This is the same reason market economies are better than centrally planned economies; capital allocation is a really hard job, and even wise and benevolent central planners have a hard time competing with the invisible hand.
Historically, AI safety has relied on a small set of highly trusted grantmakers. But as funding in the space grows 100x, we don’t think we can 100x the number of human grantmakers.
4. For-profits can access different and arguably better talent.
For-profits can use equity packages in compensation and have more latitude to pay market rates without scrutiny. This strengthens your ability to get the best people.
5. For-profits have different cultural influences.
For-profits are surrounded by different sets of peers, with different vibes and different social pressures. Nonprofit culture is more focused on sacrifice, care, deep thought, and research. For-profit culture is more action-oriented and focused on shipping often and “making something people want”.
This isn’t to say that one or the other is better; we think both vibes are valuable. But on the margin right now, the AIS ecosystem is heavily weighted toward nonprofits, and could benefit from some more shipping energy.
6. Money can be turned into impact.
The AIS space only has the money it has today because of people like Dustin Moskovitz founding for-profit companies. The pending wave of funding comes from for-profits like Anthropic & OpenAI. The people founding for-profits today can be the funding for the next generation of orgs.
What we’re not saying
“Making money proves you are providing value to the world.”
This is definitely not true. There are lots of ways companies can make money while harming the world:
- benefiting customers while creating negative externalities for others
- selling addictive products
- rent-seeking and regulatory capture
- anticompetitive practices
Starting a company rather than a non-profit doesn’t absolve you of the necessity to be constantly thinking about your mission and trying to do good. Making money shouldn’t be the goal, but it can be a useful proxy for the goal.
What might revenue models for AIS orgs look like?
Note: not all these examples are for-profits. Some of these funding mechanisms can and do happen in nonprofit structures as well. But self-funding mechanisms beyond donations are necessary in for-profits, and we think they can be a valuable way of making contact with reality in nonprofits as well.
- Selling safety services to AI labs or companies using AI tools.
- Orgs are doing this already: selling interpretability tools, agent monitoring tools, or red-teaming/security services to labs or to enterprises that use frontier models.
- e.g. Apollo Research, Goodfire, Gray Swan
- Creating user-facing products and charging for them.
- We think creating products that help people think and research better can be both useful to users, and good more broadly. This could be specific to users in the AI safety community, or part of a general raising of the epistemic waterline.
- e.g. Elicit, Pangram
- Charging other AIS orgs.
- This applies to organizations providing shared infrastructure: coworking spaces, funding infrastructure, job boards.
- e.g. Mox, Constellation, Lighthaven, EAG, The Curve
- Serving individuals in the AI safety scene.
- This could take lots of forms: therapy or productivity coaching; creating fun experiences; helping with immigration, housing, or dating.
- e.g. Manifest, 10zeros, Manifold.love
- Government contracts.
- This could be consulting for or training people in government, selling AI tools to government agencies, or working on verification technology in advance of regulation.
- e.g. METR, Gladstone AI, Equistamp
- Charging for training.
- Most current AIS courses and fellowships are free, but could be paid. This could be done by charging students directly, income share agreements, or charging orgs to use them as a hiring pipeline.
- e.g. Inkhaven, MATS, BlueDot
These ones we’re less confident make sense, but might be worth considering.
- Media funding: ads or subscriptions.
- The very best content creators can successfully monetize, but the correlation between monetizability and goodness in media is definitely tenuous.
- e.g. Dwarkesh, ACX, Semianalysis
- Trading.
- If you are good at forecasting, you might be able to make money with it. This could be a path to improving public epistemics, either indirectly by e.g. making prediction markets more efficient, or more directly by publicizing your forecasting at some point.
- e.g. Situational Awareness, FutureSearch
Manifund’s revenue model
Manifund is a nonprofit, but we mostly self-fund by charging a 5% fee on donations made through the platform. We could remove the fee and instead solicit donations to fund our operations. But we like the fee: it means the more useful we are to donors and grantees, the more money we have to operate with. If people don’t find our platform helpful to use, they’ll stop going through Manifund, and we will cease to exist. This seems better than trying to convince some overworked new grad at CG to do a BOTEC on how much value we are providing to the AI safety ecosystem.
We’d love a bunch more examples of how orgs think about this in practice and how their thinking has changed over time; maybe that’s a topic for a future piece! E.g. Apollo Research and Elicit both began as nonprofits, but spun off for-profit entities, citing the ability to scale up as the reason.
What if I don’t have a business model?
Plenty of successful companies have started with a good idea or product without having a business model, e.g. Google.
The potential benefits from scale hold whether or not you have a business model: it’s still easier to scale as a for-profit. If Larry and Sergey had begun Google as a nonprofit, it never would have accomplished as much as it did.
Obviously though, feedback from revenue only applies if you have a revenue model. And for raising funds, it simply means you now have to rely on VC trust rather than grantmaker trust. Whether this is better or worse probably depends on your specific project.
Note that while Google didn’t have a business model, they did have a product that users wanted. If you don’t have anything like that, you may be better off as a nonprofit.
What about taxes?
Having tax-deductible donations is cool. All else equal, it basically makes you 1/(1-37%) - 1 = 59% richer, because you get paid in pre-tax rather than after-tax dollars.
However, we think a lot of these other factors can be way bigger in expectation. If operating as a for-profit gives you a +10% chance of 10xing in size, then that’s worth way more than the tax deductibility.
What about investor pressure?
As mentioned before, we think you should see revenue as a proxy rather than an end in itself. However, if you raise equity capital, your investors might not see things the same way.
This is definitely worth thinking about, both in terms of finding investors who are mission-aligned, and questioning whether having investors at all will distort your organizational incentives too much.
In the AI space, the for-profit structure used is usually a PBC (public benefit corporation). This is the structure of OpenAI, Anthropic, xAI, and lots of smaller companies. The corporation creates a charter which names a public benefit, and directors are required to balance stockholders’ interests with the named public benefit.
Realistically, there isn’t much of a legal mechanism ensuring that PBCs prioritize public benefit, but it does help somewhat in shielding companies from investor pressure. It seems good but definitely not sufficient.
What maybe shouldn’t be a for-profit?
These are things that are clearer fits for nonprofit models. But we’re still excited about ways to turn these into for-profits!
- Research for publication
- Basic research to improve humanity’s knowledge is a pretty textbook example of a public good.
- Reflecting this, some of the largest nonprofits are universities.
- Orgs benefiting those who can’t pay
- If you’re focused on animal welfare, or the rights of digital beings, it’s hard to turn that into something that makes money.
- Things where revenue causes perverse incentives
- If your goal is to be a watchdog for AI labs, you probably shouldn’t take money from AI labs.
- Fundamentally unscalable things
- If there’s no scope for your thing to get bigger, that eliminates one of the big reasons to go for-profit.
- Public advocacy and persuasion
- This isn’t really something you can track with some form of revenue.
Bottom line
If you’re starting an org, ask “who benefits from what I’m making, and can any of them pay?” If the answer is yes, consider taking the money. Revenue will help you scale, and it’s one of the best feedback loops you’ll ever get. Fall back to donations only if you’re making something no one can pay for.
(All of this are my own views, and based on public information)
I agree with the main claim here - people should consider for-profit vehicles when they are exploring founding an org. And I agree with much of the content. But I wish the piece was more balanced, specifically: (1) adequatley noting the downsides of for-profit structures, and (2) updating the valuation of non-profit structures given a potential era of funding abundance.
On (1):
On (2):
It seems weird to me for someone to countenance this fact and conclude that one should therefore try to be a for-profit instead, pursuing a very different kind of capital. Instead one should consider (and expect) that grant making could become much higher volume (even if the scale is below 100x), and that the near-term future may be the most accessible time in history for founding an ambitious non-profit.
Thank you for these comments! I very much appreciate the public engagement, and care a lot about figuring out what's true here. I'm responding quickly, and framing things strongly with the understanding that there's much more room for nuance everywhere.
[quick edit, I realized I wrote the below for a "cG grantmaker" and you specifically work in Longview now, apologies. my unsolicited prescriptions for Longview are probably a bit different than for cG but I think the general points hold?]
I agree with "tails come apart", it's not obvious that the biggest corporations are the most impactful, like I wouldn't allocate more philanthropic capital to NVIDIA or SpaceX. But even these examples are illustrative of something good about forprofits, which is that allocation of credit between individual humans is much much better in forprofits -- Jensen Huang or Elon Musk are much much richer than the median employee of their org.
Why should an impact-minded founder care? Because if they push really hard and actually succeed at doing something impactful, they then have to go and pray that the grant evaluators up above agree with their assessment. While cG and founder missions are kind of aligned around wanting good impact in the world, they're also not fully, and I/Manifund endorse founders having more leverage over funders, on the margin.
Why should you, cG grantmaker shepherding the AIS ecosystem, care? Because without good credit allocation, people end up in confusing work situations, working on problems not suited to their ideal use of talent, doublecounting impact, negotiating on much fuzzier norms around "who was responsible for a specific project" being tracked in a social layer rather than in the clean numbers of equity and salary. Social layer is good for where AIS is today, but doesn't scale to handle 100x growth.
Okay, but what happens to the founder managing a $100m/y charity? Will cG and 501c3 law allow them to draw a proportionally larger salary?
There's one kind of EA answer which is "who cares, I was going to donate it to charity anyways, so it's equivalent to have this money be in my personal bank account vs in my charity vs in cG". I find elements of that answer which are compelling and noble and good, but also elements which are suspect wrt ecosystem-level feedback effects. I think it's important that successful individuals (and orgs) gain more ability to then influence what happens next in the ecosystem, and $ and equity are good numeric ways of building that feedback mechanism.
(also, seems good to nerdsnipe the non-fully-EA-aligned founders into working on problems we think are good)
One thing that has made the forprofit tech scene work is that, when a founder exits, they have extra $ to invest, and also experience running new orgs; they can the angel invest into the next generation (and also provide advice, connections, etc). I think mirroring this inside EA would be good. For example Ryan Kidd running MATS could have much more discretionary capital/ability to fund random projects that he thinks are promising but aren't "MATS Fellow" shaped, but doesn't by default. Fixing this is part of why Manifund runs AI safety regranting.
I've vaguely heard that cG is exploring doing more regranting (in the form of discretionary grant budgets to individuals with experience/taste + dealflow), and hope this is true!
I will be happy if & when this happens because it probably means I could stop working on Manifund and then go and found a thing instead. But I'm currently bearish on philanthropic funding becoming much less painful.
To raise a recent example, our fundraising for our own projects (Surplus and Mox) has been annoyingly bogged down in classic traps of philanthropic funding norms, namely "funder chicken". X asks "why isn't Y funding you instead" and Y asks the same question in reverse and we're waiting weeks to months to litigate. Grantmakers are already pressed for time and nobody has the chance to coordinate properly, and end up deferring too much.
There are clever mechanisms that are being proposed and used (funding backstops, matching offers, loans, etc) and also infra-level changes coming (Lightcone's doing a cool new S-Process thing, Manifund and grantmaking.ai are other kinds of trying to enable coordination). But fundamentally I like the forprofit norm of funders fighting for allocation in rounds and trying to compete on valuation & speed, rather than the nonprofit game of hoping somebody else steps in.
I want them to pursue philanthropic capital, at the end of the day! But I don't think the philanthropic funding ecosystem has good norms around seeding & incubating new orgs, compared to the tech startup ecosystem.
My general proposal is that big philanthropic orgs like cG should move towards being assessors of impact, similar to the final buyers in AMCs like Stripe Frontier or Operation Warp Speed. While distributing 501c3 funding, you can put up large prizes and award retro funding for concrete specific good work undertaken. (This is very not trivial fwiw, and does require teams of smart grantmakers to carefully reason through "what happened" and "how do we count it".)
But inside this ecosystem, new orgs should default incorporate as for-profit PBCs or C-Corps. And there should be a wide dispersion of intermediate funders (thnk preseed/seed/series A/B/C etc), who may be set up as 501c3 or standard forprofit, but either way providing funding to the new orgs in the form of standard forprofit equity investments.
Tech forprofit venture funding is already operating at the 100x scale that AI safety philanthropic funding hopes to get to, and so I think copying their homework (where reasonable) is wise. Again, my strong prior is that market economies perform better than centrally planned ones, at scale.
My guess is that currently cG overall is trying to instead move down in size and get into earlier-stage funding and incubation. I think more early stage funding and incubation is great, but also outside cG's core competency (eg have the people who are now in charge of those programs, previously founded successful orgs?), and I'm unsure whether it makes sense to house all of these initiatives under a single cG banner, vs have a somewhat more diversified ecosystem.
Thanks for the substantial engagement! You cover a lot, so I'll just pick out quotes and respond to ~most points.(As you note, I don’t work for CG and haven’t worked for them in over a year. I now work for Longview, but none of this is based on private information, and none of it represents views of either org).
Even if they were impactful, why would you consider allocating any philanthropic capital to incredibly rich companies? The marginal case would be terrible.
I think the right allocation of credit isn’t obviously better here. It is hard to figure out who were most responsible for an org’s success, in both for- and non-profits. CEOs get massive pay. There’s lots of reasons for that (high leverage, small market, rent-extraction via CEO’s influence on boards, equity-based pay, and so on). I don’t think you can assume that because CEO’s get enormous pay, they therefore deserve such pay (that may be true, but you’d want to actually try and investigate the credit allocation story - which is the question you started out with). I think there would be many cases of failed credit allocation in SpaceX, NVIDIA, and most other companies - i.e. many senior engineers or leaders who contributed much more value than the CEO, but were paid orders of magnitude less.
The pay is at least some signal of how the companies’ board values the executive’s performance. And if news of a CEO considering leaving or committing shifts investor sentiment and stock price, that is a good signal of how the market evaluates their performance. But these are merely okay instruments of assessing credit, and are usually strictly aligned with profit maximising. I don’t think they solve the credit allocation problem, and are a minor factor in comparing for- and non-profits.
I agree that for-profits reward extreme success much, much better than non-profits, especially given that extreme success may require taking personal risk and extreme levels of commitment. This is a problem that changes to the non-profit funding ecosystem should seek to fix.
I don’t agree that extreme CEO pay is a useful signal, so I don’t think it provides much useful information on credit allocation. It just shifts credit allocation from the grantmakers’ view to the view of a board - where such a board is significantly more constrained and incentivised away from impact optimisation.
Higher salaries are possible, and may be probable if there is funding abundance. There are some restrictions around what is ‘reasonable’ executive pay in the non-profit sector - maybe this could be a limit. But what is reasonable should scale with the size of the charity. For example, the CEO of the Gates Foundation earns $1.7M/yr. Extremely high salaries in the for-profit world are significantly equity-based, and this is the harder hurdle. I guess a key question is whether you think non-profit founders should get compensation in excess of low-seven-figures. That might be reasonable, but it's a weaker case than assuming they still get paid poorly.
I don’t really get why very high salaries would lead to greater influence on the ecosystem. Is it by directing their personal capital? I think in an era of funding abundance such capital is unlikely to be a major factor. If it is by their capital signalling apparent excellence, I don’t think that would work. I trust people according to how well they think and argue, my perception of their character, and their track record (and what other people I trust think of them). The amount of money they earn is close to irrelevant. There are countless people on very high salaries who I think are completely useless; there are countless people on very low salaires who I trust far, far more.
Agreed. Though there’s a slight worry about how internalised their altruistic motivations are, with the risk of value drift over time and making the wrong high-stakes call when incentives clash. But in general, yes we should want to leverage this talent pool.
Yep agree that past success should translate into ability to influence future grantmaking. Regranting is a method of doing this. However, my impressions is that regranting has often struggled because it takes real time to do it well. The usual failure mode of regranting is that the regrantor doesn’t spend enough money. In general you kind of need it to be a part of their job to do it effectively (like 0.2 FTE or more if I had to guess). But those with past successes have high opportunity cost - you would need to think that the regranting is a better use of their time than their counterfactual efforts at continuing their successful track record. That will be only true in some cases.
Sorry that that’s been your experience, and yes these issues are still live. One reason for optimism is that if there is funding abundance, and funders are all opportunity-constrained, the reasons for worrying about funging and coordination become less important. Seems plausible that much better solutions are out there that should be pursued anyway, as you mention.
Yes, I agree there are many lessons and structures from VC land that should be of interest to funders. Two caveats:
I don’t think earlier stage funding and incubation is ‘moving down in size’. It’s what you would do if you wanted many more organisations to be able to receive money. Orgs can only grow so fast without other things breaking, so if all your good grantees are growing as fast as possible, moving earlier is the main way to scale the relevant ecosystem. Your other option is to give lots to already-big organisations that are outside the relevant ecosystem. Maybe in some cases that will make sense, but it doesn't seem that promising a strategy.
Most VCs are not past founders. So if you like the VC approach, you shouldn’t worry about this. CG also has a great deal of experience in guiding the founding, early stages, and rapid growth of hundreds of organisations. I think that could actually be a more useful training ground than founding a single company, but they're at least comparable.
Agreed, a more diversified ecosystem would be good.
Thanks for the thoughtful comments! You and Austin already covered a lot, so I won't respond to everything.
On your point (2), I agree this is a strong point in favor of nonprofits. But I'm unsure how much to expect the future you describe--grantmaking way faster and higher volume, funding ambitious nonprofits and allowing them to grow aggressively--in some default world where a ton of new philanthropic capital gets put into CG, Longview, and similar orgs.
Is the limiting factor in grantmaking at these places not having enough money? Or is it more like wanting to avoid funding things that are net negative or flood the ecosystem with low-quality projects? I would have guessed more the latter, and in that case getting more money won't particularly help things like speed and funder chicken.
I'm sure you don't want to speak for either of those orgs, just speculating.
Thanks Carol! I think its reasonable to have these worries, and to be concerned that money doesn't solve them (these orgs need to execute well, and more direct solutions will be needed in some cases). I think there's reason for optimism here, but I don't want to speak for any orgs or base things on private info.
Perhaps another example: it seems suboptimal that Joey Savoie, after having created a successful product in Charity Enterpreneurship, is not rich as a result. Sure, he can leverage his connections and skillset and reputation to start a philanthropic advising org (Elevate) and continue to direct funding towards where he things might be impactful, but instead, what if he had just become rich, and had the ability to fund things without having to persuade his donors?
I don’t really want to focus on an individual’s career/financial decisions. I agree that in a just world, successful impact should be rewarded handsomely. But fwiw on the practical questions, I’m reasonably confident that: