This is an important and valuable question, thank you for raising it. I'll split my observations into two effects:
- Malthusian effects
- Benefits of scale
Malthusian effects
Other responses have referred to Malthusian effects, by which I mean the concern that with only finite resources, the resources will be spread between more people, and each person will have a worse quality of life.
Benefits of scale
Creating another person doesn't only create another mouth to feed. It also creates another source of ideas and creativity.
For example, each new birth has the potential to become another Norman Borlaug (who is claimed to have saved a billion lives through his research).
Even if 999,999 new people fail to come up with a ground-breaking innovation which makes the world better, if the millionth person does, it could allow everyone to benefit.
Of course, the flipside is that any new person could be the next Hitler/Stalin/insert-your-favourite-bad-guy-here.
Are the Norman Borlaugs winning over the Hitlers?
If you believe the data that seem to suggest that the world has been getting better over recent centuries, the answer seems to be yes.
There's also benefits around the fact that niche interests/needs are better accommodated at scale. If 0.01% of the population has a rare disease, and the population is 10 billion people, that's a million sufferers -- enough scale to incentivise scientific research. And if successful, maybe everyone is cured/treated. For a significantly smaller population that disease may remain untreated for a very long time.
Do the benefits of scale win over the Malthusian effects?
I don't think this is obvious, but I'm inclined to think the benefits of scale win.
If we look at recent examples of challenges that humanity has faced, human ingenuity has managed a few good successes (the aforementioned example of Norman Borlaug and dwarf wheat; the cost effectiveness of solar power has improved dramatically in recent years; smallpox eradication; saving the ozone layer). Don't get me wrong, we still have more to do! But that suggests we want more brains, not less.
Furthermore, decisions we make today should be based on how the benefits of scale will work in the future, not how they were in the past. Will we be better able to use our ingenuity to solve big problems in the future? Some would argue that AI will make us better able to explore creative new solutions (not that everyone will agree on this).
Lastly, and this isn't really answering your question, but rather picking up on a comment of yours. You said that the idea that saving lives makes the world better is a "core assumption of the effective altruism movement". I don't think this is correct. EA is a movement built around using evidence and reason to do good. If the evidence showed that saving lives was bad, the essence of EA would be unchanged. Furthermore, lots of the practice would be unchanged too -- a lot of EA activity is not linked to saving lives.
I am not here to ask if Earth can have 20 billion people or not, I'm here to ask about the dilemma of saving lives might not always be good. If you want, you can up the number to 200 billion people and see that Earth can't handle that number. At a point, you must say that saving more people is not a good thing, and it will just be a bad thing. And now comes the dilemma: if you say that saving people is not for the sake of saving people, you might be justifying murder (or preventing births) as “the most effective thing to do.”