Comments
I have the impression that the most effective interventions, especially in global health/poverty, are usually temporary, in the sense that you need to keep reinvesting regularly, usually because the intervention provides a consumable good; for example malaria chemoprevention: it needs to be provided yearly. In contrast, solutions that seem more permanent in the long-term (e.g. a hypothetical malaria vaccination, or building infrastructure), are typically much less cost-effective on the margin because of their high cost.
How do we balance pure marginal effectiveness vs eventually moving towards more permanent solutions? Could it be that by overly optimising for marginal cost-effectiveness, we might be missing a better ‘global maximum’ in the utility landscape, but we just need to descend from the current ‘local maximum’ to be able to get there eventually?
If AI agents use LLMs as their brain, how much of the harm that they end up doing is up to the chance of which token gets sampled from the output distribution?
As a dumb example, say some agent using an LLM as its backbone is reasoning "[...], so humans deserve to", and the LLM's output distribution for the next token assigns 98% probability to "die" and 1% to "live". The reasoning that follows will be influenced by which of these gets sampled.