That's fair! It's just also reasonable for me or other non-PM forecasting folks to be concerned about people making the wrong inference. I'm currently set to pay for the SSPP and other forecasting work out of my own personal research budget next year, having not found other funding yet. I had a full proposal out to cG when they shuttered the stream, though there are some other possibilities I'm exploring.
Thanks! I agree, I'm also generally skeptical of small chance * big number things - I was not intending 1% as an anchor but as an open question - and not as a probability but as a concrete percent of the funding. For example, a big org uses forecasts, but perhaps they only use them in particular workstreams responsible for X% of funding, and those workstreams could be tracked. Then out of X%, how much do they move the needle?
Not Josh, and also conflicted through the Social Science Prediction Platform (though we had pretty minimal funding from EA sources), but I wonder if it would be worth pooling non-public projects we know of and making BOTE estimates of hypothetical impact. It’s tricky because I don’t know of any RCTs (though I’m working on one now). But I’m extremely confident that across us we would think of some combination of orgs/governments that collectively spend over $100 billion per year (… I can think of that alone) that are interested in forecasts in different ways. Now, imo the vast majority of places interested in forecasting are not going to do anything substantive with it, and it’s hard to know what it means for one of these places to integrate forecasts - for example, for an org spending $X, do forecasts inform 1% of their funding or what? Of the share they inform, how much do they move the needle? If estimates from people who work on forecasts may be optimistic (I'm not paid at all for it, but I choose to work on it because I think it’s useful), happy to describe the situation to an outside observer privately.
I think one should distinguish between several things here:
Prediction markets that make a lot of $ and don't really need more because they do just fine with the profit motive
People spending a lot of time on prediction markets to prove they are a good forecaster
Infrastructure integrated into specific use cases, such as: when a funder is interested in a question so much that they will pay for forecasts, to inform and improve their other funding, or when for structural reasons the institution that has reason to benefit from the forecasts cannot fund them itself (such as some other decision-makers who do not have the mandate to support forecasting and are restricted from spending money to support it but would use forecasts in their work flow), or basic research with positive externalities.
This post really belabours the first and second bullet point, perhaps because that is where a lot of money has gone to, but there can be a lot of value in the third.
The wisdom of crowds effect kicks in with very few forecasts. In the working paper I cite elsewhere in the comments, even 5 forecasts gets you pretty far along into the WoC effect, and 10 even more so. This is for asking people what they think, not prediction markets - the latter should, theoretically, require more forecasts, since seeing the implicit beliefs of others through the market price could lead to herding etc. But the wisdom of crowds effect kicking in for very small N is well established in the literature.
I have a different takeaway as you, though, that we only know about this effect - or about the biases people have and how to adjust their forecasts - because of work on forecasting. I don't know how we'd know this stylized fact without work on it. For the wisdom of the crowds effect specifically, perhaps you could stop funding early since that one is well known, but it's sufficiently surprising to most people that there could be value in showing it for more domains, and it is really just one example of what we learn more generally from research on forecasting - and these other results on how to optimally weigh forecasts can shrink error much more even after taking the wisdom of the crowds effect into consideration. (In our work, WoC gets you a ~60% reduction in the MSE, but other small adjustments lead to an improvement of an additional ~60% reduction in error compared to the WoC estimate, and those aren't even all the improvements we can make.)
Today I would never run an experiment without using forecasts to help with power calculations. And there is very recent work I'd use to adjust those forecasts, and we're collectively not near the optimum in terms of learning what we can learn to make more accurate forecasts or integrating them into workflows. As I said elsewhere in the comments, the claims in the OP are far too strong. Even your asking a few experts - that's something that could be improved on and integrated into workflows and is part of the titular "forecasting". (It reads to me kind of like: don't do forecasting, do this other thing which is itself forecasting and is informed by and improved upon by... forecasting.)
A more defensible claim imo would be that there are some projects that are self-supporting and those should not be funded, or that in some but not all cases if the market doesn't pay for it then it's not valuable (abstracting from coordination failures and other market failures, or the externalities of basic research).
I see forecasting catching on with researchers for experimental design, which could easily save a lot of money and help make more progress. Earlier this month we updated a working paper on forecasts using data from the Social Science Prediction Platform to explicitly include results demonstrating use in power calculations. If a year from now forecasts are used a similar amount to now in economics research then that would be evidence for your hypothesis but from my perspective the concept that forecasts could be used in this way has only just started to be socialized, at least in my field. I also personally know of at least a couple of large institutions seeking forecasts and am planning a RCT on how they affect decision-making in the field.
I think LLMs are making forecasting much cheaper and easier.
If humans don't take up the use of forecasts in decision-making as much as they "should", well, LLMs may be more likely to in their own pipelines.
That's not to say that every project previously funded around forecasting was a good use of money. I would probably agree with you regarding most of the projects you have in mind, while disagreeing with the title and framing which is way too broad.
Some things it could be other than hedonic adaptation (non-exhaustive):
- People being disappointed that it's not making a bigger difference to their lives. Most experts thought there would be somewhat larger effects, so it's plausible participants did as well and are already starting to be disappointed by year 2.
- Expectations more generally not lining up to reality. For example, when someone leaves a job, we ask them how likely they think it is that they will find another acceptable job within 6 months. Almost everyone says they think they will find such a job, but then only about a third of people do in that timeframe. (Having a harder time finding a good job than anticipated is exceptionally common in all walks of life.)
- If you have money to do more things and then start to do more things, some of those things are not going to work out for you and you may have unexpected expenses. For example, you buy a car, and someone breaks into it. You move housing units, and it costs more than you expected to move or the gas bill is too high.
- There is a lot of churn in who is low-income, and while the transfers represent 40% of baseline household income, income rises in both the treatment and control group over the course of the study. So the transfers represent a relatively smaller boost in years 2 and 3. (Still, you might think that a 40% boost from baseline would affect well-being for longer.)
- Dynamic effects on other outcomes. For example, over time, fewer people work. In general, leisure is a normal good, and I think that is also the case here, by revealed preference. However, people could choose the wrong amount of it for themselves (e.g., through over-optimism in job search) and not working could lead to unexpected expenses biting harder.
- The earliest part of the study coincided with the brunt of the pandemic. Maybe it's easier to make people happier with cash during a pandemic (though note the effect could also go the opposite direction, given it was also easier to get other cash payments during this time).
One thing that is interesting is that well-being seems to be trending negative before the end of the study - not significantly, but still a bit concerning. If the study had only lasted one year, we might have incorrectly concluded the transfers made people durably better-off in terms of these subjective well-being measures. If the study lasted longer, though, would we see a continued decline, or are the negative point estimates just before the end a function of the transfers ending soon? (E.g., via regret or a negative anticipation effect?)
I still think hedonic adaptation is a very strong contender to explain the patterns we observe, because it shows up all over the place in the broader literature, to the point where if a study doesn't find an effect like that I wonder why. But, there are other things it could be. Either way, one might still be surprised the effects on subjective well-being aren't larger than they are, even in year 1. And, it's always possible to think the transfers made a meaningful difference to people's lives even if they did not show up in subjective well-being measures.
(Preference utilitarians might have a slightly easier time than hedonistic utilitarians here: while preference utilitarianism is more associated with life satisfaction or domain satisfaction and hedonistic utilitarianism more associated with affect balance, and we don't see lasting positive impacts on any of those measures, the transfers do enable more choice and people prefer more cash to less. We didn't see anyone turn down the transfers.)
I would agree that econ has the potential to have more impact than a development studies degree, but neither program is an econ program. (Maybe that specific MPP has a lot of econ content, but MPP programs in general do not, and if this one does I would not know.)
If you are going to work for an international organization, either the MPhil or MPP would be fine but the MPhil might open more doors through name recognition.
Alternatively, there are a few master's programs out there that really focus on tech-ing people up. For example, USF has an applied economics and an IDE program that are well-regarded. A bunch of master's programs are trying to distinguish themselves on quantitative skills and I don't expect all of them to require an econ background, so maybe it's worth looking around more.
Even working for a year and applying to more quantitative programs saves you a year over doing two masters.
I went from the MPhil in Development Studies straight to an Econ PhD, and I know a few other scattered cases, though it was a long time ago so I'm not sure if that path would work these days, and it was pretty unusual even then. It may be doable depending on your undergrad coursework and where you're aiming - admissions committees care a lot more about having a math background than they care about having an econ background.
Question: are you looking to continue on to a PhD later, or go to the international organization / non-profit sector, or something else? For most work in international development you don't need a PhD and the MPhil would work just fine to get you in the door.
I wouldn't assume the MPP would significantly help you get into the MPhil in Economics. And if you're set on some kind of second master's after the MPP, but not after the MPhil in Development Studies, the opportunity cost of delaying your career by 2 years would be much higher than the difference in the course costs.
UofT also has a lot of faculty you could tap into if you need support.
That's fair! It's just also reasonable for me or other non-PM forecasting folks to be concerned about people making the wrong inference. I'm currently set to pay for the SSPP and other forecasting work out of my own personal research budget next year, having not found other funding yet. I had a full proposal out to cG when they shuttered the stream, though there are some other possibilities I'm exploring.
Thanks! I agree, I'm also generally skeptical of small chance * big number things - I was not intending 1% as an anchor but as an open question - and not as a probability but as a concrete percent of the funding. For example, a big org uses forecasts, but perhaps they only use them in particular workstreams responsible for X% of funding, and those workstreams could be tracked. Then out of X%, how much do they move the needle?
Anyway, happy to chat sometime!
Not Josh, and also conflicted through the Social Science Prediction Platform (though we had pretty minimal funding from EA sources), but I wonder if it would be worth pooling non-public projects we know of and making BOTE estimates of hypothetical impact. It’s tricky because I don’t know of any RCTs (though I’m working on one now). But I’m extremely confident that across us we would think of some combination of orgs/governments that collectively spend over $100 billion per year (… I can think of that alone) that are interested in forecasts in different ways. Now, imo the vast majority of places interested in forecasting are not going to do anything substantive with it, and it’s hard to know what it means for one of these places to integrate forecasts - for example, for an org spending $X, do forecasts inform 1% of their funding or what? Of the share they inform, how much do they move the needle? If estimates from people who work on forecasts may be optimistic (I'm not paid at all for it, but I choose to work on it because I think it’s useful), happy to describe the situation to an outside observer privately.
I think one should distinguish between several things here:
This post really belabours the first and second bullet point, perhaps because that is where a lot of money has gone to, but there can be a lot of value in the third.
The wisdom of crowds effect kicks in with very few forecasts. In the working paper I cite elsewhere in the comments, even 5 forecasts gets you pretty far along into the WoC effect, and 10 even more so. This is for asking people what they think, not prediction markets - the latter should, theoretically, require more forecasts, since seeing the implicit beliefs of others through the market price could lead to herding etc. But the wisdom of crowds effect kicking in for very small N is well established in the literature.
I have a different takeaway as you, though, that we only know about this effect - or about the biases people have and how to adjust their forecasts - because of work on forecasting. I don't know how we'd know this stylized fact without work on it. For the wisdom of the crowds effect specifically, perhaps you could stop funding early since that one is well known, but it's sufficiently surprising to most people that there could be value in showing it for more domains, and it is really just one example of what we learn more generally from research on forecasting - and these other results on how to optimally weigh forecasts can shrink error much more even after taking the wisdom of the crowds effect into consideration. (In our work, WoC gets you a ~60% reduction in the MSE, but other small adjustments lead to an improvement of an additional ~60% reduction in error compared to the WoC estimate, and those aren't even all the improvements we can make.)
Today I would never run an experiment without using forecasts to help with power calculations. And there is very recent work I'd use to adjust those forecasts, and we're collectively not near the optimum in terms of learning what we can learn to make more accurate forecasts or integrating them into workflows. As I said elsewhere in the comments, the claims in the OP are far too strong. Even your asking a few experts - that's something that could be improved on and integrated into workflows and is part of the titular "forecasting". (It reads to me kind of like: don't do forecasting, do this other thing which is itself forecasting and is informed by and improved upon by... forecasting.)
A more defensible claim imo would be that there are some projects that are self-supporting and those should not be funded, or that in some but not all cases if the market doesn't pay for it then it's not valuable (abstracting from coordination failures and other market failures, or the externalities of basic research).
I have at least three reasons to be hopeful:
That's not to say that every project previously funded around forecasting was a good use of money. I would probably agree with you regarding most of the projects you have in mind, while disagreeing with the title and framing which is way too broad.
Some things it could be other than hedonic adaptation (non-exhaustive):
- People being disappointed that it's not making a bigger difference to their lives. Most experts thought there would be somewhat larger effects, so it's plausible participants did as well and are already starting to be disappointed by year 2.
- Expectations more generally not lining up to reality. For example, when someone leaves a job, we ask them how likely they think it is that they will find another acceptable job within 6 months. Almost everyone says they think they will find such a job, but then only about a third of people do in that timeframe. (Having a harder time finding a good job than anticipated is exceptionally common in all walks of life.)
- If you have money to do more things and then start to do more things, some of those things are not going to work out for you and you may have unexpected expenses. For example, you buy a car, and someone breaks into it. You move housing units, and it costs more than you expected to move or the gas bill is too high.
- There is a lot of churn in who is low-income, and while the transfers represent 40% of baseline household income, income rises in both the treatment and control group over the course of the study. So the transfers represent a relatively smaller boost in years 2 and 3. (Still, you might think that a 40% boost from baseline would affect well-being for longer.)
- Dynamic effects on other outcomes. For example, over time, fewer people work. In general, leisure is a normal good, and I think that is also the case here, by revealed preference. However, people could choose the wrong amount of it for themselves (e.g., through over-optimism in job search) and not working could lead to unexpected expenses biting harder.
- The earliest part of the study coincided with the brunt of the pandemic. Maybe it's easier to make people happier with cash during a pandemic (though note the effect could also go the opposite direction, given it was also easier to get other cash payments during this time).
One thing that is interesting is that well-being seems to be trending negative before the end of the study - not significantly, but still a bit concerning. If the study had only lasted one year, we might have incorrectly concluded the transfers made people durably better-off in terms of these subjective well-being measures. If the study lasted longer, though, would we see a continued decline, or are the negative point estimates just before the end a function of the transfers ending soon? (E.g., via regret or a negative anticipation effect?)
I still think hedonic adaptation is a very strong contender to explain the patterns we observe, because it shows up all over the place in the broader literature, to the point where if a study doesn't find an effect like that I wonder why. But, there are other things it could be. Either way, one might still be surprised the effects on subjective well-being aren't larger than they are, even in year 1. And, it's always possible to think the transfers made a meaningful difference to people's lives even if they did not show up in subjective well-being measures.
(Preference utilitarians might have a slightly easier time than hedonistic utilitarians here: while preference utilitarianism is more associated with life satisfaction or domain satisfaction and hedonistic utilitarianism more associated with affect balance, and we don't see lasting positive impacts on any of those measures, the transfers do enable more choice and people prefer more cash to less. We didn't see anyone turn down the transfers.)
I would agree that econ has the potential to have more impact than a development studies degree, but neither program is an econ program. (Maybe that specific MPP has a lot of econ content, but MPP programs in general do not, and if this one does I would not know.)
If you are going to work for an international organization, either the MPhil or MPP would be fine but the MPhil might open more doors through name recognition.
Alternatively, there are a few master's programs out there that really focus on tech-ing people up. For example, USF has an applied economics and an IDE program that are well-regarded. A bunch of master's programs are trying to distinguish themselves on quantitative skills and I don't expect all of them to require an econ background, so maybe it's worth looking around more.
Even working for a year and applying to more quantitative programs saves you a year over doing two masters.
I went from the MPhil in Development Studies straight to an Econ PhD, and I know a few other scattered cases, though it was a long time ago so I'm not sure if that path would work these days, and it was pretty unusual even then. It may be doable depending on your undergrad coursework and where you're aiming - admissions committees care a lot more about having a math background than they care about having an econ background.
Question: are you looking to continue on to a PhD later, or go to the international organization / non-profit sector, or something else? For most work in international development you don't need a PhD and the MPhil would work just fine to get you in the door.
I wouldn't assume the MPP would significantly help you get into the MPhil in Economics. And if you're set on some kind of second master's after the MPP, but not after the MPhil in Development Studies, the opportunity cost of delaying your career by 2 years would be much higher than the difference in the course costs.