[Download the PDF report.]
Author: Kaitlyn Hobbs
Review: Morgan Fairless & Mo Putera
Date of publication: December 2023
Research period: Research Training Program 2023
This report was conducted within the pilot for Charity Entrepreneurship’s Research Training Program in the fall of 2023 and took around eighty hours to complete (roughly translating into two weeks of work). Please interpret the confidence of the conclusions of this report with those points in mind.
We are also grateful to the experts who took the time to offer their thoughts on this research.
For questions about the structure of the report, please reach out to [email protected]. For questions about the content of this research, please contact Kaitlyn Hobbs at [email protected].
With a large burden of preventable and treatable diseases in Sub-Saharan Africa (SSA), mass media offers a means to relieve burdens by increasing awareness of specialised topics and encouraging health-improving behaviours (Roser et al, 2021). Types of media that can be leveraged for this purpose may include radio broadcasting, television shows and series, internet videos, or a combination of several means to deliver widespread messaging.
In this report, I propose a theory of change for using mass media to increase awareness on and, in turn, behaviours addressing preventable or treatable diseases. I also assess the evidentiary support for this intervention method. I found that while there is ample evidence for reaching large audiences with widespread media, robust support falls short on impacting health outcomes.
Using a geographic weighted factor model, I suggest Tanzania as a potential country for a new intervention, with a focus on increasing health-seeking behaviours that ameliorate neonatal morbidities and mortalities. That said, the application of this intervention method by nonprofits and non-government organisations seems to be over-saturated. Instead, I propose that funding an existing organisation would be more effective than starting a new one.
MMI: Mass Media Intervention
NGO: Non-government organisation
DMI: Development Media International
FEM: Family Empowerment Media
GW: GiveWell
FP: Founders Pledge
CEA: Cost-effectiveness analysis
BOTEC: Back-of-the-envelope-calculation
CE: Charity Entrepreneurship
NCD: Non-Communicable Disease
DALY: Disability-Adjusted Life Years
WELLBY: Well-Being-Year
SSA: Sub-Saharan Africa
COM-B: Capability, Opportunity, and Motivation for Behavior change
GBD: Global Burden of Disease
FSI: Fragile State Index
RCT: Randomised Control Trial
DHS: Demographic and Health Surveys
With intention to elicit a health-seeking or health-improving behaviour change, mass media messaging is often aimed at individuals who are capable of taking action for themselves or others, such as a caregiver. A caregiver may be formal, like a nurse, or informal, like a family member or guardian. In the context of family planning and child survival, media messaging has typically been aimed at women of reproductive ages (15-49 years of age). Equipping caregivers with accurate health information may increase timely intervention and ameliorate rates of morbidity and mortality. Several programs of this nature already exist and some have reported successful changes in behaviour outcomes.
The purpose of this report is first, to map a generalised landscape of using mass media interventions (MMIs) targeting informal caregivers for relieving health burdens and second, to investigate the feasibility of starting a new charity that implements mass media awareness campaigns in a neglected area. Specifically, Section 3 scopes the application of a new charity and proposes a theory of change; Section 4 summarises evidentiary support for key uncertainties in effective implementation; Section 5 details the geographic weighted-factor model designed for identifying regions in which a new charity may operate; Section 6 highlights where analogous organisations already exist; Section 7 evaluates the cost-effectiveness of a new charity and compares external estimates of current organisations; Section 8 and Section 9 discuss considerations for starting a new charity or funding existing ones, respectively; and Section 10 and Section 11 draw conclusions based on the content detailed throughout the report.
Founders Pledge, Charity Entrepreneurship (CE), and another researcher of the CE Research Training Program (RTP) have conducted shallow dives that estimate the promise of MMIs. In brief,
The CE research team also has thought through behavioural frameworks for MMIs and their expected influence, limitations, and designs. This becomes an important consideration for two assumptions in the theory of change proposed in this report: Assumption 9: Action is taken as a result of being informed/would not be taken without media information and Assumption 10: No barriers inhibit access to health care.
Finally, Givewell (GW) has produced a report reviewing three MMI programs: Population Media Center, PCI Media Impact, and Development Media International. Based on one quasi-experimental design and one randomised control trial (RCT), GW did not find enough evidence to support cost-effectiveness of the investigated interventions. GW points to confounding variables in a quasi-experimental report (Rogers et al, 1999), including two similar programs being aired, and has since received responses from Population Media Center that cites challenges in conducting RCTs for mass media interventions and concerns on overlooking other pertinent studies and strengths in the reviewed study by GW.
There are two approaches to developing a theory of change around using mass media to increase caregiver awareness: first, current efforts could be scaled-up in other regions. This approach would involve focussing on age groups currently addressed (mainly children under 5 years of age) but varies geographically. Second, new media content addressing neglected causes contributing to a high burden of disability-adjusted life years (DALYs) could be crafted and either target similar geographies or regions with the greatest need. This approach may target ages less commonly addressed, such as children aged 5-14 years. An algorithm used for scoping the intervention in this report is detailed below.
Although many alternative approaches to scoping down the focus of MMIs exist, I adopted following algorithm to identify geography, age, sex, and intervention:
In simpler terms, the selected MMI should focus on the disease that has the greatest burden in a particular age group of a country that scored highly in the Geographic Weighted Factor Model (GWFM). I chose this course of steps as it prioritises the scale of health burdens first, then caters a target behaviour to a region’s greatest need.
To identify where in the world MMIs may be most suitable, I assessed the global scale of disease according to the 2019 Global Burden of Disease database. Africa bears the greatest disease burden (highest disability-adjusted life years in counts and rates per 100,000 people) and generally, low to mid-range health expenditure per capita (Roser et al, 2021). As such, the remainder of this report focuses on assessing countries for implementation within Sub-Saharan Africa (SSA).
In line with the two approaches highlighted in Section 3 Theories of change, the following age groups were assessed using data from the 2019 Global Burden of Disease database: 0-4 years (both sexes), 5-14 years (both sexes) and females aged 15-49 (see Section 6.1.1 Burden).
Type of disease
Mass media for caregiver awareness typically target diseases affecting youth; that said, other burdens resulting in disability-adjusted life years (DALYs) may also be ameliorated through mass media. Worldwide, the total disease burden is shifting from communicable, maternal, neonatal, and nutritional diseases to non-communicable diseases (NCDs) and is noted to be attributed to rising incomes and standards of living. This trend is observed in Africa as well; however, whether or not increased awareness would continue this trend would provide a more comprehensive understanding of importance as this category still comprises over two-thirds of the DALYs per 100,000 people in SSA. For this report, the age group bearing the greatest burden of all-cause DALYs within SSA informed the most suitable focus for media content.
Type of health advice
Within the context of health, one consideration that influences which disease to focus media content on is whether increased awareness is more effective at prevention or treatment of illnesses. The type of disease would best inform whether prevention or seeking treatment is more suitable. Further research on comparing effects on population health of increased awareness and early identification to disease prevention, family planning programs, maternal health, child development, and nutrition could also provide a more complete understanding of which diseases could have the greatest reduction in DALYs through addressing behaviour change.
Colour-coding of certainty level are as follows:
(Green) >85% Quite certain, evidence review would provide additional strength in support.
(Orange) 50%-85% Fairly uncertain, evidence review would be helpful.
(Red) <50% Quite uncertain, evidence review is necessary.
Solid lines represent a causal chain of primary activities that focus on creating direct change. Dashed lines represent secondary activities, focusing more on monitoring and evaluation of causal chain effects.
#RTP23H10: (ER) Mass Media for Caregiver Awareness
Overall, evidence on MMIs efficacy in increasing knowledge, changing behaviours, and improving health outcomes of a target population is somewhat promising but could be further substantiated with more statistically powered evaluations. A 2014 systematic review on MMI’s for increasing child survival in low and middle income countries (LMICs) observed a publication bias towards successful interventions, which remains accurate for my review of publications since (Naugle & Hornik, 2014).
This evidence review is primarily informed by a meta-analysis of contraceptive use in SSA, a systematic review of MMIs targeting family planning and child survival, and a recent RCT conducted on an MMI in Burkina Faso (Babalola et al, 2017; Naugle & Hornik, 2014; Sarrassat et al, 2018). Additional evidence was referenced for comprehensivity. That said, the review conducted for this report is not extensive and likely overlooks other pieces of plausible evidence. As such, it should be kept in mind that the review likely suffers from publication bias and that quantitative methods used vary.
Counterfactuals of delivering mass media by a charity
Value over replacement. Little evidence was found to suggest that a government would invest in mass media for caregiver awareness on its own, although mass-media for health awareness has long since captured the attention of large NGOs like United States Agency for International Development (USAID) who continue to support mass communications for increasing health awareness in LMICs.
Alternative methods to encouraging health-related behaviours, such as in-person advocacy and knowledge-sharing sessions, are typically more costly and have a lessened expected reach of the target population. MMIs also have the advantage of being available in multiple languages and having control over content.
Reach of mass media
Evidence on target population reach varies depending on country of intervention, but we can generally expect a moderate to high exposure prevalence. A meta-analysis on mass media communication of contraception use in SSA estimated a 44% exposure prevalence, though this ranged drastically depending on country (Babalola et al, 2017). Of the publications reviewed by Naugle & Hornik. (2014), around one-third of the studies included in their review on MMIs addressing child survival reached between 61-100% of the target population. In studies not included in the systematic review, Sarrassat et al. (2018) detected a high proportion of both the target population being reached by radio broadcasts and recognition of media broadcasts by mothers interviewed in intervention groups, suggesting large media uptake.
Enhancement of knowledge about media content
Mass media has potential to be effective at increasing knowledge of a target population across multiple health topics, countries and intervention strategies but the most efficacious strategy is unclear. The meta-analysis report on contraceptive use in SSA, and systematic review on child survival report a positive effect of mass media on knowledge, attitudes and beliefs (Babalol et al, 2017; Naugle & Hornik, 2014). The meta-analysis reports a large effect size of contraceptive knowledge. In some studies, no effect was found on knowledge of media content in target populations while others did not evaluate this outcome. Studies on child survival MMI’s reviewed by Naugle & Hornik. (2014) included additional interpersonal efforts for knowledge-sharing to complement mass media communication, which would lead to confounding results. The RCT in Burkina Faso did not report on increased knowledge of health practices.
Whether absence of knowledge is the key driver to a disease burden is also dependent on the presence or absence in other health care barriers (see Assumption 10.).
Eliciting behaviour change
Generally, evidence suggests that mass-media can successfully elicit behaviour changes, although estimations of success rates are subject to a social desirability bias of self-reports and barriers such as the type of behaviour change and physical proximity to health care is expected to change efficacy. Around 81% of the publications on mass media for increased child-survival evaluated by Naugle & Hornik. (2014) reported a positive effect in eliciting the intended behaviour change, according to self-reports. The authors of Sarassat et al. (2018) also note an increase in self-reports of care-seeking and treatment, although this may be subject to socially desirable bias. The RCT in Burkina Faso did not detect a change in self-reported habitual behaviours but did note an increase in health-seeking behaviour (Sarrassat et al, 2018). Still, positive results were limited to households living with 5 km of a health facility. Further, a positive association between knowledge of pneumonia symptoms and care-seeking was observed by Glennerster et al. (2021). Bowen et al. (2013) estimates a 6.6% increase in mosquito bed-net use as a result of mass media (music video, public service announcements, and celebrity spokespeople) in Cameroon according to their propensity score matching model. Though it is intriguing that a habitual change (as opposed to a one-time change) in behaviour was reported, there is concern of social desirability bias of the in-person survey conducted for the analysis and the low power of quasi-experimental design.
Media uptake and consequential changes in behaviour are further detailed in Assumption 9.
Evidence supporting effectiveness of behaviour changes as a result of MMIs is mixed. This is largely attributed to a lack of high-powered studies demonstrating an effect of behaviour change on health outcomes (Naugle & Hornik, 2014). While randomised-control trials (RCTs) provide high-powered insights to causal relations, authors of the 2014 systematic review on MMIs propose their inapplicability for mass-media due to desirability of large scale and long-term campaigns (Naugle & Hornik, 2014). The authors instead suggest combining several methods to evaluate the impact of intervention on behaviour change. Researchers attribute challenges in conducting RCTs on MMIs to difficulty isolating geographies from the reach of radio broadcasts (Murray et al, 2018). Of those that exist, generalising results across causes and interventions remains largely uncertain.
Expected impact on health outcome
Evidence on the impact of MMIs on child survival is mixed but the most powered study shows no reduction in child mortality and is supported by other studies (Sarrassat et al, 2018). Kasteng et al. (2018) also reports no effect on reduction of child mortality by single-cause MMIs. In contrast, Murray et al. (2018) used the Life-saving Statistical Tool (LiST) to model child and maternal mortality reduction as a result of the clustered randomised trial conducted in Burkina Faso by Sarrassat et al. (2018). Their model suggests a 7.1% and 3% reduction of child and maternal mortality rate, respectively. DMI’s involvement in the study, several listed assumptions in the model inputs, and overall theoretical approach (as opposed to relying on empirical data) reduces the reliability of the study in informing my stance on MMI efficacy. My lack of understanding of LiST’s modelling approach also reduces my ability to accurately assess these results. A quasi-experimental study found that knowledge of pneumonia symptoms did not necessarily lead to increased care-seeking in four out of six Sub-Saharan countries evaluated (Noordam et al, 2017).
Evidence supporting marginally increased contraceptive use appears to be more promising and statistically powered. Glennerster et al. (2021) report a 5.9% increase in contraceptive rates in the treatment group relative to control, according to their RCT. An independent survey conducted in Kano, Nigeria reports a 75% increase in contraceptive use 11 months after Family Empowerment Media’s (FEM) family planning radio campaign began. Results from the survey are less statistically powered than the RCT results so a marginal increase in contraceptive use holds more evidentiary weight.
Lasting effects
Lasting effects of MMIs on behaviour and health outcomes are inconclusive as their evaluation is neglected in the studies reviewed. MMIs may have lasting behavioural effects beyond the period of intervention that are not evaluated in the reviewed studies. One study from 1990 reported a decrease in knowledge and habitual behaviour change three years after campaigning, but holds weak evidentiary weight given the year of publication (Naugle & Hornik, 2014).
In attempts to maximise uptake of media information and behaviour changes, approaches to delivering information can be modified according to the goal and primary reasons for the desired behaviour not being exhibited. Comparative evidence on the efficacy of such approaches was not explored, although several models can be leveraged. Counterfactual estimates on whether a behaviour change would occur without media intervention are also not explored due to time limitations.
Theoretical approaches to mitigate concern
Theoretical models exist that assist in identifying a wide-range of potential barriers; for example, the COM-B (Capability, Opportunity, and Motivation for Behavior change) model is a categorization of constructs that affect behaviour and is detailed in Assumption 10: No barriers inhibit access to health care. Other theoretical models for behaviour change exist but were not explored.
Study Examples
DMI incorporates storytelling of a protagonist either overcoming obstacles through a behaviour change or revising their goals in order to evoke emotional response to media campaigns (Glennester et al, 2021). Other tactics exist and could be explored in depth once the behaviour targeted for change is determined. The Behavioural Insights Team’s (BIT) EAST framework may guide details of the approach.
FEM approached maximising media uptake by profiling their listeners through qualitative analysis (Family Empowerment Media, 2021). They developed six general personas to characterise their audience for targeted messaging.
Evidence does not support this assumption to hold in practice instead, it highlights several key considerations impacting efficacy of MMIs. For ease of organisation, barriers to accessing healthcare noted in studies are organised according to the Capability-Opportunity-Motivation-Behavior (COM-B) model. COM-B proposes several considerations that may enable or inhibit accessing appropriate health care. It is noted by Mayne (2017), a working paper on the COM-B Theory of Change Model, that not all capability, opportunity, and motivation assumptions need to be addressed in an intervention. Exact considerations may become apparent after an intervention cause and geography is selected.
The following examples of COM-B model components found in publications about MMIs vary in terms of evidential strength. Most barriers were identified through surveys that are specific to the region in which they were conducted and are not expected to be reliably generalised. The Demographic and Health Survey (DHS) in SSA is likely the strongest survey in terms of generalisability given its large sample size and coverage.
Physical capability
Physical access to health care is likely a crucial consideration for efficacy of improved care-seeking and receiving timely health treatments.
Psychological capability (e.g. knowledge/understanding the consequences of not taking action and the benefits of taking action)
Social opportunity (e.g. pressures from family or community, fear of being judged, perceived norms and expectations.)
Physical opportunity (e.g. cost and availability of intervention method(s).)
Reflective and automatic motivations (challenges presented by behaviours that involve evaluation and planning, e.g. belief systems; challenges presented by behaviours that involve emotions or impulses, e.g. habits and desires.)
Little evidence was found on unintended effects from mass media campaigning within the studies reviewed. Nonetheless, possible negative effects can still be considered. Some considerations include: a heightened sense of anxiety or other emotional dysregulations due to either tone of messaging or increased knowledge of harmful consequences to inaction, misinterpretation of messaging or spread of misinformation, and rise of social judgements as a result of campaigning. Finding quality evidence on this and other potential flow-through effects were not prioritised with the time spent on this report. It’s possible that these considerations could be mitigated through strategic content design and delivery. With more time, expanding a search for evidence on unintended effects in other subject areas, such as the psychological impacts of widespread health information during COVID-19 (Giri et al, 2021), could serve as proxies for this application.
#RTP23H10: (GWFM) Mass media awareness for caregivers
Neonatal disorders in Tanzania was the top candidate for mass media intervention according to the geographic and cause assessment. The following weighted geographic assessment evaluates country-level suitability for a MMI in SSA. It considers the scale of disease burden as well as approximations of neglectedness and tractability. Weights of each variable reflect both the quality of data used and uncertainty associated with proxies.
Section 4.1 overviews the general approach to evaluating countries, including Burden, Tractability, and Neglectedness. Top candidate countries and their disease burdens are reported in 4.1.4 Results for children under 5 (both sexes), children 5-14 years (both sexes) and females ages 15-49 years. The cause of the greatest DALY burden for each country informs which health conditions to target in the intervention. I advise spot-checking top candidate regions by considering whether a country has experienced a coup d’etats as it may impact media control, cooperation and overall tractability.
I assessed 46 countries in SSA using a geographic weighted factor model (GWFM). The total score for a given country was given by:
Where,
For ease of interpretation and comparisons, the total score was normalised using z-scores.
Table 1. GWFM parameters and their associated weights and normalisation method.
Parameter | Weight | Normalisation |
Burden | 0.6 | Log |
Tractability | 0.2 | Capped Z-score |
Neglectedness | 0.2 | Negative log of absolute value |
The rate of all-cause DALYs per 100,000 population was assessed to identify burden. DALYs also include years lost due to death, making it a more comprehensive indicator of disease-related burden; however, this analysis could be repeated with mortality rates only, if death-aversion is more heavily valued.
DALYs per 100,000 were assessed for communicable diseases, non-communicable diseases, and nutritional disorders. The category “other non-communicable diseases” was excluded from the comparisons because it was an aggregate of several smaller diseases. Disaggregating this category could be done with more time in the future.
I estimated tractability by averaging several indicators of connectivity and adding WHO’s Health centres per 100,000 indicator and the Fragile State Index.
4.1.2.1 Degree of Connectivity
Since the means of delivering media content could vary (for example, using radio, internet, or mobile messaging), several variables encompassing multi-modal delivery were considered:
Due to incomplete data on individual and household usage, the maximum value across all three variables was used as a proxy for connectivity.
A potential spillover effect from living in an intervention region was proxied using the percentage of inhabitants within range of at least a 2G signal. This comprised only 10% of the connectivity estimate (5% of the total tractability estimate) due to uncertainty of the expected effects.
4.1.2.2 Health Centres per 100,000
Is defined as “Number of health centres from the public and private sectors, per 100,000 population” and serves as a proxy for health service availability. Health centres were chosen instead of hospitals as they typically offer services that are non-urgent, although this may vary by country.
It is expected that this is somewhat an underestimate of health service infrastructure availability. With additional time, additional metrics for density of other health infrastructure and services or quality of care could be included to improve representation of access to adequate health care.
4.1.2.3 Fragile State Index
The Fragile State Index (FSI) represents the general volatility of a nation based on several indicators on social, political, economic, and cohesion.
4.1.2.4 Data Vintages & Quality
Data was taken from the International Telecommunications Union (averaged from 2018-2022), World Bank Development Indicators (2021), WHO's Health centres per 100,000 (2013), and the Fragile State Index (2023). Changes that occurred past the data vintage year may not be captured.
4.1.2.5 Missing data
4.1.2.6 Improvements
Data Sources
Logic/Application
Future Projections
Neglectedness was inferred by the reduction in DALYs from 2010-2019, reported by GBD. Countries with smaller reductions are considered to be in greater need of support. Neglectedness was normalised by taking the negative log of the absolute value of DALY rate reduced. This ensured that regions with smaller reductions gave a higher total score.
Improvements
Reduction in DALYS (from 2010 to 2019) - number of NGOs operating
estimated reduction in DALYs
Table 2. Top 5 countries proposed by GWFM for MMI targeting children under 5 (both sexes).
Rank | All Cause | Z-Score | Cause of Greatest Burden - DALYs per 100,000 (Communicable) | Cause of Greatest Burden (Communicable) | Cause of Greatest Burden - DALYs per 100,000 (Non-communicable) | Cause of Greatest Burden (Non-communicable) | Nutritional Disorders Count |
1 | Tanzania | 1.678943363 | 38,333.55 | Neonatal disorders | 1,165.82 | Digestive diseases | 5,370.57 |
2 | Togo | 1.609476358 | 36,574.01 | Neonatal disorders | 485.68 | Skin and subcutaneous diseases | 2,406.50 |
3 | Ghana | 1.399101421 | 36,491.93 | Neonatal disorders | 1,425.17 | Digestive diseases | 3,776.30 |
4 | Benin | 1.234103426 | 50,729.89 | Neonatal disorders | 651.03 | Neurological disorders | 3,661.75 |
5 | Guinea-Bissau | 1.131141199 | 48,477.55 | Neonatal disorders | 1,432.54 | Neoplasms | 3,404.93 |
Table 3. Top 5 countries proposed by GWFM for MMI targeting youth aged 5-14 years (both sexes).
Rank | All Cause | Z-Score | Cause of Greatest Burden - DALYs per 100,000 (Communicable) | Cause of Greatest Burden (Communicable) | Cause of Greatest Burden - DALYs per 100,000 (Non-communicable) | Cause of Greatest Burden (Non-communicable) | Nutritional Disorders Count |
1 | Tanzania | 1.64550585 | 1,205.22 | Enteric Infections | 745.51 | Mental Disorders | 1,368.07 |
2 | Togo | 1.627004669 | 1,902.86 | Neglected tropical diseases and malaria | 767.86 | Mental Disorders | 1,427.84 |
3 | Eswatini | 1.255881021 | 2,865.05 | HIV/AIDS and sexually transmitted infections | 747.66 | Mental Disorders | 627.87 |
4 | Ghana | 1.252129594 | 1,571.56 | Neglected tropical diseases and malaria | 706.16 | Mental Disorders | 1,056.06 |
5 | Benin | 1.131992961 | 2,074.81 | Neglected tropical diseases and malaria | 738.13 | Mental Disorders | 1,156.05 |
Table 4. Top 5 countries proposed by GWFM for MMI targeting women in the reproductive age range (15-49 years).
Rank | All Cause | Z-Score | Cause of Greatest Burden - DALYs per 100,000 (Communicable) | Cause of Greatest Burden (Communicable) | Cause of Greatest Burden - DALYs per 100,000 (Non-communicable) | Cause of Greatest Burden (Non-communicable) | Nutritional Disorders Count |
1 | Ghana | 1.399509104 | 4,568.86 | HIV/AIDS and sexually transmitted infections | 1,884.91 | Mental Disorders | 473.86 |
2 | Togo | 1.337027461 | 3,783.00 | HIV/AIDS and sexually transmitted infections | 2,028.00 | Mental Disorders | 412.80 |
3 | Senegal | 1.29656711 | 2,791.47 | Maternal disorders | 2,212.92 | Mental Disorders | 593.12 |
4 | Benin | 1.230991983 | 2,676.21 | Maternal disorders | 2,413.50 | Neoplasms | 381.54 |
5 | Angola | 1.171252959 | 7,225.61 | HIV/AIDS and sexually transmitted infections | 2,211.28 | Mental Disorders | 441.31 |
Given that the scale of DALYs per 100,000 children under the age of 5 is around 30-fold greater than that of children ages 5-14 and around 10-fold greater than the DALY burden of women aged 15-49, I chose to investigate the greatest cause of the burden of the top country in this category: neonatal disorders in Tanzania.
The following list of NGOs operating in LMICs were identified during research but is not expected to be comprehensive:
Other local initiatives likely exist and should be investigated depending on region of interest.
National Ministries of Health: Health ministries in sub-Saharan African countries may use mass media to raise awareness about various health concerns, but the extent and quality of their efforts was not investigated.
According to a Cochrane Review on pregnancy interventions to avoid preterm births (PTBs), a principal cause of neonatal morbidity and mortality, encouraging pregnant women to take folic acid vitamins, zinc, antibiotics for infections, and attending health appointments can reduce PTB (Institute of Medicine (US) Committee on Improving Birth Outcomes, 2003; Medley et al 2018). These examples of health-promoting behaviours are not extensive but may be relatively simple and low-cost to elicit through mass media.
Start-up Activities
First, the charity should identify a region within Tanzania to pilot a mass media intervention. The capital city of Dodoma may be a suitable start as it likely has health and broadcasting infrastructure as well as a high burden but I recommend investigating its suitability further. I also recommend connecting with Nutrition International to discuss their efforts and challenges in transitioning from iron and folic supplements to multimicronutrient supplements and provision to pregnant women in Tanzania.
Next, the charity should identify potential partners and build relationships through contacting the local health networks, governments, and radio broadcasters within a selected region of Tanzania.
Media content design
The media could either be designed and delivered by radio show hosts in collaboration with health experts while the charity facilitates the collaboration and other operations, or the charity could design the content with heavy input and validation from health experts.
Monitoring and Evaluation
The charity would implement monitoring and evaluation efforts, as overviewed in Feedback loops.
Before mass media is shared, baseline rates of health knowledge, health-seeking and health-improving behaviours should be determined by leveraging current health monitoring or implementing community surveys. Through a community survey, it’s also important to ask about current physical barriers, social beliefs, and other reasons that inhibit engaging in health-improving behaviours. This survey can be completed again at several time points throughout the intervention and be corroborated by rates of health-seeking behaviour recorded by health professionals. Finally, the overall associative effect of mass media on decreasing neonatal morbidities and mortalities can be determined through hospital records on the prevalence of preterm births.
This section summarises our concerns (or lack thereof) about different aspects of a new charity putting this idea into practice.
Table 6: Implementation concerns.
Factor | How concerning is this? |
Talent | Low Concern |
Access to information | Moderate Concern |
Access to relevant stakeholders | High Concern |
Feedback loops | Low Concern |
Funding | Moderate Concern |
Scale of the problem | High Concern |
Neglectedness | Moderate Concern |
Execution difficulty/Tractability | Moderate Concern |
Negative externalities | Low Concern |
Positive externalities | Low Concern |
Several attributes of charity founders would be helpful, but I do not expect them to be particularly challenging to find. A founder should be personable, open-minded, and possess acumen in managing stakeholder relations and project management would also be valuable. It would be helpful if new charity founders were generalists with some background in health and/or media communications. Quantitative skills and experience in monitoring and evaluation may also be additional qualities of strong candidates.
Information
Access to health care monitoring systems is imperative for evaluating impact. This is expected to be of moderate concern as it depends on the following conditions:
Relevant stakeholders
Given the variety of partners (radio broadcasters, health facilities, local organisations, as well as local and/or federal governments), managing stakeholder relations is of high concern and would be a valuable attribute of charity founders.
Intermediate outcomes like reach, behaviour change and increased knowledge can be investigated by surveys. This is expected to give relatively quick and reliable feedback. Since radio listenership is already monitored by stations, detecting the reach of MMIs is not expected to be a concern. Ensuring that shows are airing may also utilise similar monitoring systems. A cross-sectional survey may also be conducted to detect whether knowledge, attitudes or beliefs towards the health concern have been influenced by the MMIs.
Measuring causality in health outcomes with confidence may be on the timescale of 1-3 years. While behavioural change may occur shortly after media is delivered, feedback loops for health outcomes may take a year before becoming significantly detectable in studies. The RCT run by Sarrassat et al. (2018) was 3 years long, and another RCT to be run by FEM will take 3 years. That said, the charity does not necessarily need to run an RCT to estimate impact and may instead rely on data provided by health systems on hospital visits and rates on mortality and morbidity.
According to the RTP CEA on starting a new charity,
The existence of a grantmaking organisation for MMIs leads me to think receiving fundraising in this field is feasible yet competitive. Media Impact Funders is a grantmaking organisation that maintains a database for funders and MMI programs that includes impact assessment designs and toolkits. They also house a curated collection of impact reports and host a media impact festival. Better insights on grantmaking in MMI is included in the organisation's report on Global Media Philanthropy.
The Bill & Melinda Gates Foundation also may be interested in supporting funding based on their support for Media Impact Funders’ research.
The high estimated cost-effectiveness of DMI and FEM may attract funding from organisations affiliated with the Effective Altruism community, such as GiveWell. For scaling DMI, the primary concern is supporting the operational costs estimated, which are around 13X more than the CE suggested annual operational cost of $250,000.
The scale of the problem is of high concern as the burden of disease for various age ranges in Sub-Saharan Africa is apparent and unlikely to be resolved without intervention (recall 2.1 Cause area).
Given the large number of organisations using mass media for impact, the methods do not seem neglected; still, specific causes and geographic areas are not addressed. While there are many organisations implementing MMIs in SSA, certain diseases remain overlooked and could be further examined.
Tractability is a moderate concern and is region-dependent. As mentioned in 6.1.2 Tractability, recent political events that could influence success of MMIs (such as government takeovers that lead to banning specific news sources) may not be captured. It’s advised to investigate the latest conditions in a selected intervention region.
Negative externalities: Given the evidence available, I do not expect negative flow-through effects to outweigh the benefits of implementation. Potential consequences may include:
Positive externalities: Assuming health interventions are adequate and available, additional indirect returns (aside from death aversion) may include:
The following uncertainties remain after evidence review:
Initially, I had attempted to compare cost-effectiveness between current interventions, scaling-up current interventions, and starting a new intervention. This quickly became nuanced (as highlighted in Section 7.1.2 Fickleness of CEAs for comparative purposes) so I redirected my focus to comparing published cost-effectiveness analyses (CEAs) on FEM’s family planning campaigns and DMI’s campaigns addressing child mortality. After, I conducted my own CEA on starting a new charity in Tanzania that estimates cost-effectiveness of increasing awareness for causes of neonatal mortality. My initial efforts to evaluate DMI and FEM are preserved in the CEA workbook but are not commented on in this report.
Previous CEAs on DMI and FEM have been presented by Founders Pledge, DMI and FEM and are summarised in the table below. Rethink Priorities has published a review on FEM and Founders Pledge’s cost-effectiveness analyses that makes several suggestions for improving estimates and is ultimately in agreement that FEM is cost-effective.
Table 5. Summary of CEAs and back-of-the-envelope calculations (BOTECs). Dollar evaluations are reported in USD 2023 unless indicated otherwise. Bottom cells highlighted in blue represent conversions to $/DALY using Founders Pledge’s conversion (provided by RTP), where applicable.
| BOTEC Author | FEM | DMI - Child Survival | New Charity |
| Founders Pledge (Medium Scenario) | $5.89 per WELLBY
25X GiveDirectly | $35.14 per WELLBY
4X GiveDirectly | - |
$22.65 per DALY | $135.15 per DALY | ||
| DMI Report | - | $142.09 per DALY* | - |
|
- | $7-$27 per DALY§
$602 per death averted§ |
- |
| The Life You Can Save Website | - | $30-$600 per life saved§ | |
| FEM (using GiveWell’s BOTEC) | 26.9X GiveDirectly | 1.1X GiveDirectly | 4X GiveDirectly** |
| RTP | - | - | $378.10 per DALY |
* Societal costs included, adjusted for inflation since 2015.
** For 5 West African Countries.
§ Unclear whether values are reported in 2023 USD
Caution should be taken when comparing CEAs reported by different organisations as inputs, outcomes measured, discounts, and other considerations may vary, adjusting valuations. A few examples of discrepancies I observed in cost-effectiveness reports for MMIs include:
Taken together, variability in absolute estimates proposed between different organisations are challenging to directly compare with confidence. That said, evaluations made by the same organisation across several hypotheticals may be most reliable as they apply the same structure to all scenarios, potentially controlling for some of these inconsistencies. As such, I chose to rely on FEM’s BOTEC and Founders Pledge’s BOTEC, both of which have arrived at the same relative conclusion; FEM is reported to be more cost-effective than DMI. RethinkPriorities also proposes similar relative estimates despite listing uncertainties.
#RTP23H10: Mass Media for Caregiver Awareness - Cost-effectiveness Estimate
Although not directly comparable to cost-effective estimates of FEM and DMI, the remainder of this section details the CEA I’ve developed for starting a charity in Tanzania and highlights comparisons between other CEAs.
The RTP BOTEC suggests a geometric mean of $378.10 USD 2023 per DALY averted in Tanzania with a best case estimation of $89.54 and worst case of $2,197 per DALY averted. The wide range can be explained by the compounding uncertainties in the worst-case scenario, which were kept conservative. Estimations heavily rely on details of the theory of change and, although they consider societal costs incurred, do not include long-term societal costs saved.
7.2.1 Effects
A new mass media awareness charity in Tanzania addressing neonatal deaths would affect women aged 15-49 years who own a radio, are expected to be reached by radio broadcasting, and are likely to exhibit a change in behaviour as a result of the media messaging.
[See PDF report for image]
The effects of increased health-seeking behaviour are expected to reduce neonatal deaths annually.
7.2.2 Costs
The following costs were included in the CEA:
| Start-up Costs | Type of input | Citation for input |
| Total fixed charity start-up costs | RTM told me to use this number | Best case: Fixed costs of starting the charity. $125k set up costs assumed. We hold this factor constant across all CEAs Worst case: 2x best case. |
| Operational Costs | ||
| Overhead/operations | ||
| Ongoing annual charity organisational/overhead costs | RTM told me to use this number | Costs roughly based on 4 staff members plus additional buffers. We hold this factor constant across all CEAs. This is assumed to encompass monitoring and evaluation, research, and implementation costs. |
| Societal costs | ||
| Average costs to government per user | Citation | Costs incurred to health care facilities for providing care. Taken from DMI’s estimates by Kaleng. Et al (2018). "The provider (societal) cost per additional health facility visit was $18.3 ($21.6)". |
| Annual cost to beneficiary | Citation | Costs incurred for adopting MMI target behaviour (e.g. care-seeking). Includes transport. Taken from DMI's value on pg 5 of Kaleng. Et al (2018) (household costs, not including societal costs). |
| Time adjustment | ||
| Years of operation for the organisation | Estimated Input | Best case: 3 years (taken from DMI in Burkina Faso) Worst case: 5 years. |
| Counterfactual costs of employees | ||
| Counterfactual impact of employees per year | RTM told me to use this number | The counterfactual impact of the employees can depend on their current donations, what job they are currently in and what other positions they could have taken. We hold this factor constant across all CEAs for new ideas as we are assuming that the staff for each new charity would have the same counterfactuals. |
| Number of Founders | RTM told me to use this number | Costs roughly based on 4 staff members plus additional buffer. We hold this factor constant across all CEAs for new ideas as we are assuming that the staff for each new charity would have the same counterfactuals. |
| Number of employees | NA | Assumes 0 because we are using radio hosts who are already employed |
| Time costs of volunteering (per hour) | RTM told me to use this number | We hold this factor constant across all CEAs for new ideas as we are assuming that the staff for each new charity would have the same counterfactuals. |
| Number of volunteers | RTM told me to use this number | We hold this factor constant across all CEAs for new ideas as we are assuming that the staff for each new charity would have the same counterfactuals. |
| Number of hours per volunteer per year | RTM told me to use this number | We assume 5 hours of volunteering time per person over 52 weeks of the year |
I was not able to find costs or estimates for airing a radio show in Tanzania so the CEA insteads assigns a budget within which 40% (worst case) to 80% (best case) of women ages 15-49 years who own a radio are reached by messaging. Societal costs were proxied from Kaleng et al. (2018) on DMI’s campaigns in Burkina Faso and are likely to change estimations in the context of Tanzania.
On Friday, November 24, 2023, I shared this research in a 5-minute presentation to six other participants of CE’s Research Training Program during a decision-making meeting. The average of all votes concluded that this intervention idea seems promising but funding should be directed towards current organisations in this space as opposed to starting a new one.
There is value in supporting FEM’s programs as the organisation intends to run additional RCTs using their new radio technology. This would enable more evidence-backed insights on the effectiveness of MMIs to be developed.
Family Empowerment Media
According to a thread in the EA Forum, in March 2023, GiveWell recommended granting $500,000 to FEM. FEM states a financial gap of $342,000 for 2023.
Development Media International
Kaleng et al. (2018) propose scale up scenarios for DMI ranging from $7 per DALY averted in Malawi to $27 per DALY averted in Burundi. Costs are reported in USD 2015.
A supported theory of change, sufficient evidence for impact, promising cost-effective evaluations, and saturation of this type of intervention in LMICs ultimately informs the conclusion to direct funding towards existing organisations.
Overall, using mass media interventions to increase caregiver awareness in Sub-Saharan Africa seems promising given the burden, high potential for reach, sufficient supporting evidence (despite some evidentiary gaps on impact), and cross-validated cost-effectiveness.
Given the high concentration of current organisations using mass media to raise awareness for health concerns and the expressed need for further evidence on impacts of MMIs, introducing a new organisation may not be the best use of funds. Instead, investments could be made to (1) fill evidential gaps through RCTs, (2) improve efficacy of program delivery through innovative technology and approaches, or (3) scale-up effective programs.
As mentioned in 9 Directing funding towards an existing entity, there seems to be a comparative advantage to funding FEM, which aims to address evidential gaps through conducting studies while continuing to provide aid cost-effectively. Additional secondary research could help advise the direction of program expansion to address neglected diseases in areas with high burdens or to replicate current cost-effective programs in neglected regions.
With more time, the following concepts and publications would be investigated:
Hullo, this report is fascinating! Upvoted. Just a quick note that Founders Pledge estimate of DMI's child survival program is out of date (which totally makes sense, as we haven't published our full eval of DMI's child survival program).
We currently rate this program at 4x GD, not 12x GD- this is after applying a really strong adjustment (16%; internal valdity adjustment of 30%, external of 54%), to try and deal with the uncertainty from the Sarrassat et al. RCT. thanks!
Hi Rosie,
Thank you for taking the time to review and for the correction! I have updated the table in Section 7.1 to reflect this estimation reported in FP's BOTEC.