Comments
Tl;dr
This blog post outlines doebem's comprehensive methodology (full report here) for evaluating opportunities for effective donation in Brazil, following the Cause Prioritization Research and the In-Depth Cause Study. The process involves four phases:
- Mapping and Screening: Identifying and filtering organizations addressing prioritized causes.
- Pre-selection: Evaluating organizations based on specific criteria to narrow down to the most promising candidates.
- Organization and Intervention Assessment: In-depth assessment of selected organizations, including field visits and an analysis of institutional strength.
- Cost-Effectiveness Analysis: Estimating the social impact generated for each actual donation to every intervention, and comparing the results to finally recommend the best donation opportunities to our donors.
The final stage involves a detailed cost-effectiveness analysis, using controlled extrapolation of scientific evidence, and calibration factors to estimate the impact of said interventions. The goal is to identify and recommend the opportunities that generate the highest social impact per donated Brazilian Real in Brazil.
Introduction
A few months ago, on this forum, we shared the Cause Prioritization Research conducted in Brazil. Through an extensive analysis of pressing issues in the country, the report, available in English and Portuguese, utilized the ITN framework (Importance, Tractability, and Neglectedness) to structure our search for the priority causes. That study identified three strategic priorities: non-communicable chronic diseases, hunger and nutritional deficiencies, and lack of access to safe water.
In this blog post, we present the comprehensive methodology utilized in the subsequent steps of our research. In sum, our methodology consists of three consecutive phases:
After the Cause Prioritization Research (identifying the priority causes) and the In-Depth Cause Study (deeper research on each selected cause), we have the Evaluation of Opportunities for Donation. Its goal is to evaluate donation opportunities, identify the solutions in Brazil that generate the highest social impact per Brazilian Real donated to the prioritized cause, and support the decision of which opportunities for donation will be recommended to our donors. The full research report can be accessed here.
Methodology
The evaluation process follows a funnel-like flow, filtering promising organizations and interventions. Initial stages are more superficial and comprise many donation opportunities, and the process gets deeper until it culminates in the cost-effectiveness analysis with a more restricted group in later stages. The final stage involves a deep dive into the specific sociospatial, organizational, and intervention context being evaluated. As the process progresses, evaluations become more objective, and time- and resource-intensive. The following outline summarizes the process.
Mapping and Screening
The initial stage involves identifying and screening organizations implementing interventions to address the prioritized causes in Brazil. Mapping is primarily conducted through online research on organizations’ websites, social media, and third-sector information platforms. Additionally, recommendations from doebem stakeholders, including cause-specific experts, can be considered. Screening is based on the quantity and quality of publicly available information. The evaluator assesses whether the available data is sufficient to assert that the organization implements promising interventions. Organizations lacking accessible, updated, or organized public data are excluded at this stage, as recommendations must be easily understandable for donors. Typically, this phase involves evaluating dozens of organizations, ideally around 15 passing the screening.
Pre-selection
Once approved for screening, organizations are evaluated on the criteria listed below.
This evaluation is carried out independently by at least three team members at doebem, following the best practices to mitigate issues like cross-evaluator noise. It is more in-depth than the previous stage but essentially uses online public data.
You can view a deep dive into each criterion here.
Based on the available data and insights from the In-Depth Cause Study, we evaluate the potential impact of interventions on the affected populations using Likert scales for criteria such as suitability, centrality, clarity, trustworthiness, previous evaluation, and potential impact. This practice also helps us predict and ameliorate the effects of bias along the evaluation process. Each criterion is rated on a scale, and scores are averaged across evaluators. The impact criterion has twice the weight of others. Organizations with the highest scores indicate positive consensus, while lower scores suggest negative consensus.
The scoring guides the Pre-Selection Panel, where evaluators meet to discuss and adjust scores to reach a consensus. The results are presented to the doebem Board for final validation, which may involve further data collection if needed. The goal is to narrow the list to approximately five organizations for the next evaluation stage.
Organization and Intervention Assessment
The remaining organizations are invited to participate in an evaluation led by doebem. Accepted organizations fill out a comprehensive form covering institutional details like team composition, activities, revenue, fundraising strategies, administrative and accountability processes, challenges and goals, and relationships with communities, other organizations, and public authorities. They also cover specifics about the intervention, including history, reach, costs, and other implementation aspects. The form helps us validate previous information and provides an extensive overview. Lastly, the conditions for a possible partnership with doebem are checked, and organizations are asked to submit their latest Financial and Activity Reports.
After reviewing the documents, additional meetings may be needed for clarity and stronger evidence, leading to the selection of up to three organizations for on-site visits. Fieldwork must include a visit to the organization's main office, an interview with the team, another with the leadership, a visit to the implementation sites, and interviews with beneficiaries. Although costly and time-consuming, these visits are essential for understanding implementation challenges and impacts, validating hypotheses, and building trust, ensuring the evaluation's quality and reliability.
The doebem research team uses the collected data to complete an Institutional Strength Matrix for each organization, assessing them across 20 criteria divided between institutional aspects and the intervention at focus. Institutional criteria include legitimacy, credibility, openness to external audits, financial sustainability, reputational risk, transparency, operational capability, and strategic resource use. Intervention criteria cover the importance of the solution, its clarity, technical foundations, local relevance, evidence of effectiveness, promotion of equity, externalities, scalability, sustainability, and the technical capacity of the implementing team. Each criterion is evaluated and summarized with a colored code (green, yellow, red) to indicate the level of concern. The completed matrix is reviewed by the doebem team and used to inform cost-effectiveness analysis and decision-making by the doebem Board. You can access the Matrix and other instruments used in the evaluation here.
Cost-Effectiveness Analysis
The final stage of doebem's research process is the cost-effectiveness analysis. In this stage, we estimate the societal impact generated for each Brazilian Real donated to each intervention and compare the results to recommend the best donation opportunities to our donors.
Our cost-effectiveness analysis involves a controlled extrapolation of scientific evidence from similar interventions applied to their respective contexts. This process addresses the challenges posed by the limited application of robust evaluation methods, such as experimental or quasi-experimental designs, in Brazil's third sector. Given these limitations, we rely on impact evaluations from other interventions to estimate the impact of the intervention under consideration.
Recognizing the challenges and limitations related to external validity, we prioritize studies that follow best methodological practices and are closely aligned with the intervention's characteristics, like context, time, and territory. This controlled extrapolation is further refined using calibration factors to better translate the observed impact in the reference study to the evaluated intervention's context.
Calibration factors include:
- Territory: Controls for relevant differences in vulnerability between the territory of the reference study and the intervention territory, under the hypothesis that impacts tend to be greater in more vulnerable areas.
- Population: Incorporates specific characteristics of the communities where the interventions are implemented, which may amplify or reduce the expected impact.
- Time: Adjusts for temporal changes between the reference study year and the current year of cost-effectiveness analysis, especially social improvements that may mitigate the negative effects of the social problem investigated.
- Intervention: Controls for significant differences between the reference study intervention and the intervention analyzed in the cost-effectiveness analysis.
- Institutional Strength: Incorporates relevant aspects from doebem's Institutional Strength analysis, based on documental analysis, interviews, meetings, and on-site visits, which may impact the intervention's effectiveness.
Each factor for each impact indicator and each organization is assigned a numerical value, with 1 indicating no expected difference in impact. Values below 1 represent a reduction in impact magnitude, while values above 1 represent an increase. These values are determined by the doebem team through extensive discussion.
Additional information required for the cost-effectiveness analysis includes:
- Intervention Characteristics: Information on cost, lifespan, and reach of the intervention are fundamental inputs for the calculation.
- Socio-economic and Health Data: Secondary data from official Brazilian sources, such as the Population Census, National Household Sample Survey, and the Ministry of Health, are used to better characterize the beneficiaries and more accurately estimate impact magnitude. The most recent and disaggregated data are prioritized.
- Intertemporal Discount Rates: used to present the value of both monetary and non-monetary impacts. These values can be specific to the Brazilian context, such as using the Brazilian federal funds rate for discounting monetary impacts, or borrowing from other reference evaluation organizations like GiveWell and Rethink Priorities, which have conducted extensive studies on these rates, particularly for discounting non-monetary values.
- Disease Burden Metrics: Widely used in epidemiology and public health, these metrics quantify disease burden in populations and assess policy effectiveness.
This comprehensive approach ensures accurate and relevant recommendations for doebem donors, balancing scientific rigor with practical applicability. As one of the objectives of cost-effectiveness analysis is to compare different interventions, including those from various causes, doebem chose to calculate the impacts of interventions in DALYs. This single metric combines effects on individuals' morbidity and mortality.
The calculation of the intervention's impacts considers three dimensions, which currently represent a broad spectrum of expected effects:
- Morbi-mortality: Impact of interventions in reducing the number of hospitalizations and the number of deaths caused by the problem or its implications.
- Healthcare spending: Secondary monetary impacts generated by the reduction in episodes of health complications.
- Developmental effects: Impacts indirectly related to health, such as employability and schooling, caused by the intervention and which can be transformed into income gains for the population benefiting.
These impacts are calculated and summed using moral weights from GiveWell to equate monetary units and healthy life years, resulting in a cost (R$) per healthy life year for each intervention. To ensure robustness, a sensitivity analysis is conducted to explore variations in parameters and assumptions. This exercise aims to identify significant differences in cost-effectiveness among solutions, rather than providing precise estimates, to guide efficient donations. The findings are then presented to the doebem team and Board for final recommendations to donors.
In conclusion, the research team at doebem has developed a comprehensive and rigorous methodology to identify the most effective donation opportunities in Brazil. By meticulously analyzing potential interventions across multiple dimensions, including cost-effectiveness, institutional strength, and broader societal impact, doebem is able to provide donors with highly informed and strategic recommendations. This approach ensures that contributions can be directed towards the solutions that generate the greatest positive change per donated Brazilian Real, maximizing the social impact of charitable giving in Brazil. Moving forward, doebem will continue to refine and apply this methodology to continuously expand the portfolio of high-impact donation opportunities for its donors.
The doebem team would like to thank volunteer and Effective Altruism community member Felipe Amorim (@Felps) for his support in writing this blog post.
