Introduction

Building on part one of this series, “From Instinct to Insight: How Evidence Strengthens Philanthropy,” this blog post examines the multifaceted nature of evidence within the philanthropic sector. I will explore various forms of evidence, their roles in decision-making, and the tools available for evaluating their effectiveness. The aim is to enrich our understanding of how strategically applied evidence can drive impactful philanthropy.

What is Evidence?

Evidence consists of factual information supporting claims or beliefs and encompasses a broad spectrum, from peer-reviewed articles to personal experience. The value and applicability of each type of evidence will vary depending on the context and the questions being addressed. A critical distinction to remember is the difference between data and opinion.

For philanthropists, evidence acts as a guide for informed decision-making, effectively navigating the complexities of resource allocation. The challenge is to identify reliable evidence, interpret it and integrate it within decision-making frameworks. However, everyone tends to balance evidence with their tastes, preferences, biases, life experience and all the other things that clouds scientific judgement. Scientific evidence is but one contributor to how decisions come about- and it is important to be transparent about that.

The concept of a ‘hierarchy of evidence’ is pivotal in evaluating the reliability of different types of research and can help guide philanthropists in making informed decisions. Originating from evidence-based medicine, this hierarchy categorises research methods according to their potential to reduce bias. At the top of this hierarchy are systematic reviews and meta-analyses, which synthesise data from multiple studies. These are followed by randomised controlled trials (RCTs), which provide the highest level of individual study evidence due to their design in minimising bias and establishing causality.  Below RCTs are observational cohort studies and case-control studies, which can suggest associations but not establish causality. Descriptive studies, expert opinions, and case reports rank lower in the hierarchy, providing valuable insights that are, however, more susceptible to bias. This is because experts, and the authors of these last group of studies are often sponsored by corporations or people with their own particular view to transmit. By understanding this hierarchy of evidence, we can better assess the strength of evidence supporting various philanthropic interventions and choose the most reliable sources to inform our decisions.

Hierarchy of Evidence 2010 Greenhalgh T[i]

Types of Evidence in Philanthropy

Evidence-based philanthropy leverages diverse evidence types, each offering unique insights:

Empirical Research: Quantitative studies, such as Randomised Controlled Trials (RCTs) and meta-analyses, are considered the Gold Standard of empirical research due to their ability to determine cause-effect relationships between interventions and outcomes. For example, an RCT assessing a deworming program in Kenya highlighted significant improvements in child health and school attendance and demonstrated conclusively that these outcomes were directly attributable to the intervention itself.[i] This level of rigour ensures that the observed outcomes are solely the result of the intervention, eliminating other potential influencing factors.

Qualitative Data: Qualitative research captures the nuanced experiences of beneficiaries through interviews, focus groups, and ethnographies, adding depth to quantitative measures and highlighting the social impacts of interventions. For instance, a case study in Bangladesh revealed how small loans impacted women’s empowerment.[i]

Mixed-Methods Approaches: I believe that mixed-method approaches combining quantitative and qualitative evidence provide a more comprehensive understanding of an intervention’s impact. Such approaches effectively merge the robustness of statistical outcomes with the richness of beneficiary narratives. By asking, ‘Does it work?’ through quantitative data and exploring ‘why and for whom?’ through qualitative insights, we can better understand both the efficacy and contextual nuances of programs. Such methods illuminate the direct impacts and enhances our grasp of how and why specific outcomes are achieved, making it invaluable for holistic program evaluations.

Expert Opinion: While expert opinions are invaluable, particularly in shaping responses during crises such as the early stages of the COVID-19 pandemic, they must be scrutinised for potential biases. Experts, including those from the Scientific Advisory Group for Emergencies (SAGE), play a critical role in guiding philanthropic strategies. However, it is essential to consider their potential conflicts of interest or inclinations towards particular viewpoints, as these can influence the neutrality of their advice.

Stakeholder Feedback: Input from beneficiaries and community members ensures that interventions align with their needs and priorities. The Charities Aid Foundation highlights that consistent qualitative evidence, obtained through methods like surveys, is valuable for understanding beneficiaries’ perspectives on programmes.[i] This sort of evidence is thought to be particularly useful in determining whether a programme is acceptable or feasible.

It is important to note that in both philanthropy and healthcare, the ‘best’ evidence-based decision may not always align with the wishes of the patient or other stakeholders, highlighting the need to consider personal values and circumstances. Just as a patient may choose an alternative to the medically recommended treatment, such as chemotherapy, based on personal preferences or quality of life considerations, philanthropists should also weigh the wishes and feelings of stakeholders when deciding on interventions. The context in which decisions are made profoundly affects their outcomes and acceptability. Therefore, understanding and integrating stakeholder preferences and the specific contexts of interventions are essential for ensuring that philanthropic efforts are not only effective but also resonate meaningfully with the communities they aim to serve.

Evidence-informed approach to practice Joss & Alla 2021 Australian Institute of Family Studies

Evaluating the Quality of Evidence

Effective philanthropy depends on high-quality evidence. Tools like the GRADE framework and CASP checklists can help us to assess the reliability and relevance of evidence, ensuring that philanthropic decisions are based on robust data.

• GRADE Framework: Originally developed for healthcare, the GRADE approach has been adapted to assess evidence quality when looking at multiple studies in charitable interventions. It evaluates factors such as study design, risk of bias, imprecision, inconsistency, and indirectness to rate evidence from high to very low quality.[i]                                                                                                            
• Critical Appraisal Tools: Instruments like the CASP checklists assess the trustworthiness, relevance, and results of published papers. These tools guide users through a series of questions to systematically evaluate the designs of many studies.

Bridging the Skills Gap: Collaborating with Experts

Not all philanthropists possess the skills to effectively evaluate evidence. Collaborating with experts and organisations like Three Dimensional Impact Ltd can enhance the impact and scalability of philanthropic efforts by employing evidence-based approaches.

Conclusion

As philanthropy increasingly embraces evidence-based practices, understanding the types and applications of evidence is essential. By integrating empirical research, qualitative insights, expert opinions, and stakeholder feedback, philanthropists can significantly enhance the effectiveness and sustainability of their efforts. Investing in understanding what works not only helps avoid costly mistakes but also maximizes the potential for positive change.

References

[1] Greenhalgh, T. (2010) How to Read a Paper: The Basics of Evidence-Based Medicine (Chichester, UK: Wiley-Blackwell, 2010), p. 18. The diagram follows a discussion of the pitfalls of “decision-making by anecdote” (pp 4–5)

[1] Miguel, E., & Kremer, M. (2004). Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities. Econometrica, 72(1), 159-217

[1] Kabeer, N. (2001). Conflicts Over Credit: Re-Evaluating the Empowerment Potential of Loans to Women in Rural Bangladesh. World Development, 29(1), 63-84.

[1] Charities Aid Foundation (CAF). (n.d.). Evidence vs Impact: The Basics. Retrieved from [link]

[1] Alla, K. Joss, N. (2021) What is an evidence-informed approach to practice and why is it important? Australian Institute of Family Studies (https://aifs.gov.au/resources/short-articles/what-evidence-informed-approach-practice-and-why-it-important)

[1] Happier Lives Institute. (n.d.). Quality of Evidence. Retrieved from https://www.happierlivesinstitute.org/research/quality-of-evidence/