Open dialogue – an essential pushback to myopic impact reporting practices

Open dialogue is an essential pushback to myopic impact reporting practices, solely promoting financial effectiveness.

Since the SDGs and the UN Agenda 2030 kicked off in 2015, there has been a growing trend of prioritising linear knowledge production methods over programme evaluation centred on participation and democratic processes of community inclusion.

We, Elisabeth Gade (NORAD) and Laura Marano (Norge Unlimited), have been reflecting on the practical implications of this trend or this bias towards big data and a technicist approach to the age-old challenge of efficiency over quality. The pragmatism of relying solely on “scientific evidence” of efficiencies (such as an effective altruism philosophy) should be opened to public debate. We need more open dialogue about the pitfalls that lie ahead if we continue to disregard or systemically devalue experiential knowledge over controlled scientific experiments such as RCTs (Random Control Trials).

Like the weather, predictability and early warning systems are never perfect. In fact on the 2nd of July the Washington Post shared a really insightful map of the USA illustrating how weather forecasting is geographically limiting. The accuracy of weather forecasting depends on where in the US you live, coastal areas have weather patterns that are easier to predict.

This is proof to the point that scientific systems of predictability are essential and save thousands of lives every year. But…they are not fit for purpose in all contexts (or in this case geographies).

With this analogy of the weather, clearly one can choose indicators that speak to a personal way of knowing, ones habits and behaviours or choose to blow with the wind and just wing it…or listen to intuition – that guttural, physiological level of what feels right. A bit like my morning routine of:

1. Check Yr
2. Open a window
3. Actually, look out of the window and up at the sky
4. Decide how to dress my 7 year old for school…

The conundrum – always the same. Do I just go with Yr that seems to think it’s raining right now, but the sun is actually shining? Or should I just do what I always do? Overdress my child with way too many layers of wool, then cotton and finally a waterproof/windproof/snowstorm-proof of some kind?

The logic – you can’t be too prepared right!?

A sense of being informed, to then make a decision that impacts someone else other than myself, seems to be a sensible thing to do. Obviously. Billions of humans (like you and I) are making such mundane choice every day without questioning why we look to the app before we check-in with our common sense.

Our work involves a lot of communication between a plethora of interest groups with sometimes conflicting worldviews. It is often the case that we choose to move beyond the statistics and headlines using mainly quantitative data. The traditionally linear knowledge systems of modern era economic theory are ineffective in offering the contextual analysis necessary for making sense of the complexity of today. 2023 saw our planet earth crossing 6 of the it’s “planetary boundaries”. This year we continue to experience the hottest temperatures ever recorded. As we type this, Hurricane Beryl is tearing through the Caribbean and southern states of the USA at unprecedent speeds due to this phenomenon of high temperatures.

How we respond in a crisis continues to improve but remains imperfect.  Like weather forecasting you can never be 100% sure of what to expect in all corners of the world all the time. Therefore, in our day to day we are incrementally becoming systems thinkers and innovators or disruptors (i.e. that person who asks the awkward questions). Our training, education and experience teaches us to be mindful of nuance, diversity, participation, the importance of thorough research of the context (the root causes of the social challenge, history, diversity of language and geography) and how to make the invisible (intangible yet material/consequential) visible in complex programmes.

A systems thinking approach is essential for measuring progress on the SDGs. Considering the innate biases of the technologies we are so reliant on, qualitative methods should be prioritized and even preferred when donors/ grant givers/ fund managers require place-based social entrepreneurs or NGOs to share evaluation data. This is not news – an institutional critique of a more scientific approach to philanthropy offers an eloquent reminder of the risks of a winner-takes-all approach to reporting systems that do not adjust to scale. We often must adjust our tone and narrative to the specific audience we are connecting with. This requires an adjustment to what is considered a proportional expectation of an entrepreneur’s or an NGOs’ capacity, resources and a sensitivity to their team member’s self-efficacy and resilience to changing goalposts.

Frameworks for project/programme evaluation should encourage more data collection practices that are mindful of the:

Quality of data

• Ethical implications of the design process
• Positionality of the researcher
• Legitimacy in claims of identity (i.e. who is included or excluded in the term local community)
• Considerate and inclusive of the intersectionality of participants

As a way of improving contextual understandings of what is happening at a neighbourhood level or a village level, practitioners* often know the value of localised modes for interpreting information and gathering data/evidence. In readings for my masters dissertation at UiO (2022) I learned that AbdouMaliq Simone’s “people as infrastructure” can be a valuable frame for investing in urban regeneration. When the dichotomy of urban-rural and informal-formal is blurred and too simplistic the best means of progress towards a shared goal (such as the UN agenda and SDGs for instance) is a co-created participatory approach. Embracing participation is about prioritizing the process of being included in practice, rather than document successes by focusing solely on the product or solution (e.g. an agreement, a partnership, a policy paper, a home solar electric system).

* practitioners, meaning: social workers, public health officers, social entrepreneurs, project managers and community organisers.

We are aware of the privilege of Norway’s near universal access to mobile data and affordable technology (public/free wifi, 5G etc). Easy access to technology is a societal advantage, nevertheless we need to be mindful of not making assumptions of knowing it all. We argue that evaluation frameworks based solely on big data and the ability to filter what is most material to a project or programme is a strategic advantage. It is important to ask the question of what could be missed? or better – what is lost in this process?

Time and resources could be invested in standing back to reflect on the scale and proportionality of a programme’s evaluation requirements (in dialogue with stakeholders). There needs to be an ethical consideration of what is working best for the target group and what is working best for funders.

When we choose to focus on the interests of the most disadvantaged and the most excluded in a society we need to be mindful of the invisible barriers created by funding processes. Applying frameworks that account for qualitative methods and participatory ways of knowing is an important start to ensuring the credibility of data and the validity of findings.