As people around the world look to support earthquake relief efforts in Nepal, scholars from Stanford and the London School of Economics and Political Science offer new research that can help donors make better decisions about where and how to contribute their money.

“NGO reports tend to focus on quantity in delivery, such as numbers of homes and people served—but not on quality,” write Yong Suk Lee (Stanford) and J. Vernon Henderson (LSE).

In a forthcoming paper, the coauthors evaluate reconstruction efforts in Indonesia following the 2004 earthquake and tsunami, and find two trends: aid agencies that directly execute their services—point-to-point—perform the highest quality work. And, when agencies contract their services, higher quality work is performed when a global, not domestic, implementer completes the work.

Knowing this reality, and with improved disclosure of outcomes, the coauthors hope that donors would be able to make more informed choices.

Fishing village survey 

Figure 1. A map details the survival rate of the population and flood damage within northern Indonesia in 2004. Darker shaded areas show a higher survival rate, lighter shaded areas show a lower survival rate. Striped areas denote flooding, largely on the northeastern border. Boundaries marked with thicker lines are ‘kabupaten,’ or county divisions, and lightly colored lines are ‘kecamatan,’ or sub-county units larger than a village alone. (Courtesy of Yong Lee).

Through fieldwork and three rounds of surveys – in 2005, 2007 and 2009 – Henderson and Lee investigated aid work in Aceh, an area of coastal villages in northern Indonesia (Figure 1).

Humanitarian efforts there focused on “hard aid” such as construction of houses and fishing boats. Total aid delivered amounted to $7.7 billion and was implemented by international and domestic aid agencies—some directly and some as contractors—as well as the Indonesian government.

First, Henderson and Lee conducted a pilot survey, and then with a cohort of surveyors from the University of Indonesia, held interviews with village leaders and fishing families. Participants were asked to rate their housing accommodation, and if applicable, how their fishing activity compared to before the disaster.

“Mostly, we sat with villagers to see how willing they were to talk about aspects of aid,” Lee said. “Since it was several years after the tsunami hit, people were pretty open throughout the process.”

Data from those surveys was combined with information from the Recovery Aceh-Nias relief project database maintained by the government and the U.N., as well as demographic information provided by participants.

Delivering aid: Global v. local

Empirical analysis revealed that aid agencies such as the Red Cross and Catholic Relief Services reflected higher quality aid delivery (at a mean quality near 3.00), while agencies such as Save the Children and Concern Worldwide reflected lower quality (at a mean quality between 1.0-1.5).

“What’s surprising is that reputation didn’t really line up with what was expected,” Lee said, citing a few renowned agencies that didn’t receive high marks.

Lee said this could be explained by the fact that aid agencies that specialize in disaster recovery are better equipped, while a learning curve might exist for agencies with wider missions.

Global aid agencies are more likely to have logistical experience given their reach across multiple disaster situations. And while all NGOs face reputational costs for their results, global aid agencies are greater exposed to criticism because, by size, they’re more visible.

Yet, while global aid agencies and implementers may have the raw skills, local implementers have the cultural know-how.

“Local implementers might not have the most experience – like how to construct a house or manufacture a fishing boat – but they will likely know what’s actually desired,” Lee said. “So, there are obvious tradeoffs at play.”

For example, villagers reported bad ventilation in houses. This was because some aid agencies used small windows and concrete instead of wood material more traditionally used in Indonesia. Some boats were impossible to use because of improper design; they sank upon first use or fell apart after a few months.

Image

Collection of photos from fieldwork in Aceh, Indonesia, provided courtesy of Yong Lee. Upper left: A house built in an aid project village shows windows retrofitted after initial construction. Upper right: Boats constructed by aid agencies for fishing activity are refashioned to serve as water taxis for people and cars. Lower: Fishing boats sit unused on the side of the road many of them impossible to use, according to villagers surveyed.

Logistics and oversight

Aid delivery depends in many ways on the location and scale of the disaster. But, a few main aspects can determine if an aid agency doing its own work or operating as an implementer meets or exceeds expectations.

Henderson and Lee suggest that agencies that were highly supervisory had greater positive outcomes from their workers. In the case of Aceh, better monitoring and insistence on quality by leadership is a likely corollary between construction of better quality homes and boats.

“Rather than just give money, NGOs need to really oversee the projects. Organization and management are essential facets,” Lee said. “And that requires a lot of additional effort on their part.”

Oversight is especially relevant in disaster situations because of the often-overwhelming need for reconstruction. A flood of less-skilled workers enters the market to fill this gap, and on average the quality of work degrades.

“It’s much more difficult to impose quality control at this point,” Lee said. “So the implication that comes out of it is how does the implementer effectively utilize less-skilled workers.”

Getting to know the implementers and evaluating their work in-progress would help ensure quality on behalf of the aid agency. And, better dissemination of information about aid outcomes would help assure donors that their monies are being applied in the best possible way.

Future study

Most “hard aid” delivered to Aceh’s villages had finished by 2010, but “soft aid” such as democracy promotion and women’s empowerment stayed longer.

Henderson and Lee conducted one final survey in 2011. The data has been offered as open source material for researchers along with the larger data set.

Noting this, Lee said, “We’re thrilled that people are looking into the data further. It’s exactly what we wanted.”

Research projects applying the data include the impact of the tsunami on Aceh’s local economies and health effects on the population, among other areas.