We got the call from a nonprofit whose flagship livelihoods programme had been running for eleven years and had, quietly, been declining for three of them. The annual funder report had all the right numbers. Enrolment was healthy. Attendance was healthy. What no one could explain was why the women who had joined weren't coming back.

The client thought we'd build them a dashboard. They thought the problem was that they weren't measuring enough. It's a common instinct. When something's going wrong and you can't see it, you assume the answer is more data.

The answer was usually less data, better asked.

The extractive survey problem

The programme's annual evaluation was a 42-question survey designed at headquarters, translated by a consultant, and administered by field staff who had a quota to hit. The women who took it were paid nothing for their time. They were asked about things like "programme satisfaction" and "perceived economic outcome." They gave the answers they thought the field staff wanted to hear, because saying otherwise felt rude.

None of the questions asked what was hard. None of them asked what was missing. None of them left room for someone to say "the sessions run at 2pm and I have to be home to feed my children by 3, so I stopped coming." That answer was the entire story, and the survey couldn't hold it.

A working definition: extractive research takes information from the community and returns nothing except a report the community will never read. Almost every "impact evaluation" you'll ever see is extractive. That's the industry default. It doesn't have to be yours.

What we did instead

We spent five months embedded in six of the communities the programme served. Not doing formal research — we're not academics — but doing the kind of thing a good consultant does when the client will pay them to listen: showing up, drinking a lot of chai, asking questions we'd have been embarrassed to write on a form.

What we learned:

  • The sessions ran at the wrong time of day. This one accounted for most of the drop-off.
  • The credit product was designed by someone who had never taken a loan of that size.
  • Two of the three most active collectives had, without telling anyone at HQ, quietly rewritten the programme themselves. Their version worked better than the official one.
  • The literacy component was patronising. Members wanted a phone-first version. HQ hadn't asked.

None of this would have shown up in a survey, because none of the questions were asked. And once you've heard it, you can't unhear it. The programme was redesigned in the next quarter — not by us, by the members. We helped write it up.

Three questions worth stealing

If you're running a programme and you're worried something's drifted, don't reach for a survey. Reach for three questions, asked over chai, in the local language, by someone the members aren't afraid to disagree with:

  1. "What would you change about this if it were yours to change?" Everyone has an answer. Almost nobody's been asked.
  2. "What's the thing you nearly quit over?" This is where the real design flaw hides.
  3. "If someone from your community was thinking about joining — what would you tell them to expect?" The answer to this is the honest version of your programme description.
"They didn't just hand us a report. They handed us a programme our members helped write."

That quote is the whole point. If your consultation produces a document nobody at the community level would recognise, you consulted the wrong people. If it produces a document they helped write, you consulted the right people, and the document is worth more than any dashboard.

Listening isn't glamorous. It doesn't show up nicely in a funder deck. But it's what actually shifts a programme, and nothing else does.