Charity Navigator’s New Focus on Results

October 23, 2013

Charity Navigator’s decision earlier this year to add Results Reporting as one of its main criteria for awarding its coveted stars has generated concern in nonprofit board rooms across the nation: especially among those with long-standing four-star ratings.

Based on Charity Navigator’s early research on nonprofits’ results reporting, concern is likely in order. Panic, however, is not. While Charity Navigator is now posting results reporting-based data for some of the 10,000 nonprofits it rates, no organization’s star ratings will be affected for a few years.

In its announcement of the evolving change, the charity ratings organization noted that “this analysis will not impact any charity’s star rating until we have gathered the data for all 10,000.” It went on to note that it may need four years to complete the complex analysis necessary.

While Charity Navigator’s decision may be bad news for some nonprofits, it should not be a shock. Well-respected foundations, corporations and government agencies have been pushing agencies in this direction for more than a decade.

The days of measuring success by how many people you have “seen” or “served” in our data-driven times are nearly at an end for nonprofits, especially those in the human services business. For example, nonprofit leaders say many child-focused programs, even those that are sports-oriented, have been required for years to document that their program models prevent drug use in the long-term or that a significant percentage of participants went on to graduate high school.

It is unfortunate that producing results documentation may present financial and technical challenges to small or startup nonprofits on shoe-string budgets. The reality though is that, while Charity Navigator reports on nonprofits, its target audience is donors.

In the past, some donors have blasted Charity Navigator for placing a major emphasis on criteria like low administrative overhead rather than on service effectiveness. The new Charity Navigator methodology for measuring a nonprofit’s results has five main elements. In general the elements track whether:

1) Fundraising materials reflect how the nonprofit allocates its resources as reflected in its most recent Form 990

2) Causal logic that drives or predicts expected outcomes adds up under normal circumstances

3) The nonprofit is a member in good standing of a validating organization with standards and codes of conduct, or of certification bodies that relate positively to outcome measures

4) Methods for collecting and reporting client feedback (constituent voice) are comprehensive, promote honesty, are collected over time and assess whether clients believe their lives have been improved

5) Reports on the measured impact of the charity’s programs are published on a routine basis (at least every five years)

Results should be derived from commonly accepted research techniques that are easy to understand. And, the nonprofit should explain what is changing as a result of evaluation.

After Charity Navigator evaluated more than 200 family and children-oriented programs, its chief executive director Ken Berger reported: “The vast number of nonprofits do not report in a meaningful way on their results.”

The survey tool used had 14 questions. Groups that won eight positive checks were considered high scorers. While the nonprofit community understands Charity Navigator’s need to move in this direction, many question whether any evaluation system can be one-size-fits-all. Charity Navigator research analysts are studying the applicability of the tool across its 34 charity “cause areas,” which cover national and international organizations.

Some nonprofit executives and consultants contend that agencies serving, for example, transient populations such as the homeless, will have a much harder time gathering constituent feedback and data on long-term benefits than agencies that work with the same families over a period of years. Advocacy organizations will also have a more difficult job identifying specific outcomes and time frames for achieving their goals than those that provide hands-on services.

Nationally, organizations have found many innovative ways to measure constituent voice, including through the use of mobile devices. International agencies such as Path – a global health nonprofit promoting new technologies and practices – have found it difficult to gather patient feedback from “primary constituents.”

Keystone Accountability, a consulting agency that worked with Charity Navigator to develop its new methodology, continues to help nonprofits to develop systems for listening to the people they serve. David Bonbright, the agency’s head, is seeking foundation money to create a website to help nonprofits collect feedback and other data that Charity Navigator will find useful for results reporting.