Demystifying the Social Impact Evaluation Process

I started my career in academic public health research where the quality of an intended intervention is measured by rigorous standards. Policy makers and foundations use research from these well-designed studies to decide where to invest financial and human capital. My career shifted to social impact work, where I found that far too often, well-intentioned interventions are not adequately measured. While social impact organizations should not be held to the same standards as academic institutions, they need to use evaluation to demonstrate the value and efficacy of their work to funders and their boards, and to help contribute to the social sector evidence base.

What are the barriers to evaluating social impact work?

Evaluation and creating measurement frameworks can be an intimidating process to many in the social impact sphere. They may not know where to start an evaluation, what to measure and what to do with the data once it has been collected. In addition, nonprofits may not have the human or financial resources to dedicate to this work.

Why is it important to evaluate your work?

Evaluation gives social impact organizations concrete data that demonstrate what they did or didn’t do, what the impact was, and what they need to do differently. Evaluation provides accountability and it enables social impact organizations to share best practices. Social impact evaluation is also critical because it gives funders and policymakers the evidence base required to make informed, pragmatic decisions.

How do you evaluate social impact work?

While you may intrinsically know that your organization is doing great work, you need to define, measure, and display your impact to your funders, your staff and your board. Though this may seem challenging, there is a relatively straightforward framework that most organizations can use to measure impact, improve their work and create a learning culture.

The Evaluation Process

The evaluation process should be considered a learning loop in which the findings inform program improvement and the evaluation framework. The process involves several steps designed to: describe how program activities are intended to achieve specific outcomes, define the indicators that measure those outcomes, and collect and analyze data on those indicators. To ensure that the scope is feasible and the data will be useful, evaluation should also include a careful process of identifying the most important data to help show your progress, identify challenges and improve your work.

The graphic below illustrates steps in an evaluation process:

Theories of change and logic models

Theories of change and logic models are helpful tools used in evaluation. However, there is often confusion about the difference between a logic model and a theory of change and when each should be used. They are tools with a similar purpose, but different applications. In general, logic models summarize program implementation while theories of change apply to organizational or program strategies.

Theories of change explain how and why change happens as well as the role of an organization or initiative in contributing to progress. A theory of change can also be a visual with text that depicts the big picture strategy. It is a simple way to articulate contextual elements that affect an organization’s work, show how the organization’s strategy fits in, and what will be different as a result of the work.

A logic model is a framework used to delineate the relationship between a set of activities and the outcomes expected as a result of those activities. A logic model can be used to communicate what a program does and how success is defined. The graphic below illustrates components of a logic model:

Evaluation Design Approaches

Evaluations should provide data that enable organizations and funders to understand the difference a program or initiative has made in comparison to similar programs, in comparison to the absence of a program, or in comparison to conditions before the program was implemented. Nonprofit organizations can choose a non-experimental approach or a quasi-experimental/experimental approach, depending on the type of programming and the financial and human resources available to support the evaluation.

Non-Experimental Design

Non-Experimental Design approaches involve the use of a “within group” comparison, for example, a pre- to post-intervention comparison to assess change over time. This approach describes how people who received the intervention are different at the end of the intervention compared to the beginning.

Quasi-Experimental and Experimental Designs

Quasi-Experimental and Experimental Designs involve the use of a comparison (quasi-experimental) or control (experimental) group to support claims of causality by comparing participants’ outcomes with those of individuals who did not receive the intervention. This approach seeks to discern whether the changes observed in participants are directly linked to the intervention.

Summary

An evaluation can help ensure that the activities an organization is implementing have their intended effect. Evaluations should also be used for organizational reflection, decision-making, and continuous program improvement. Findings should be shared with other organizations working in the same space to facilitate systemic learning. Data from evaluations also have tremendous value in organizational marketing and communications to elevate a program’s visibility and highlight an organization’s accomplishments.

Helpful Resources:

The Evaluation Center

This website provides checklists of good practices for evaluation management, evaluation models, evaluation values and criteria, meta‐evaluation, and other evaluation related topics. http://www.wmich.edu/evalctr/

Evaluation Handbook by W.K. Kellogg Foundation

This useful “How To” tool walks programs through the evaluation process.

http://www.wkkf.org/DesktopModules/WKF_DmaItem/ViewDoc.aspx?LanguageID=0&CID=281&ListI D=28&ItemID=2810770&fld=PDFFile

Theory of Change by ActKnowledge and Aspen Roundtable

This website provides background about theories of change and technical assistance and training on creating theories of change.
http://www.theoryofchange.org/index.html

ORGANIZATIONS

American Evaluation Association (AEA) is the main professional association for evaluators. The website provides information on publications, trainings, programs, and other online resources.

http://www.eval.org/

The Evaluators' Institute (TEI) offers short‐term professional development courses in a range of evaluation topics.
http://www.evaluatorsinstitute.com/