Foundations and trusts play a critical role in supporting the work of nonprofit organizations, dispersing more than $40 billion a year to address a broad spectrum of community need.
Yet, those impressive figures say little about the “social return on investment” that foundations hope to achieve, nor do they illuminate whether or not the programs funded are effective. That’s why the Colorado Health Foundation has joined a growing number of philanthropic organizations that quantify their results through portfolio measurement.
A new report from Grantmakers for Effective Organizations states that when done well, portfolio measurement can help grantmakers, their grantees, and partners improve outcomes. GEO says that effective measurement enables grantmakers to assess performance “beyond the individual grant” and help improve foundation-wide performance.
Fay Twersky, Senior Fellow at the Hewlett Foundation, describes effective measurement as defining scope by positioning an achievable goal — stated as the problem to solve — at the center of strategy development. In that way, Foundations can increase their ability to measure the difference they’ve made.
In the 2009 publication, “Evaluation in Philanthropy: Perspectives From the Field,” GEO and the Council on Foundations outlined an approach grantmakers should take to measure results of past investments. The report pointed out that measurement isn’t merely about “quantifying proof;” it is about learning how to gather information and draw lessons from grantees’ work by learning what happened, why it happened, and how grantmakers and their partners can achieve better results down the road.
The underlying principle to these and other commentaries on measurement is learning. Effective measurement promotes organizational learning which, in turn, enhances grant effectiveness and builds organizational performance.
In 2008, the Colorado Health Foundation Board of Directors adopted 12 Measurable Results to guide the Foundation’s investments and support its grantees in measuring the effectiveness of their work. The evaluation method we have developed to track and report on these Measurable Results serves as a valuable learning tool, helps us refine our strategies, and become a better grantmaker.
As the data we gather, learn from, and share broadens and deepens, it will reveal how – and to what extent – the health initiatives we support are improving the lives of Coloradans toward the vision of making Colorado the healthiest state in the nation.
The method behind the model
Before I explain our evaluation model, let me provide a bit of context about what we do at the Colorado Health Foundation.
Simply put, the Foundation invests in the good health of the people of Colorado by targeting three areas: Healthy Living, with an emphasis on good nutrition, physical activity, and chronic disease management; Health Care, focusing on coordinated and integrated primary care; and Health Coverage, with the goal of insuring as many Coloradans as possible.
Along with those investments and partnerships comes the expectation of meaningful, positive returns. In other words, we want to know how the dollars we invest come back to the people of Colorado in the form of improved health.
So how does our evaluation model work? Instead of relying on extensive grant-by-grant specific metrics, we have developed and continue to refine an evaluation model that emphasizes clarity and simplicity — with the intention of reducing the data-collection burden for grantees. We have done so by focusing on collecting fewer, more meaningful data points. Along with providing guidance for our grantees, these results provide us with valuable lessons on the most effective use of our funds.
Each Measurable Result is research-based and highly predictive of a healthy outcome. For example, research shows that increasing physical activity (one of our Measurable Results) is associated with managing a healthy weight. Thus, numeric increases in this Measurable Result help us track our progress in that area of focus.
This basic model is deeply rooted in outcomes-based research. For example, let’s say a grantee estimates it could increase physical activity for approximately 500 children. In order to “count” any child in our Measurable Results, that child would need to engage in the physical activity levels recommended by the Centers for Disease Control and Prevention (which sets the guideline that children need to participate in at least 60 minutes of moderate to vigorous physical activity on three days a week). Thus, our evaluation model helps us track the number of children who are engaged in CDC-recommended levels of physical activity.
In implementing our new evaluation model, we needed to build a data collection system for the Measurable Results – including an online application process for grantees as well as an electronic review process for program staff. This period included a high level of training for program staff and soliciting feedback from grantees about the newly developed metrics.
Within the last three years, more than 300 of our grantees have been collecting data on one or more of the 12 Measurable Results. Approximately 80 percent of our grants contribute to one or more Measurable Results.
Reaping the benefits
After three years of collecting evaluation data, I’m happy to report the benefits for our staff, mission, and grantees have been … well, beyond measure.
From a practical standpoint, the Measurable Results have allowed us to generate cost ranges for particular kinds of programs. For example, we calculated that a group of 27 grants working to increase physical activity levels had a cost range of approximately $7 to $2,600 with a median of $251 per person. Foundation staff uses these kinds of numbers to determine whether a grant proposal is within a reasonable cost range. While some interventions may justify high cost because of the high intensity of the intervention, the cost ranges were used to determine if a proposed grant budget was within a typical range of cost. This has been an extremely helpful data point in our due-diligence process.
One Foundation staff member told me that the Measurable Results have become an anchor in conversations about what we do and don’t fund. Program staff communicate that funding decisions are weighted based on the ability of a proposed project to produce one or more Measurable Results. While there are other factors that impact funding decisions (such as organizational capacity and effectiveness), the ability to impact one or more of the Measurable Results is a major factor in funding decisions.
One example of how Measurable Results data refine our Funding Strategies: Early evaluation data from a group of grants working to enroll more Coloradans in public health insurance programs indicated that efforts to “cast a wide net” at events such as health fairs weren’t as successful as targeted enrollment strategies. Meanwhile, strategies that used free-and-reduced lunch data to identify families that might be eligible for public health insurance programs were found to be more effective.
Recent findings
To gain an understanding of how we are doing in each of our community outcome areas, we analyze data from groups of similar grantees that have completed at least one year’s worth of work. Recently, we released findings from five groups of grantees that reported Measurable Results. These results are based on grants approved in 2009, with 100 percent of the grantees reporting information in all but one of the categories.
The findings to date:
Healthy Living
- 20 grants funded by the Foundation increased the number of children and adults who engage in moderate or vigorous physical activity by 5,564
- 8 grants funded by the Foundation increased the number of children who receive healthy meals at school and have access to healthy vending choices by 18,222
Health Coverage
- 11 grants funded by the Foundation increased the number of children and adults who have adequate health coverage by 11,261
- 16 grants funded by the Foundation increased enrollment of eligible Coloradans in Medicaid and the Child Health Plan Plus (CHP+) by 2,837
Health Care
- 28 grants funded by the Foundation increased the number of underserved Coloradans who receive integrated care by 44,291 (93 percent of grantees completed reporting in this category)
Based on conservative estimates from grants awarded in 2009 and 2010, our grantees’ work impacted the following:
- Approximately 27,000 Coloradans through Healthy Living grants
- Approximately 11,000 Coloradans through Health Coverage grants
- Approximately 52,000 Coloradans through Health Care grants
Potential impact
Based on grants approved in 2011, our grantees have committed to deliver evidence-based programming to one in four Coloradans who are living in poverty (or approximately 250,000 Coloradans living at 150 percent of the Federal Poverty Level). Because our grantees use evidence-based programming and work with people over an extended period of time, the likelihood of achieving healthy outcomes is high.
The learning road ahead
Though evaluation might not be right for every philanthropic organization, it’s certainly made a measurable difference to the Colorado Health Foundation. Over time, the data will provide a rich inventory of learning material and analysis for the Foundation, the philanthropy community, and health stakeholders throughout Colorado.
As more grantmakers around the country explore evaluation, we believe our model will help us measure our impact and progress in making Colorado the healthiest state in the nation while guiding the work of our partners in forging new frontiers of innovation.
Anne Warhover is the president and CEO of the Colorado Health Foundation and serves as a member of CEP’s Board of Directors. You can follow the Foundation’s work on its website, Twitter, or Facebook.
