Where the ARC meets the EFC: Program Evaluation

A Program Evaluation of the Appalachian Regional Commission’s Water and Wastewater Infrastructure Projects, FY2009-FY2016

In 1964, President Lyndon B. Johnson stood in Martin County, Kentucky and declared “an unconditional war on poverty in America.” As part of this initiative, the Appalachian Regional Commission (ARC) was created: a congressionally appropriated commission that invests in economic development projects in Appalachia, an area of the country that was lagging behind in public health, infant mortality, education, and income level. Fast forward to today, and Appalachia still struggles with job loss and economic downturn. Many counties in central Appalachia built economies around coal mining, and as America’s energy profile shifts away from coal and towards other options, coal towns often need help investing in local infrastructure to improve quality of life and promote economic development.

What is the ARC?

Appalachia often fits many different definitions and geographic bounds. For the purposes of the ARC, Appalachia extends across 13 states, from Alabama to southern New York, and covers 420 counties. Each of these counties are given a specific designation by the ARC based on their economic distress level, with the hope that, through strategic investment, the ARC region can be competitive with the rest of the country.

ARC invests in many types of projects, but all of them fall within the bounds of their strategic goals. These include investing in the Appalachian economy, promoting education for a steady workforce, investing in infrastructure, leveraging the cultural and natural assets of Appalachia, and building leadership and community capacity. Each year, the ARC invests in over 400 projects in Appalachia, awarding project grants to nonprofits, state or local government agencies, or local development entities. in 2014, the ARC Strategic Plan reported that since 1965, the ARC has invested roughly $3.8 billion dollars in over 25,000 projects and leveraged an additional $6.40 of private investment for each $1 the ARC invested.

Where does the EFC come in?

In addition to the project grants the ARC issues, periodically the ARC awards contracts to conduct program evaluations of a portfolio of projects. In order to most effectively fund projects in the future and determine how these projects impacted the local economies and quality of life for communities, the ARC sends out an RFP for a team of experts to analyze this project portfolio and determine the impact. Jeff Hughes, Shadi Eskaf, and myself, from the Environmental Finance Center, in combination with Maureen Berner, a UNC School of Government professor and renowned program evaluation expert, and Tiffany Drape and Leigh-Anne Krometis, teachers and researchers at Virginia Tech with on-the-ground knowledge of water and wastewater in Appalachia, answered this most recent call and will be conducting the program evaluation.

On Thursday, October 18, the program evaluation project was kicked off at the ARC headquarters in Washington, D.C. | Left to Right: Leigh-Anne Krometis, Maureen Berner, Austin Thompson, Tiffany Drape, Jeff Hughes, and Shadi Eskaf.

Together, this team aims to conduct a program evaluation of over 400 water and wastewater projects in Appalachia occurring between FY2009-FY2016. The team will conduct a literature review of innovative design, financing, technology, and partnerships; clean and merge data from the ARC’s ARCnet database with other supplemental databases; conduct a survey of grantees on economic development and quality of life improvement outcomes directly tied to projects; visit project sites and develop case studies; and conduct focus groups. With all of this data, the team will conduct both qualitative and quantitative analyses to determine the impact of this portfolio of projects, and provide policy recommendations based on the findings. Together, all of this work completes the program evaluation.

So what is program evaluation?

Somewhere, deep in our minds, we remember going through the scientific method. Starting from a young age, schools drill the concept of experimentation and data collection into students. Most people understand the concept of a treatment group and a control group. It is pretty simple, really: one group undergoes a treatment and the other group is given a placebo. Ideally, other variables that impact the outcome are controlled for, so that the difference in mean outcomes for the two groups reflects the impact of the treatment. Under this traditional experimentation method, it is easier to assess the actual impact of the treatment and control for variables that might create error or bias than natural experiments.

One of the challenges of assessing the impact of policy choices is the experimentation process. Policies and programs rarely have a very similar “treatment” group and “control” group. Often times, the areas undergoing policy changes are inherently different from the areas that do not undertake the same policies, making it challenging to compare two similar groups. Despite these limitations, policymakers need to find out if these changes have a meaningful impact.

Policy changes and program implementation are often expensive, time consuming, and pull limited resources in one direction. In order to address the impact of these programs, some statistical methods have to be employed, resulting in program evaluation.  By collecting as much applicable data as possible and decreasing or controlling for any extraneous variables, program evaluation allows researchers to create a counterfactual (control group), and assess the impact of the “treatment,” or area with the program in place, relative to the “control.”  These statistical methods, combined with case studies, focus groups, and surveys help to paint the entire picture of the impact of a portfolio of projects.

How does program evaluation relate to environmental finance?

The ARC’s strategic plan is all about investing in Appalachia’s future. To date, the ARC has a wide portfolio of projects, investing in a variety of project types in the region. While the ARC has been widely supported by both political parties, routine program evaluations are necessary to ensure that the investment in infrastructure, education, etc., is resulting in the anticipated impact and utilizing federal appropriations effectively.

A dollar spent on one project is a dollar removed from another. The ARC operates under the same economic principal of scarce resources as any other entity. These funds are providing clean water and proper wastewater disposal for rural communities otherwise unable to complete such projects. This investment improves public health, water quality, and promotes economic development.

Our goal is that this project provides the ARC with a creative list of innovations, a thorough analysis of the portfolio of water and wastewater projects, and recommendations for future strategic investment. The next 17 months will be a whirlwind effort to collect, clean, and analyze a diversity of information related to water and wastewater projects. We expect that time to be spent learning, connecting with grantees in Appalachia, and providing a program evaluation that impacts the ARC, and more importantly, the communities of Appalachia.

More information on the EFC and its role in this project can be found here.

Austin Thompson joined the EFC at UNC in 2018 as a project director. In this role, she conducts applied research and provides technical assistance and training for environmental service providers. Thompson holds a BS in Biological Sciences from the University of South Carolina and a Master’s of Environmental Management from Duke University, with a concentration in Environmental Economics and Policy.