Frequently Asked Questions

Program evaluation is the systematic collection of information to make decisions and improve the effectiveness of programs, services, and activities. The job of the evaluation consultant is to help determine what information to collect, how to collect it, and to analyze and report findings based on the information collected. Our approach to program evaluation includes both process (descriptive information) and outcome (measurement of change) components. We apply research methods using both quantitative and qualitative measures, and a variety of data collection tools and techniques. Program evaluation is not the same as basic research.

How does program evaluation differ from research?

Many people think that program evaluation is the same as research. As a result they may strive to apply and combine principles and methods from both, but the result is they usually do neither one well.

Program evaluation is sometimes known as action research because it focuses on collecting and analyzing data for the purpose of informing decisions, clarifying options, reducing uncertainty, and presenting findings within the context of a dynamic and changing environment. By comparison, basic research aims to draw definitive conclusions about new knowledge, or to test a new theory. The findings from basic research establish a truth and may be generalized across time.

Program Evaluation

Research

Purpose of data collection

	To inform decision making
	To clarify options
	To reduce or minimize uncertainty
	To provide information about programs and policies, in context

	To discover new knowledge
	To test theories
	To establish truth
	To generalize from findings

Orientation

Decision-oriented

Conclusion-oriented

Aim

Aimed at action

Aimed at Truth

	Summative evaluation is more like basic research in that it usually is a reflection of what happened in a program, and the conclusions resemble a “report card”.
	Formative evaluation is the “evaluate-as-you-go” approach, with findings reported at intervals to permit program modification. Formative evaluation most closely resembles the characteristics listed under Program Evaluation.

Why do an evaluation study?

Evaluation is the means by which funding sources learn what they got in return for their investment. Nowadays most funding sources require evaluation, particularly if they are providing new resources associated with a grant initiative.

Program evaluation also allows management to determine what is working well, and identify areas to improve or change. A comprehensive evaluation includes elements of ongoing needs assessment, quantitative and qualitative data collection, and analysis at periodic intervals to measure change over time or change in response to program adjustments.

How much does an evaluation study cost?

Years ago federal agencies recommended their grant applicants budget 10-15 percent of their program budget for an evaluation study. This means that one might allocate $30,000 to $45,000 per year for a grant of $300,000 per year. For some grants, evaluation budgets have been as high as 25 percent.

More modest evaluation budgets require more of the grantee, and the evaluation consultant may shift his/her function to serve in a more advisory capacity. High end evaluation contract costs are usually associated with multi-site programs, database development, data entry, and multiple reports. In contrast, conservative evaluation consultant costs may be associated with simply defining the measures to be tracked, development of tracking tools, and analysis of data collected.

How do you integrate program development with evaluation?

Program planning and program evaluation are two sides of the same coin. We use a logic model framework to help illustrate the linkages between the two, to show the relationship between what has been planned, implementation activities, and outcomes.

There are several commonly used “logic model” frameworks including the one we use and the “results accountability” framework associated with First 5 initiatives throughout California. These models share some similarities and maintain some distinguishing characteristics, usually a reflection of personal preference by the user.

What should we put in our grant proposals about “evaluation”?

We participate in development of grant proposals with many clients, to help them address the section of the application that requires they say something about their plans to evaluate the program or service activity they are planning to implement. It is very useful for grant applications to include something about the evaluation of the proposed effort.

The evaluation section may range from a few paragraphs to a more comprehensive evaluation design. At a minimum, an outside evaluation consultant will want to know: (1) program goals and objectives; (2) proposed activities; and (3) intended short and long term outcomes. Remember, a grant proposal is much like a “take-home exam.” Follow the instructions carefully, and include as much as you can to explain your proposed program and its evaluation.

Evaluation findings reports should be prepared with the audience in mind. The Board of Directors for your organization will want to know about the program milestones and outcomes in a general way. They will want to know how well the program is doing, insofar as it relates to continued funding.

The funding source will probably want the most detailed and comprehensive reports from the evaluation. Usually this includes a good description of who you are serving, what they are receiving from you, and what results from the services you provide. Very often, the funding source defines the terms of the outcomes they want tracked and reported.

Some programs use evaluation findings to locate new funding support, to generate interest in garnering local support to sustain a program, and to share findings with potential supporters or detractors of the program. We have worked with client organizations who took the evaluation findings and incorporated them in newsletters, promotional brochures, and oral presentations to policy makers and potential supporters or partners.

How do we engage the folks we serve in the evaluation process?

As we develop a detailed evaluation design, we engage several kinds of stakeholders in the process. We solicit input from the client organization, from management and the line staff alike. We share our research questions with them, and identify potential data sources, and ask for their input regarding what they are willing and unwilling to do to help get the data. We explore all possibilities together.

Often we develop and draft data collection tools in collaboration with client organizations and their staff. We get their initial input, present one or more drafts for their review and changes, and then finalize a tool that represents a compromise between their needs and those of the evaluation. Similarly, we learn what types of data reports will serve their needs, how often they want the reports, and whether they want to generate them or have us prepare and submit reports to them.

We also take data findings at regular intervals before various constituent groups (e.g., staff, program clients) to interpret what the data seems to tell us. We ask what they see in the numbers, how they interpret trends and patterns, and reflect on the findings together.

We have even experienced program clients sharing findings to the Sacramento County Board of Supervisors, a powerful illustration of empowerment evaluation as well as the success of individuals served by the program.

In short, we strive to engage a variety of stakeholders in the evaluation before, during, and after data collection. Through this process we actively promote empowerment evaluation and enhance the capacity of all stakeholders to become better consumers of contracted evaluation services, as well as better advocates for their programs and the program evaluation findings.