|
Program evaluation is the systematic
collection of information to make decisions and improve
the effectiveness of programs, services, and
activities. The job of the evaluation consultant is to
help determine what information to collect, how to
collect it, and to analyze and report findings based on
the information collected. Our approach to program
evaluation includes both process (descriptive
information) and outcome (measurement of change)
components. We apply research methods using both
quantitative and qualitative measures, and a variety of
data collection tools and techniques. Program
evaluation is not the same as basic research.
Many people think that program evaluation
is the same as research. As a result they may strive to
apply and combine principles and methods from both, but
the result is they usually do neither one well.
Program evaluation is sometimes known as
action research because it focuses on collecting
and analyzing data for the purpose of informing
decisions, clarifying options, reducing uncertainty, and
presenting findings within the context of a dynamic and
changing environment. By comparison, basic research
aims to draw definitive conclusions about new knowledge,
or to test a new theory. The findings from basic
research establish a truth and may be generalized across
time.
| |
Program Evaluation |
Research |
| Purpose
of data collection |
 |
To inform decision making |
|
|
To clarify options |
|
|
To reduce or minimize
uncertainty |
|
|
To provide information
about programs and policies, in context |
|
|
|
To discover new knowledge |
|
|
To test theories |
|
|
To establish truth |
|
|
To generalize from
findings |
|
|
Orientation |
|
|
Decision-oriented |
|
|
|
Conclusion-oriented |
|
| Aim |
|
|
Aimed at action |
|
|
|
Aimed at Truth |
|
|
|
|
|
Summative
evaluation is
more like basic research in that it
usually is a reflection of what happened
in a program, and the conclusions
resemble a “report card”.
|
|
|
|
|
|
Formative
evaluation is
the “evaluate-as-you-go” approach, with
findings reported at intervals to permit
program modification. Formative
evaluation most closely resembles the
characteristics listed under Program
Evaluation.
|
|
Evaluation is the means by which funding
sources learn what they got in return for their
investment. Nowadays most funding sources require
evaluation, particularly if they are providing new
resources associated with a grant initiative.
Program evaluation also allows management
to determine what is working well, and identify areas to
improve or change. A comprehensive evaluation includes
elements of ongoing needs assessment, quantitative and
qualitative data collection, and analysis at periodic
intervals to measure change over time or change in
response to program adjustments.
Years ago federal agencies recommended
their grant applicants budget 10-15 percent of their
program budget for an evaluation study. This means that
one might allocate $30,000 to $45,000 per year for a
grant of $300,000 per year. For some grants, evaluation
budgets have been as high as 25 percent.
More modest evaluation budgets require
more of the grantee, and the evaluation consultant may
shift his/her function to serve in a more advisory
capacity. High end evaluation contract costs are
usually associated with multi-site programs, database
development, data entry, and multiple reports. In
contrast, conservative evaluation consultant costs may
be associated with simply defining the measures to be
tracked, development of tracking tools, and analysis of
data collected.
Program planning and program evaluation
are two sides of the same coin. We use a logic model
framework to help illustrate the linkages between the
two, to show the relationship between what has been
planned, implementation activities, and outcomes.
There are several commonly used “logic
model” frameworks including the one we use and the
“results accountability” framework associated with First
5 initiatives throughout California. These models share
some similarities and maintain some distinguishing
characteristics, usually a reflection of personal
preference by the user.
We participate in development of grant
proposals with many clients, to help them address the
section of the application that requires they say
something about their plans to evaluate the program or
service activity they are planning to implement. It is
very useful for grant applications to include something
about the evaluation of the proposed effort.
The evaluation section may range from a
few paragraphs to a more comprehensive evaluation
design. At a minimum, an outside evaluation consultant
will want to know: (1) program goals and objectives; (2)
proposed activities; and (3) intended short and long
term outcomes. Remember, a grant proposal is much like
a “take-home exam.” Follow the instructions carefully,
and include as much as you can to explain your proposed
program and its evaluation.
Evaluation findings reports should be prepared with the
audience in mind. The Board of Directors for your
organization will want to know about the program
milestones and outcomes in a general way. They will
want to know how well the program is doing, insofar as
it relates to continued funding.
The
funding source will probably want the most detailed and
comprehensive reports from the evaluation. Usually this
includes a good description of who you are serving, what
they are receiving from you, and what results from the
services you provide. Very often, the funding source
defines the terms of the outcomes they want tracked and
reported.
Some
programs use evaluation findings to locate new funding
support, to generate interest in garnering local support
to sustain a program, and to share findings with
potential supporters or detractors of the program. We
have worked with client organizations who took the
evaluation findings and incorporated them in
newsletters, promotional brochures, and oral
presentations to policy makers and potential supporters
or partners.
As we develop a detailed evaluation
design, we engage several kinds of stakeholders in the
process. We solicit input from the client organization,
from management and the line staff alike. We share our
research questions with them, and identify potential
data sources, and ask for their input regarding what
they are willing and unwilling to do to help get the
data. We explore all possibilities together.
Often we develop and draft data
collection tools in collaboration with client
organizations and their staff. We get their initial
input, present one or more drafts for their review and
changes, and then finalize a tool that represents a
compromise between their needs and those of the
evaluation. Similarly, we learn what types of data
reports will serve their needs, how often they want the
reports, and whether they want to generate them or have
us prepare and submit reports to them.
We also take data findings at regular
intervals before various constituent groups (e.g.,
staff, program clients) to interpret what the data seems
to tell us. We ask what they see in the numbers, how
they interpret trends and patterns, and reflect on the
findings together.
We
have even experienced program clients sharing findings
to the Sacramento County Board of Supervisors, a
powerful illustration of empowerment evaluation as well
as the success of individuals served by the program.
In
short, we strive to engage a variety of stakeholders in
the evaluation before, during, and after data
collection. Through this process we actively promote
empowerment evaluation and enhance the capacity of all
stakeholders to become better consumers of contracted
evaluation services, as well as better advocates for
their programs and the program evaluation findings.
|
