Project Evaluation
Tips
Terms:
Performance
measures are a tool to measure work
performed and results achieved. It
describes (a) what is to be measured,
and (b) the methods of measurement.
Inputs are
resources needed to complete program
activities. Inputs include labor,
materials, equipment and supplies.
Outputs are
the direct products of program
activities. Outputs include the number
of people receiving a service or
product, the number of services or
products delivered. For example, an
output may be the volume of containers
collected. Outputs may be used to
measure outcomes.
Outcomes are
benefits or changes that result from the
program activity. They are influenced by
program outputs. Outcomes may be related
to behavior, skills, knowledge,
attitudes, values, condition, status or
other attributes.
Outcome indicators
are specific items of data that are
tracked to measure how well a program is
achieving an outcome.
Outcome targets or
goals are objectives for a programs
level of achievement.
Why does the
Department of Conservation want you to
measure performance?
-
Its a quantitative
assessment of performance, quality or
cost used to guide the decision-making
process, guide staff, and define
objectives.
-
To calculate a
return on investment (i.e., the grant
funds).
-
To demonstrate that
funds provided through the city/county
payment program made a difference and
contributed toward an increase in the
beverage container recycling rate.
-
To analyze findings
and help you improve your program.
Step-by-step
approach to developing a system for
measuring program outcomes and using the
results:
1.
Understand the difference between
inputs, activities, outputs and
outcomes.
2.
Develop a timeline
and assign someone responsibility for
the planning and implementation of
outcome measurement.
3. Choose
the outcomes you want to measure. Set a
reasonable target or goal. Define a
set of outcomes that track the benefits
participants experience as a result of
the program. Some programs will have
more than one outcome. Consider the
following when setting a target:
outcomes for the previous reporting
period, positive outcomes achieved by
similar programs, resources available,
external influences (e.g., economy
fluctuations, seasonal fluctuations,
etc.) Your target should be achievable
but challenging and lead to increased
results from the previous year. For
example, placing collection bins in
convenient locations at parks will
increase the volume of beverage
containers collected over 12 months by
50% and decrease the amount of beverage
containers in the trash and hauled from
parks to landfills.
4. Specify
the outcome indicators you need to
collect and measure. They should be
observable and measurable. Using the
example above, indicators may be the
number of people that place empty
containers into collection bins or the
volume of containers collected.
5.
Establish a baseline. A
baseline is the beginning point of
outcome indicators to be used for
evaluation purposes. A program can use
its own data as a baseline against which
to compare future performance. It can
also use data from another program. It
may be the volume collected, knowledge
or skill level, current attitude,
behavior, values, condition, status or
other attributes.
6. Prepare
to collect data on your indicators.
In this step, you should determine the
data collection method and determine
where to get the data you need. Make
sure your data collection method is
consistent by training data collectors,
if you are not collecting it yourself.
(You will need to identify the data you
are measuring on page 2 of your funding
request form and explain the method you
used to collect the data.) Another
suggestion is to develop written
procedures to document your work. This
may come in handy if a staff change
should occur. If you are tracking volume
and your program is certified, you must
use the DR-6s (shipping report form) as
your source of data to be consistent
with the volume reported to the
Department by your recycler/processor.
Things to keep in
mind when determining your measurement
system:
A one-size-fits-all
measurement system is not realistic. For
instance, you can use:
a) Mechanical tests
and measurements, such as scales
b) Surveys
c) Trained observers who rate behavior
or environments
d) Staff who interact with program
participants or event organizers
e) DR-6s (the Department of
Conservation shipping report form used
by certified or registered program
participants (e.g. a certified
recycling center), recycling center
receipts, call tracking reports, etc.
7.
Develop a database for your data.
8. Try out
your outcome measurement system. You
should conduct a trial run or "pilot
test" that includes collecting the data,
analyzing and reporting it. This will
help you identify overlooked outcomes,
inadequately defined indicators,
inconsistencies in reporting, cumbersome
procedures and analyzing and reporting
dilemmas.
9. Analyze
your data and report your findings.
Compare actual outcomes with targets
identified at step 6. This is the
information you will use to describe
your outcomes on your funding request
form on page 2. For example, compare
your baseline data with the data you
periodically collect and calculate the
percentage change (increase or
decrease).
10. Monitor
and evaluate your system periodically
and make continuous improvements.
Build in a formal review of your
program. Your program will be affected
by changes in the environment. Your
outcome measurement system needs to keep
pace with these changes. Look for ways
your processes have shown improvement,
where it is slipping and what continues
to be troublesome. Consider changes that
occurred in the program and how to
adjust your system to accommodate these
changes. Solicit staff input and other
participants for feedback.
11. Use
your findings to increase the
effectiveness of your program. If
you have reached your goals, establish
new goals for ongoing progress. If you
have not reached your goals or outcomes
it may be time for CHANGE. Identify
factors that influence your programs
ability to reach its goals such as:
-
Why is the existing
program not working?
-
What are reasons
for poor participation?
-
Are other key
players affecting the outcome?
-
Are there
partnerships that could be formed to
help you meet your goals?
-
Is timing/season a
factor?
-
Was the frequency
of service appropriate?
-
Was the target
audience you selected appropriate?
-
Was the environment
supportive and appropriate for your
program?
-
Was there enough
awareness about the program?
-
Was it convenient?
-
Were there barriers
to prevent you from collecting
accurate data?
Change might involve
creating new and innovative ways to
carry out your program; such as, a new
way to increase collection of beverage
containers for recycling. Changes need
to happen in order for your program to
progress and make you successful in
meeting your program goals.
Once youve
determined the changes you need to make,
develop and implement a strategy and
timeline that will move you to where you
want to be.
Not meeting your goal
does not necessarily mean change.
Perhaps you underestimated the length of
time involved to effectively administer
your program.
Click here for examples of
goals, ways to evaluate your project and
measurable outcomes for various beverage
container activities.