Evaluation Plan for Smart investing@your library^®

The Evaluation Plan is an important part of your application. It should be both formative, to ensure continuous improvement in the quality of the implementation of the project, and summative, to measure progress toward achieving your stated goals. The evaluation plan should encompass all aspects of your project, including your marketing and outreach efforts.

Use the evaluation tools here to help you design your evaluation plan. We have included a sample form and a completed form for illustration purposes.

You may download a PowerPoint presentation on outcome-based evaluation from the ALA Office for Research and Statistics.

Grant recipients will also attend a one-day training during the 2009 ALA Midwinter Meeting in Denver on how to implement the evaluation plan described in the grant application.

Project Evaluation Planning

What is project evaluation?

Simply put, the library needs to know if it accomplished what it said it would, with the budget it had, and in the time it had to do it.

The purpose of project evaluation is to understand the effectiveness of the project as it was designed, to discover where improvements could be or were made to increase the project's effectiveness, and to demonstrate the project's value to the community.

The evaluation model for Smart investing@your library^® follows both a formative and summative approach.

What did you accomplish?

This is formative evaluation and focuses on the activities of the project as a measure of effectiveness. Collecting the following sorts of figures about the project will get you the information you need:

• Number of training sessions for staff and number of staff that successfully completed the training;
• Number of people attending investor education programs;
• Number of people successfully completing the guided instruction or tutorials;
• Number of people that were more confident and knowledgeable about investing.

Did you accomplish what you set out to do?

This is summative evaluation and looks at how well you did what you said you were going to do. For instance,

• Were the overall outcomes (goals) of the project met?
• How well were the overall goals of the project met?
• Were there obstacles that had to be overcome for the project to be a success?
• Did the marketing and outreach plan for the investor education program reach the people you wanted it to? Did those individuals, for example, attend the information sessions or formal training, or take the guided online tutorials?

How does my library develop an evaluation plan?

Evaluation isn't difficult, but it does require some planning. And as a grant recipient, you will have help from the professionals. At the 2009 ALA Midwinter Meeting in Denver you will receive one day of training on how to implement the evaluation plan you designed on your application. All expenses associated with this training including transportation, accommodations and meals will be paid by the ALA.

Basically, there are six project evaluation components that need to be included in the plan for your library's project.The first four are from the grant application:

1. A clear project description, details and rationale.
2. The intended direct results (measurable and achievable) of the project:
1. Goals (up to four)
2. Outcomes
3. Marketing and outreach plan
4. Methods for measuring the four goal areas (including project outcomes and the marketing/outreach plan). During the project period you will:
5. Measure the results of each goal area (including project outcomes and marketing/outreach plan); and
6. Report on the effectiveness of each goal area (including project outcomes and marketing/outreach plan).

Project Evaluation Components

What is measured and how?

The "what is measured" part of evaluation is determined by the project description, rationale, goal areas (including project outcomes and marketing/outreach plan). The goal areas are:

• Observable
• Measurable
• Achievable

The goals can be seen, counted, and are realistic to achieve. The "how" of measurement is determined by the type of intended outcome.

Some examples that may be useful in your evaluation planning:

• Training modules successfully developed by a certain date
• Measure if the deadline was met
• Staff trained in using investor education resources and online investor education modules during a particular timeframe
• Measure the number of staff trained and in what timeframe
• Establish a pilot group of patrons to participate in the program
• Was the library successful in establishing the pilot group? How many patrons agreed to participate?
• Conduct training for the pilot group
• Did all patrons in the pilot group participate in the investor education training?
• How many dropped out throughout the project period?
• Was the training successful?
• How many patrons in the pilot group successfully completed the training?
• For those patrons completing the training, how confident were they in their ability to find and use the investor education resources?
• Interview patrons in pilot group to learn how confident they were in using the investor education resources.
  [Note: The reality may be that you don't have adequate time to interview in all situations, so consider a "sampling" strategy for gathering exit interviews (interview every third person) or consider an exit questionnaire that can be dropped off at a service desk, or placed in a box upon leaving the library.]
• Did some patrons require additional investor education training, or did they ask to repeat a particular training?
• Count the number of patrons in the pilot group that requested and completed additional investor education training.
• Interview patrons in the pilot group to learn how confident they were in using the investor education resources after additional training was received.
• Was there increased use of the investor education resources?
• How often did the pilot group reserve computers in the library?
• Count the number of reservations by the pilot group for library public access computers.
• Count the number of unique visitors to the online investor education resources from inside the library.
• Did the pilot group use the online investor education resources from outside the library?
• Count the number of unique visitors to the online investor education resources by non-library IP addresses.
• Marketing materials and strategies were effective
• Feedback about marketing materials and strategies (including the project Web site) from library patrons and community partners was positive.
• Requests from patrons to join the pilot project increased.
• Requests from community partners to learn about the project increased.
• Requests from community partners to participate in the pilot project increased.
• Use of investor education resources by library patrons outside of the pilot project group increased.
• Marketing materials had to be reprinted X number of times to keep up with requests.

The following illustrates a possible goal, objective and direct result evaluation scenario:

Goal: Increase patron use of investor education and protection resources.

Objective: Train 10 public service staff in using investor education and protection resources and demonstrate to the public how to successfully use these resources.

Evaluation Plan: Evaluation will occur at three benchmark periods during the pilot project:

1. Completion of pilot planning phase (no later than xxx date);
2. Completion of five (5) training sessions with target group (no later than zzz date);
3. End of pilot training period (no later than yyy date).

Direct Result:

1. Ten public service staff successfully trained.
2. Trained library staff provided direct training in the use of investor education and protection resources to at least 50 library patrons during the project period.
3. 75% of library patrons trained demonstrated "high levels of confidence" in their ability to find and understand the investor education and protection resources.

Report on Project Effectiveness

Did the project accomplish what it intended to? It is here that you will summarize the overall successes in achieving stated goals, account for various issues or events that may have positively or negatively impacted the successful implementation of the investor education program as planned. Also, this is where the sustainability of the project should be discussed. How will it continue or expand and what resources will be used to maintain the program?

Evaluating how well a project achieved set goals and outcomes is an ongoing activity and provides opportunities to correct for influencing factors. For instance, were there mid-course corrections required in the project implementation to achieve goals? Describe these here and how the library adjusted the project plan to achieve goals, including any timeline impacts, etc. This is also the place to discuss what might have been done differently to increase the success of the project.

Copyright © 2008   |  American Library Association