Evaluation
Evaluation is essential to the management of your communications program. It enables you to determine if your program implementation was effective and if your program met its original objectives. Armed with the results of your evaluation, you can make decisions about modifications needed to improve the initiative. In addition, an effective evaluation can increase the program's credibility within your community and with current and future funding sources such as CMAQ.
Your first step will be to determine the purpose of your evaluation. Specifically, you need to:
- Evaluate the progress of a clean air campaign
- Evaluate the impact of a clean air campaign
- Evaluate the results of a clean air campaign
If you have questions that aren't answered in this section, there is a good chance that the answer can be found in the Ask The Experts section or in our FAQ section. Additionally, the EPA has prepared an in depth toolkit called "Demonstrating the Benefits: A Program Planning and Evaluation Toolkit for Air Communicators)" which can be found here.
Progress
Process evaluation provides a checklist to help you focus on what needs to be accomplished, which helps you get the job done. It creates accountability for meeting specific objectives, which can be an effective motivator.
A process evaluation does not need to be elaborate or complicated. Keep it simple by updating your progress at least once a month. Periodically, assess your objectives and modify them, if necessary. Share the results with the people involved in your program as well as with other It All Adds Up participants.
It All Adds Up to Cleaner Air has put together a comprehensive spreadsheet to help keep track of costs, resource allocation and expenses. You can download the same spreadsheet populated with sample data here.
Measurable objectives in the spreadsheet include, but are not limited to:
- Number of government, non-profit and business partners
- Types and numbers of products or materials distributed to each media/PR outlet
- Types and numbers of ads for which you bought placement
- Number of inquiries received from your target audience(s)
- Program expenditures
- Cash donations and dollar-value of in-kind support for your initiative
Here are some examples of record-keeping tools to help you answer process-related questions.
- How many copies of initiative materials were delivered to each distribution channel?
- How To Measure:
- Regular inventory of materials
- How To Use Information: To keep an accurate count of materials available
- Did your coalition members distribute/use initiative materials?
- How To Measure:
- Follow-up telephone calls to coalition members
- How To Use Information: To determine if coalition members are effectively using and distributing materials
- Did your coalition members complete tasks for which they were responsible?
- How To Measure:
- Survey of coalition members
- Review of meeting minutes
- How To Use Information: To modify assignments or responsibilities, as appropriate
- Did the media use the materials (e.g., PSAs/paid ads, press releases, etc.) provided?
- How To Measure:
- Audit of publications
- Media monitoring services (e.g., Burrelle's press clipping service or Video Monitoring Service)
- Follow-up telephone calls
- How To Use Information: To track media coverage and to determine if additional lead time or coalition-member involvement are required to encourage use of the materials
- If your materials included a contact phone number or Web address, how often did your target audience contact you, and how did you respond?
- How To Measure:
- Form to track phone inquiries and length of time taken to reply
- Web site tracking form or hit counter
- How To Use Information: To determine if and how the information is reaching the target audience(s)
- How many people were exposed to the message(s) through program activities?
- How To Measure:
- Event attendance records, such as sign-in sheets and head counts
- Print circulation and viewer/listener information
- How To Use Information: To measure the number of participants reached through specific activities
- How much time was devoted to planning and implementing the initiative?
- How To Measure:
- Track number of staff hours (paid and volunteer)
- How To Use Information: To determine how staff allocated their time and make decisions regarding future use of staff
- Were the funds designated to implement the program sufficient?
- How To Measure:
- Track budget requests and expenditures
- How To Use Information: To ensure the budget is being allocated effectively
- Were activities completed on schedule?
- How To Measure:
- Develop program timelines and track schedules
- How To Use Information: To determine if deadlines are being met
Impact
In order to determine the impact that your clean air campaign is having on your target audience, it is important to administer a survey with a standard series of questions over an extended period. This will enable you to develop a set of metrics that will be very helpful in determining how successful your program has been in delivering its message. You will therefore need to administer your survey before the clean air campaign begins as well as during/after it is complete.
Although you may need to develop your own survey, through extensive research and testing, the federal partners learned quite a bit about the kind of questions that can (and cannot!) yield meaningful information on the effectiveness of transportation and air quality public education programs. Therefore, the impact evaluation questions linked below are aligned with realistic objectives of changing awareness, knowledge, and attitudes about transportation choices that contribute to cleaner air and reduced traffic congestion.
Simple steps in creating an impact evaluation:
- Determine who to survey—The target audience for the It All Adds Up messages is active licensed adults. Determine if this is your target audience. Only survey those whom you plan to expose to your messages, or use those who are not exposed as your "control" group and increase your sample size.
- Determine what survey method to use—A telephone survey is the most common method for conducting impact research, because it works well in collecting data on awareness and knowledge, as well as attitudes. It offers considerable control over how respondents are selected, can be completed in a reasonable time period, and its procedures can be replicated. For help in locating a company, use one of the directories referenced here in this section or your local business directories under "market research and analysts." Other possible methods include a mail survey, a self-administered questionnaire to a group of people, an intercept survey, a door-to-door survey, and a Web site-based survey.
- Create (or decide on) a specific survey—You may need to create a survey of your own, however It All Adds Up to Cleaner Air has created some surveys to help you get started.
- Determine sample size—Relatively large samples of your target audience (e.g., 500 people) are needed to demonstrate modest changes in awareness, knowledge, and attitudes. Consult someone with knowledge of statistics for greater detail. The American Statistical Association Web site also is a good source for information about sampling and statistical questions however the chart below should provide a decent starting point. In most cases, an estimate that is within four percentage points is sufficient for the analysis of awareness and attitudes.
Sampling Margin of Error 95 percent confidence level— For results at or near 50%
Sample Size Margin of Error 50 ±20 percentage points 100 ±10 percentage points 200 ±7 percentage points 300 ±6 percentage points 400 ±5 percentage points 500 ±4 percentage points 750 ±4 percentage points 1,000 ±3 percentage points 2,000 ±2 percentage points - Implement Survey—In order to understand the change in attitudes/behavior it is important to set up a set of initial metrics that future surveys can be compared to. Therefore you will need to conduct a survey before your clean air campaign begins as well as after it is completed (and possibly during the campaign as well to see how attitudes/behaviors are changing.)
- Analyze Results—You will need to compare your results to your benchmark (pre-campaign) survey. You will need to determine the best way to analyze your results. Statistical significance or change varies according to sample size. In general, it is difficult to prove that a change is statistically significant between surveys (and not a sampling error) with a small sample size. For careful in-depth analysis, it is important to check with a statistician however the chart below should give you a decent starting point.
Percentage Point Change between Survey Results to be Considered Significant 95 percent confidence level— For results at or near 50%
Sample Size Needed Difference 50 20 percentage points 100 14 percentage points 200 10 percentage points 300 8 percentage points 400 7 percentage points 500 6 percentage points 750 5 percentage points 1,000 4 percentage points 2,000 3 percentage points
In determining your sample size, think about what you would consider to be a reasonable amount of change to expect, and a change that would influence your decision-making process. Be realistic, so you don't set yourself up for failure. For example, if you would only consider your program successful if you got changes of ten percent or more, and you think that is realistic, then you only need 150 respondents. However, the likelihood of changing awareness and attitudes is more in the range of two to five percent, therefore, a sample size of 500 - 2,000+ is usual for these kinds of surveys.
Here are some sample surveys from other communities:
For more information on opinion and market research as well as information and resources for conducting an impact evaluation, check out the sites below:
- American Association of Public Opinion Research
- Includes guidelines for public disclosure of survey results, a code of ethics, and a directory of research suppliers.
- American Marketing Association
- Includes a wide variety of information about marketing and marketing research, including articles about research techniques, sampling, and a directory of research suppliers.
- Council of American Survey Research Organizations
- Includes a "Surveys and You" section with frequently asked questions, a code of ethics, and a directory of members.
- Dodd's Directories, Inc.
- A directory of market research suppliers.
- Marketing Research Association
- Includes the Blue Book research services directory, a code of ethics, and information about the data collection industry.
- GreenBook
Volume I - Worldwide Directory of Marketing Research Companies and Services
Volume II - Worldwide Directory of Focus Group Companies and Services
http://www.greenbook.org/ - Features the entire GreenBook in a user-friendly, interactive format.
- Qualitative Research Consultants Association
- Includes frequently asked questions about qualitative research and a directory of members.
- Quirk's Marketing Research Review
- Includes articles on a wide variety of research topics and a directory of research suppliers.
- World Public Opinion
- Includes articles on current topics in research, a glossary of research terms, and a directory of research suppliers.
- @ResearchInfo
- Includes a Market Research Roundtable discussion forum, where you can post questions about research, and a directory of research suppliers.
Results
When the campaign is done, it will be necessary to report on the campaign and its relative success. This may be a combination of both subjective and quantitative data or mostly one or the other. You will want to include your program objectives, methodology, results of the evaluation surveys as well as any other pertinent information regarding the campaign. Your report may vary depending on who it is written for. For example a report written for CMAQ funding may be different that that written for the public.
Below are some examples of reports other communities have produced from clean air campaigns:
- Chattanooga-Hamilton County Air Pollution Control Board
- Bi-State Regional Commission
- North Texas Clean Air Coalition Citizen Survey Executive Summary Report
In addition to showing long-term changes in people's behaviors a successful clean air campaign should eventually show corresponding improvement in air quality or traffic congestion. However, isolating the impact of a communications program and proving its effect on long-term behavior requires considerable resources that can equal or exceed the cost of the communications program. Such evaluations may be part of long-term, large-scale government research, beyond the scope of the It All Adds Up to Cleaner Air initiative.