CHECK: Program Evaluation
There are two primary required components to evaluate public awareness programs: the Annual Implementation Review and the Effectiveness Evaluation.
Annual Implementation Review
The Annual Implementation Review must be conducted each year and, at its core, is simply intended to confirm that the operator’s program was implemented. However, it also provides an opportunity to change course if, for example, a particular initiative is not performing as expected or stakeholder feedback indicates a change may be warranted.
Annex B provides sample annual implementation review forms that can be used in part, in whole, or not at all.
Annual reviews should be documented and can be considered as input for the Effectiveness Evaluation.
Effectiveness Evaluation
As in the First Edition, operators are still required to conduct an effectiveness evaluation at least once every four years.
The purpose of an effectiveness evaluation is threefold:
- Determine if program objectives are being achieved,
- Review and understand the data or metrics for each stakeholder audience, and
- Determine if changes to the program are warranted based on a determination of program effectiveness.
So, how do you determine if your program is effective? First and foremost, make sure program objectives are well-defined and achievable. Strong objectives will give you direction on implementation and will also provide something specific to measure program outcomes against.
Feedback from stakeholders can be an important factor in determining program effectiveness. Did they know how to reach you/the operator? Did they know when/why to call? Are they reporting potential damage to a pipeline or requesting removal from a direct mail list?
Are there any trends that stand out when comparing data over program years? How about case studies?
If changes have been identified and implemented from findings from the Annual Implementation Review or prior Effectiveness Evaluations, what outcomes have you noticed? Are your numbers or stakeholder feedback changing as you implement your program differently?
Some operators may choose to utilize the Public Awareness Program Effectiveness Research Survey (PAPERS) program to measure the effectiveness of their public awareness programs among their four key audience groups. PAPERS is conducted every two years to satisfy the Program Evaluation requirements within RP 1162. More information on the PAPERS Program is available here.