CHECK: Guidance on the Five Metrics

Five metrics factor into an operator’s determination of program effectiveness: Reach, Recall, Understanding, Behavioral Intent, and Achieving Program Objectives.

Four of the five key metrics should be assessed as part of an operator’s effectiveness evaluation for their public awareness program:

  • Reach: Did the program (or elements of the program) reach the intended audience(s)?
  • Recall: Does the audience remember receiving information from the operator?
  • Message Comprehension: Did the audience understand baseline messages or content provided by an operator?
  • Behavioral Intent: Did the audience state their intent to take certain actions we want them to take? Do they intend to behave in alignment with the guidance or messaging provided through the operator’s program?

Recall as a data point in effectiveness measurement is new to the Third Edition, along with the shift towards behavior intent rather than behavior change as a required measure of program effectiveness. Learning from measurement of similar programs in peer industries, behavior intent allows an operator to gauge whether a stakeholder will take a specific action or align their behavior with the guidance or messaging provided. This is a much more meaningful measure of program effectiveness and is actually measurable.

The final metric is achieving program objectives. Although operators may choose to add additional program objectives, they are required to develop objectives based on:

  1. Awareness
  2. Prevention
  3. Recognition and Response

A Public Awareness Program’s ability to meet its objectives is an essential element or measure of the program’s effectiveness and can be determined—at least in part, depending on how the objectives are written—through reach, recall, message understanding, and behavior intent.

Although no longer required as a measure of effectiveness, “bottom line results” continue to be essential in determining whether program objectives have been met and overall program effectiveness. Examples of bottom-line results can include:

  • stakeholder requests for further information
  • stakeholder reports of possible damage, unauthorized activity on a right of way, leaks, or a potential incident
  • one-call data
  • number of encroachments
  • reports of damaged pipeline markers
  • third-party incidents that did or did not result in a release of product
  • participation in liaison meetings
  • findings from post-incident reviews on third-party incidents
  • third-party near-miss events with or without a valid one-call notification