There is a saying that one cannot solve a problem if you don’t know it exists. Assuming that you know a problem does exist, then how does one ensure that actions taken to mitigate or resolve that problem are effective? The answer is metrics.
Metrics are standards of measurement 1 that can be used to quantify many things, such as the effectiveness of security awareness training. Metrics are used all the time to measure technical aspects of cybersecurity, but the reliability of metrics for security awareness training is sometimes debated. Part of the debate is around what is used to measure the success or failure of the training. For example, if the only metric used is whether or not employees sat through a mandatory computer-based training (CBT) course, that metric does not demonstrate whether or not the training altered their behaviors 2.
More interactive measurements sometimes involve phishing simulations, USB drops, and Social Engineering simulations to test employee security awareness. The problem is that emphasis is typically placed on the “failure rate” or “click rate,” which can have a negative connotation and present users in a light that emphasizes mistakes rather than successes 3. Let’s focus on phishing and how phishing simulations can be an effective part of a security awareness training program.
Ongoing in-house phishing campaigns help to build “muscle memory” in identifying a phish 4. Additionally, a robust training program will use different phishing templates because the responses will vary from one template to another 3. More sophisticated phishing simulators will include tracking and metrics for multiple phishing campaigns, and for multiple behaviors by those who received phishing emails, including whether the recipient opened the phishing email, clicked on an embedded link or downloaded an attachment, and whether or not the recipient reported the phishing attempt to in-house security 5. While each of these metrics has value overall, the reporting rate is being seen as a more useful indicator of security awareness.
Some security software companies have begun shifting from a pass/fail emphasis in their metrics to one of “resilience,” taking into account how many times a phishing exercise was reported and using that to determine how “resilient” an organization is. For example, if a series of phishing campaigns had an average failure rate of 11%, but the average reporting rate was 13%, the “resilience factor” comes to 1.2 6. This kind of measurement, combined with positive reinforcement such as rewards and public recognition for those who report phishing attempts, can be effective in motivating others in the company to be more attentive to security issues 4.
At the end of the day, the real goal is to help people be more aware of security issues, and how to avoid being victimized by attackers. To do that effectively, security training needs to be more than just “death by PowerPoint,” and it needs to have real-world measurable outcomes. Phishing simulations are cheap and effective training tools, and the metrics they provide over time can provide clear insights into security awareness throughout an organization.
- Definition of METRIC. Merriam-Webster https://www.merriam-webster.com/dictionary/metric.
- Winkler, I. & Manke, S. 4 ways metrics can improve security awareness programs. CSO Online https://www.csoonline.com/article/2134095/4-ways-metrics-can-improve-security-awareness-programs.html (2013).
- Bailey, M. Measuring Up: Metrics, Benchmarks, and Communicating Security Awareness Training Success. Proofpoint https://www.proofpoint.com/us/blog/security-awareness-training/measuring-metrics-benchmarks-and-communicating-security-awareness (2020).
- IANS Faculty. How to Deal with Individuals Who Repeatedly Fail Phishing Simulations. IANS https://www.iansresearch.com/resources/all-blogs/post/security-blog/2022/05/05/how-to-deal-with-individuals-who-repeatedly-fail-phishing-simulations (2022).
- PhishingBox. Phishing Simulator. PhishingBox https://www.phishingbox.com/platform/phishing-simulator.
- Bailey, M. Reporting Phishing Simulations: The Essential Metric to Measure in Phishing Awareness. Proofpoint https://www.proofpoint.com/us/blog/email-and-cloud-threats/reporting-phishing-simulations-essential-metric-measure-phishing (2021).