Author Archives: enicaise

Cybersecurity managers and CISOs often ask why they should run regular phishing exercises and not just having their users following a yearly computer-based training (CBT), as they do for most of the other topics.

Beside the commercial sales pitch a security firm could deliver (full disclosure, besides my research activities, I also work as a security consultant), what does sciences says about that?

Not so much, unfortunately.

One of the first attempt to answer this question was made by Aaron Ferguson in 2005. Ferguson is an NSA visiting Professor at the famous West Point US Military Academy. He sent phishing emails to 512 West Point cadets after they received 4 hours of computer security instructions. The phishing email, called the West Point Carronade by Ferguson, tricked 80% of the cadets. While the scenario was quite targeted and the context highly favourable to make this scenario successful, it was quite a success, despite the training.

In 2010, Davinson & Sillence trained users using “Phishing Phil”, an online game about email phishing. Their goal was to evaluate the impact of the level of risk communicated on the users’ behaviour. By their own words “There was no effect of the training programme on secure behaviour in general”. Unfortunately, they did not measure the actual behaviour of the users before and after the training.

In 2013, Jansson & von Solms conducted a series of phishing exercises on an academic institution in South-Africa. He ran four scenarios in parallel on seven different groups, in two waves. The subjects who clicked during the first waves received and embedded training (meaning, the link they clicked or the attachment they opened displayed a warning about their insecure behaviour) and a warning email. They also had the opportunity to follow online training by clicking on a link displayed in the warning page. The next week, the same users received either the same email, either a different one. There was 42.63% less click during the second wave than the first. This seems to indicate that simple feedback and a short training right after having clicked (embedded training) can reduce phishing susceptibility.

In 2019, Gordon et al. conducted a series of phishing exercises on 5416 employees of a US healthcare institution. After the 15th exercise, they identified the “offenders” (those who clicked 5 times and more on the previous exercises). They provided computer-based training to these offenders a continued to measure their results to the 5 next exercises. While the phishing exercises reduced the click ratio for both the offender and the non-offender groups, the CBT provided to the offenders did not decrease the click rates than the non-offenders (low-risk) group.

That is not a lot of data to form an opinion. Still, it seems that a simple message embedded in the message received by people clicking in a phishing exercise’s link is more efficient than a CBT to reduce phishing susceptibility.

References:

• Davinson, N., & Sillence, E. (2010). It won’t happen to me: Promoting secure behaviour among internet users. Computers in Human Behavior, 26(6), 1739–1747. Scopus. https://doi.org/10.1016/j.chb.2010.06.023
• Ferguson, A. J. (n.d.). Fostering E-Mail Security Awareness: The West Point Carronade. Retrieved 3 December 2019, from https://er.educause.edu/articles/2005/1/fostering-email-security-awareness-the-west-point-carronade
• Gordon, W. J., Wright, A., Glynn, R. J., Kadakia, J., Mazzone, C., Leinbach, E., & Landman, A. (2019). Evaluation of a mandatory phishing training program for high-risk employees at a US healthcare system. Journal of the American Medical Informatics Association, 26(6), 547–552. Scopus. https://doi.org/10.1093/jamia/ocz005
• Jansson, K., & Solms, R. von. (2013). Phishing for phishing awareness. Behaviour & Information Technology. https://www.tandfonline.com/doi/abs/10.1080/0144929X.2011.632650

During a scientific literature review of Phishing-related articles, I stumbled onto a fascinating article on the “Deterrent effects of punishment and training on insider security threats” (Kim, 2020). All scientific articles deserve attention, but this one caught mine a bit more as punishment, or at least the fear of it, is considered an ineffective option to reduce risk, at least without providing a way to cope with the threat. This assumption is often based on research performed on health-related communication. They often tend to measure an attitude (how we feel about something) or an intent (what I think I will do in the future in a context) rather than an action.

Also, phishing is, from my point of view, a specific case as it often occurs as an “accident” during a “normal” activity (going through and reading our emails). Hence, it’s likely linked to a lack of good habits and vigilance than disregard for cybersecurity policies.

On the other hand, our vigilance depends on the context. If we consider any email as suspicious, we will probably be less likely to fall for a phishing email. However, it might create an additional cognitive workload and increase the users’ level of stress (or not, it hasn’t been evaluated, up to my knowledge).

Kim et al. tested the effect of punishment in a real-life setting using an exciting paradigm. To avoid the typical laboratory experiment’s contextual impact, they performed their studies in a governmental organization in Korea. They sent a first phishing email to a group of employees. They then split the people who failed the test in two groups: one that received a punishment (a visit of the security team, a temporary loss of the access to the network and a threat for a bad note for its annual performance review) and a second, control, group of unpunished people.

Twenty weeks later, they sent a second phishing email and compared the click ration between the two groups. 17,5% of the punished group clicked on this second email link while 43.2% of the not punished one clicked on it. Although the sample size is relatively limited (101 persons in total for both groups), the effect is significant (p=0,005). Also, it is noticeable that the results were significantly different between people with a low or a high position in the organization, the employees having a low position clicking significantly less than their high position colleagues (punished: 7,1 vs 46,7% – p=0,002 and not punished: 37,8% instead of 71,4% – p=0.210).

These results are based on a tiny sample and must be treated with the necessary scientific doubt. Still, they raise some questions. Twenty weeks is a very long time. It seems more efficient, in the long term than any training (phishing exercises training maintains effectiveness for a month, at most three, depending on the research).

Also, as Kelsey Pipers reminds us in an article published on Vox, context matters. The results obtained in one context can often hardly be replicated in another one. Still, we should put that to the test and measure if any punishment can effectively reduce the risk of phishing in another context.

References:

• Bora Kim, Do-Yeon Lee & Beomsoo Kim (2020) Deterrent effects of punishment and training on insider security threats: a field experiment on phishing attacks, Behaviour & Information Technology, 39:11, 1156-1175, DOI: 10.1080/0144929X.2019.1653992
• Kelsey Pipers (2020) Why we can’t always be “nudged” into changing our behaviour, Vox.com, https://www.vox.com/future-perfect/2020/2/26/21154466/research-education-behavior-psychology-nudging