6

I'm looking at an organization that requires that all employees undergo an annual one-hour online cybersecurity training (watch a video and take a quiz, apparently built using SANS's end-user security awareness training).

Is there any evidence on whether this is effective and how effective it is? For instance, is there any measure or estimate of how much this kind of online cybersecurity awareness training reduces security incidents, or improves outcomes in some other measurable way? Should one expect to see a 5% reduction in security compromises? 50% reduction? 0% reduction?

I did check on SANS's Securing the Human site, figuring that if there was any evidence or quantitative data they would show it prominently, but they don't seem to say anything about it.

D.W.
  • 98,420
  • 30
  • 267
  • 572
  • Could you disambiguate "awareness presentation" from "training" when you use the term "awareness training" which looks at the frontier between the two? – dan Nov 03 '15 at 22:14
  • Related but not identical: http://security.stackexchange.com/questions/63430/is-user-education-actually-doable – Scott C Wilson Nov 09 '15 at 19:40
  • Can we deduce the efficacy of training from the fact that bad guys don't just attach .exe files to spam mail that much anymore? (They use infected Office docs/PDFs or use Exploits to deliver malware.) – Scott C Wilson Nov 09 '15 at 19:48
  • @ScottWilson, no, we can't. (An alternative hypothesis is that spam filters have gotten better at detecting those kind of attacks and have made those attacks less effective.) – D.W. Nov 09 '15 at 19:50

1 Answers1

4

The efficacy of a 1-hour annual online series of videos to affect user behaviour is very low (industry stats are 0-5% change in behaviour - perhaps statistically insignificant). Compliance is higher within the days after training, but then trails off very quickly. This type of training needs to be coupled with other supports in order to see results, but it is possible to see positive results up to 70% (consistent adoption of targeted behaviours) with certain supplements to this kind of training. Repetition of training and support and follow-up are perhaps more important than the knowledge transfer itself.

As for supplements, the most effective methods include regular prompting of behaviours and providing immediate feedback to the user as to the correctness of the presented behaviour. The most common form of this is simulated phishing, but it can include any behaviour the organization wishes to see.

In some phishing simulation programmes, studies have shown a decrease in users clicking links in emails of up to 70%. This usually requires regular testing and users slowly learn what to do over time. The key is regularity and immediate feedback.

This same approach can be done for password policies, tailgating, locking computers, incident reporting, USB device handling, etc.

Awareness is awareness. Knowledge transfer is knowledge transfer. But behavioural change is a different ballgame. It starts with knowledge (sometimes), but then it needs to transition to action. And that can't be done with a 5 minute video.

(I am writing a book on this very topic)

schroeder
  • 123,438
  • 55
  • 284
  • 319
  • Agreed. Awareness is defined by NIST Special Publication 800-16 as follows: “Awareness is not training. The purpose of awareness presentations is simply to focus attention on security. Awareness presentations are intended to allow individuals to recognize IT security concerns and respond accordingly." Thus, while awareness is important to disseminate IT security needs of the organisation, the effectiveness of the training is dependant on a lot of other variables as well. https://www.sbs.ox.ac.uk/cybersecurity-capacity/system/files/Awareness%20CampaignsDraftWorkingPaper.pdf – Joe Nov 03 '15 at 06:27
  • *"If you are hoping that awareness will result in lowered security incidents or lowered occurrences of non-compliant behaviour"* -- yes, I sure am! Or rather, whether or not it does have that kind of result is exactly the question I'm asking. Training would be pointless if it didn't lead to improvement in security outcomes, wouldn't it? I mean, we don't do security awareness training because we think security awareness is good in its own right, but rather because it is a means to an end: we think it will improve security and help the organization resist attacks more effectively. – D.W. Nov 03 '15 at 06:56
  • @Joe, I'm a bit confused by your comment. I wonder if perhaps my use of the word "awareness" has caused some miscommunication? See [the Securing the Human web site](https://www.securingthehuman.org/enduser/), where they say "our goal is to not only ensure you are compliant but provide training that changes user behavior and helps your organization manage risk" - in other words, it is explicitly about influencing behavior, and it is training. As far as other variables, I'm fine with an answer that is caveated in that way, but I still wonder if there are any measurements at all. – D.W. Nov 03 '15 at 07:00
  • @schroeder, wow, that sounds extremely useful and exactly the sort of thing I was asking about! If you might feel inspired to supplement your answer with that information and some examples or characterization of the supplements, that'd be awesome and exactly what I was wondering about. – D.W. Nov 03 '15 at 21:37
  • @D.W. edited. "Now, with more specificity!" – schroeder Nov 03 '15 at 21:52
  • @schroeder, excellent! Thank you. Can't wait to see your book. (Don't be shy about plugging it in this answer when it comes out, if you feel comfortable with that!) – D.W. Nov 03 '15 at 22:02
  • @D.W. I actually feel odd plugging my book since I'm a mod, but we'll see how it goes - link to book's promo page added. – schroeder Nov 03 '15 at 22:29
  • So even under the best hypothetical training/education regimens studied almost a third of simulated phishing attacks still succeed in getting the user to click. That's ... depressing, if entirely unsurprising. (Actually, I'm a little surprised on the positive side that any programs have ever made that such of a measurable difference, even in a lab.) – mostlyinformed Nov 04 '15 at 00:23
  • @halfinformed I run a phishing research platform. In a recent test against 150 people in an organization, I had 6 clickers within 60 seconds of sending a blast email. Even after the organization "circled the wagons" sent out company-wide alerts, etc, I still got a depressingly high number of clickers. Phishing really is the best way into an organization. – schroeder Nov 04 '15 at 00:39
  • 1
    @Schroeder Yeah, that's the reality that everyone's start to come up against nowadays. It's funny, but you even get that from pentesters & tool writers who specialize in *other* penetration vectors. I was watching 2 or 3 conference presentations on YouTube last week about using web app vulnerabilities to get past the perimeter and I'd swear they all started with the same disclaimer: "Now, of course, uh, you know the best way to make your initial breach is usually going to be malware delivered via phishing. But in case you want to do it another way... [begin presentation topic]" – mostlyinformed Nov 04 '15 at 01:04