End user security awareness measurement

Question

Apart from the conventional email phishing tests, what other security Key Performance Indicators can be used to measure end user security awareness in an Organization?

---

Looking at the SANS critical security control #9:

• 9.4 Validate and improve awareness levels through periodic tests to see whether employees will click on a link from suspicious e-mail or provide sensitive information on the telephone without following appropriate procedures for authenticating a caller; targeted training should be provided to those who fall victim to the exercise.

I am trying to come with an indication that looks beyond the phishing emails or providing sensitive information over the phone. For example, how would I know people are aware of the dangers of using infected USBs in the organisation? Disabling the use of the USB prevents the security breach but does not benifit end user awareness of the issue. Users usually see it as an annoying control that stops them from performing their tasks. USB is just one example, how would I know that users are aware of the dangers of sharing their usernames? and similarly other wrong practices. getting their answers on the subject by surveys is one part, are there other indications?

Answer 1

A rudimentary training program should minimally educate users on critical issues. Measuring its effectiveness provides the opportunity to ensure that users are getting the relevant information they need to do their jobs safely and effectively.

Send out a survey that assesses awareness of job-specific information security issues and see how your employees score.

1. How often should you change your password?
2. What are appropriate methods for transmitting confidential information to a business partner?
3. Whom it is safe to discuss your personal details with over the phone?
4. How can you identify a secure site?
5. How can you identify a secure e-mail?
6. What is "phishing"?
7. What are the security risks in using social networks?
8. What information have you (or would you) put on a social network?
9. How can computer viruses be avoided?
10. What types of incident should be reported?

etc.

Answer 2

Going out on a limb here, I believe in testing by regular drills and Red Team simulated attacks. If the Red Team succeeds in social engineering or other kinds of compromise, the awareness is nil.

Reasoning: KPIs and surveys and whatnot measure and report averages. The attacker is not interested in averages, he's going to attack (citing from a recent comment by one of the Sec.SE denizens) the laptop belonging to the CFO's secretary, or whatever weakest spot he finds.

You are also looking for anti-complacency measures. I cannot find a better way to keep users on their toes than regular drills, spot checks and the very real threat of an actual attack.

Answer 3

What we decided for one KPI is the amount of incidents reported. In the current phase, we say we want more incidents to be reported: not enough people report/are aware they should. When the amount rises, this will mostly be because of increased awareness. From a certain point (no real steady increase in reports) we will switch to Less incident reports means less incidents. This means that we do not rely on reports yet, but will in the future.

Another KPI might be the amount of people actually trained: track them. Third KPI could be the amount of policies people remember by doing a survey. However, I found out people only fill in multiple choice surveys with enough dedication to take the response seriously.

Answer 4

SANS Secure The Human now publishes lists of KPIs for you to consider:

https://securingthehuman.sans.org/resources/metrics

They split it up into "Tracking" metrics (who has completed the training) and "Impact" metrics to try to measure the impact training has had.

• number of people successfully phished over time
• number of people reporting phishing
• number of infected computers
• number of people posting sensitive info on social networking sites
• etc.

Basically, any behaviour that you train for can be turned into a metric (tracking number of people, not the number of instances, for obvious reasons).

In my book (shameless plug) I list a number of non-traditional metric types that can be considered. For example:

• number of false positive reports for users in the first 2 weeks after training or an awareness push
• numbers of repeat offenders
• number of unique visitors to the internal awareness site
• number of queries to the security team
• number of reports of errors in the training material (after you include a couple purposeful errors)
• etc.

You really need to treat this like you would for tracking users on a sales website (ask your web user tracking team, they can help with ideas). You train behaviours and you need to track the change in behaviour over time.

Answer 5

With awareness, what we seek to achieve is a basic understanding of threats and responses. It differs from training and education which pushes for more detailed understanding and knowing the rationale behind the various measures taken.

Therefore, the key to promoting awareness is simply through exposure. The more the staff are exposed to such security messages, the greater the awareness. One easy and cheap way to promote awareness is through printing security messages on mugs, mousepads, magnets etc. and giving them out to the staff.

You can hold contests for designing the above mentioned trinkets and give out prizes. To measure KPI, you can base it on the quantity/quality of submissions. If you are feeling adventurous, you could even plant clues inside these trinkets and award prizes to employees who manage to solve the riddle or problem.

By generating hype and getting more people to talk about it, you would have achieved your objective of awareness.

I am not a fan of surveys or questionnaires as most people simply study to pass it and do not really internalise the knowledge.