4

I work in an organisation with 3 levels as far as information security is concerned. I'm sitting at level two where we develop policies and also assist with the standards. One of the most difficult things which have come to light is how to measure compliance to policies. The risk managers at level 1 need some sort of measures to evaluate whether they comply with the set policies or not. I have tried to put metrics (KRIs) together in order to achieve the purpose.

How practical and detailed should the metrics be?

schroeder
  • 123,438
  • 55
  • 284
  • 319
Katlego M
  • 51
  • 8
  • Are these self-written policies for good measure or do you have to comply with certain regulations and do the therefore implemented policies follow the requirements given by the regulations? – Tom K. Sep 19 '18 at 13:14
  • These are all self written policies for good measure. They ofcourse are guided by industry best practice standards. – Katlego M Sep 19 '18 at 13:18

2 Answers2

5

A good information security policy should very clearly state a number of things. At least the following:

  1. The reason why it was implemented
  2. its goals
  3. its scope
  4. its boundaries (the things it does not apply to)
  5. clear requirements towards the assets that are part of the scope
  6. controls that meet these requirements
  7. responsibilities

There are (at least) two kinds of policies that are self-written. The ones that set organizational rules and the ones that set technical rules. The latter are sometimes called 'standard' or 'guideline', for instance 'Guideline for configuring Windows Server 2012'. If you are the person that is giving names, try to give the same name for the policies with the same scope.

A very small policy that sets organizational rules could look like this.

  1. Employees noticed that visitors tend to stare at their screens
  2. To reduce screen staring this policy will be implemented
  3. This policy applies to employees of level 1
  4. It does not apply to employees of level 2 and 3
  5. & 6 All employees are required to lock their screens if they have a visitor in their office. If the visitor needs to look at the screen, because of business reasons, employees have to make sure, that the visitor is only able to see what she needs to see. This can mean to close all other windows, even if this harms productivity while the visitor is there.

7 managers are required to monitor this behavior on a sample basis.

How can you measure compliance with policies that set organizational rules?

You can't really. (and also auditing)

You can ask employees if they comply with the policy. On the one hand this is risky, because your employees will often times answer as they think is socially expected and so will their managers. On the other hand if the process of asking is thorough, this can yield good results. If your employees do not necessarily have to fear backlash if they don't comply with a policy, they will be more honest. A nice side-effect: can also ask them why they don't follow the given rules. There is often a discrepancy between how processes are planned and built - and how they are "lived". You could test employees with regards to what they know about a policy and its details. But then you will only know if they know about the policy, not if they enforce the rules that are set there.

Because of these problems some organizations do the following: when it comes to policies that set organizational rules.

  1. implement policy
  2. employees have to sign statements that they will comply with policies that are published companywide and that apply to them
  3. publish policies companywide and/or to the respective groups

How can you measure compliance with policies that set technical rules?

Auditing.

The best way to do this, is auditing. There are all kinds of audits with all kinds of different scopes. If you want to measure if your controls are effective (= meet the requirements) and if they have been implemented correctly you should hire an auditor or look up auditing techniques. I would recommend hiring because 1) audits are tedious and 2) having an outside perspective can be very helpful to find flaws in processes and controls.


Get to the point! How do auditors measure compliance?

Well... there are different techniques which really depend on what you want to measure.

There is the basic: X machines of Y, so Z% have vulnerability with a CVSS score of 7 or higher. - You'd measure compliance with a policy that sets rules with regular vulnerability scans. Compliance will most of the time come down to "yes/no/partly". The rules that have been set in a policy could look like the ones you can see here.
For instance:

"The auditor should verify that management has controls in place over the data encryption management process. Access to keys should require dual control, keys should be composed of two separate components and should be maintained on a computer that is not accessible to programmers or outside users. Furthermore, management should attest that encryption policies ensure data protection at the desired level and verify that the cost of encrypting the data does not exceed the value of the information itself. All data that is required to be maintained for an extensive amount of time should be encrypted and transported to a remote location. [...]"

These can be very easily measured with interviews of experts and technical audits. For instance: what TLS version is in use, how is back-up data encrypted, etc.

There are also more complex goals that can be set up before an audit but these are probably not what you are looking for, because they require complex processes and controls to audit. These things are normally not set up via policies.


In the end it comes down to the auditor and the technique she chooses. If you hire someone, arrange this beforehand.

If you don't choose to audit you will have to define how much risk is reduced after a policy has been implemented and: 1) employees don't know about the policy, 2) employees always comply with the policy and all the varying degrees between these two.

Tom K.
  • 7,913
  • 3
  • 30
  • 53
  • Hi Tom, thanks for the input. We hjave internal Audit at level 3, so the plan is to self assess ourselves before audit comes. I figured we could use metrics but some policy statements are not measurable and your input has really helped me to understand that. – Katlego M Sep 20 '18 at 06:52
  • Hi @KatlegoM, could you name some examples of policy statements like these? – Tom K. Sep 20 '18 at 07:16
  • Ensure the completeness of the governance structure. Improve staff morale and attitude towards identifying/reporting security incidents. – Katlego M Sep 20 '18 at 09:44
  • 1
    1. governance structure: Define completeness for yourself (100%) and analyze the level it is at right now (X%), set a level of expectation for future years and steps that can be done (+Y%). This will maybe involve some guess work. 2. Make some assumptions: How many information security incidents have there been per year over the last X years, while X is the number of years ago, the policy has been implemented. Has the number of reported incidents gone up or gone down? Also: ask your employees if and why they (do not) report an incident. Most important factors are: fear, knowledge, awareness. – Tom K. Sep 20 '18 at 11:50
0

I would suggest to go for SMART objectives to measure the performance effectively and accurately.

Ensure you have answers for following questions when defining the SMART objective/KPI:

Is the goal Specific?

Is the goal Measurable?

Is the goal Achievable?

Is the goal Relevant?

Is this goal Time-specific?

The following are some of the examples of such SMART objectives/KPIs:

enter image description here

Some examples on SMART KPI/Objectives for Compliance side: enter image description here

Source: http://isoconsultantpune.com/establishing-information-security-objectives/

Sayan
  • 2,033
  • 1
  • 11
  • 21
  • Note that the OP used the term "KRI", not "KPI". You can't typically use SMART measures against KRIs. Your table does not include things that measure against policy compliance. And depending on the policies in question, it can be impossible to derive SMART measures from them. – schroeder Sep 19 '18 at 18:59
  • If I'm not wrong he has mentioned, the team needs measurements and they have tried KRI as well (not only he talks about KRI)... For measurement, you can use KPI/Objectives and added samples of SMART KPI for compliance for reference... – Sayan Sep 20 '18 at 00:23
  • Hi, Thanks for your input. I focused on KRIs because as an IT and security risk manager, I ought to wear my Risk management hat while trying to solve this. However, I still believe that your Smart approach is somewhat relevant because the level 1 people need practical measurenments to assess themselves for compliance. I was thinking along the lines of incorporating metrics which are more specific. Here is an example below: – Katlego M Sep 20 '18 at 09:54
  • - Policy Objective - Establish processes to report and manage out of band security incidents and information breaches - Indicator of compliance - All out-of-band communications are communicated via Telegram - Testing Plan - Check for any out-of-band communication which are reported through unapproved channels such as WhatsApp, Hangouts and other IM applications. - Frequency - Monthly - KRI (Metric)- Number of out-of-band incidents reported using other IM channels - Responsibility - Incident Response Team – Katlego M Sep 20 '18 at 11:11