What is the difference between a penetration test and a vulnerability assessment?

Question

What is the difference between a penetration test and a vulnerability assessment?

Why would you choose one over the other?

What deliverables would you expect to receive and how would you rate the quality of them?

Answer 1

I'm sure I posted an answer to this previously, but my google-fu must be weak this morning. From my blog post on Penetration Taxonomy, we have a list of testing types that is gaining acceptance. Also, working with the Penetration Testing Execution Standard we hope to further develop this. This list should help explain how to choose one over another.

Deliverables are almost a separate issue - and should be defined by the need and the audience (eg for a governance body you would not expect the same detail as you would provide to a technical remediation team, but you would want to include business risk information). There is also a Reporting stream within PTES development to try and codify this area:

Discovery

The purpose of this stage is to identify systems within scope and the services in use. It is not intended to discover vulnerabilities, but version detection may highlight deprecated versions of software / firmware and thus indicate potential vulnerabilities.

Vulnerability Scan

Following the discovery stage this looks for known security issues by using automated tools to match conditions with known vulnerabilities. The reported risk level is set automatically by the tool with no manual verification or interpretation by the test vendor. This can be supplemented with credential based scanning that looks to remove some common false positives by using supplied credentials to authenticate with a service (such as local windows accounts).

Vulnerability Assessment

This uses discovery and vulnerability scanning to identify security vulnerabilities and places the findings into the context of the environment under test. An example would be removing common false positives from the report and deciding risk levels that should be applied to each report finding to improve business understanding and context.

Security Assessment

Builds upon Vulnerability Assessment by adding manual verification to confirm exposure, but does not include the exploitation of vulnerabilities to gain further access. Verification could be in the form of authorised access to a system to confirm system settings and involve examining logs, system responses, error messages, codes, etc. A Security Assessment is looking to gain a broad coverage of the systems under test but not the depth of exposure that a specific vulnerability could lead to.

Penetration Test

Penetration testing simulates an attack by a malicious party. Building on the previous stages and involves exploitation of found vulnerabilities to gain further access. Using this approach will result in an understanding of the ability of an attacker to gain access to confidential information, affect data integrity or availability of a service and the respective impact. Each test is approached using a consistent and complete methodology in a way that allows the tester to use their problem solving abilities, the output from a range of tools and their own knowledge of networking and systems to find vulnerabilities that would/ could not be identified by automated tools. This approach looks at the depth of attack as compared to the Security Assessment approach that looks at the broader coverage.

Security Audit

Driven by an Audit / Risk function to look at a specific control or compliance issue. Characterised by a narrow scope, this type of engagement could make use of any of the earlier approaches discussed (vulnerability assessment, security assessment, penetration test).

Security Review

Verification that industry or internal security standards have been applied to system components or product. This is typically completed through gap analysis and utilises build / code reviews or by reviewing design documents and architecture diagrams. This activity does not utilise any of the earlier approaches (Vulnerability Assessment, Security Assessment, Penetration Test, Security Audit)

Answer 2

In addition to the answers already provided, I'll chip in on why you may choose one over the other. Vulnerability assessments are more useful if you're not sure of your current security posture and want to get an idea of where your issues may lie.

It's worth noting though that like all security work vulnerability assessments are not all created equal. For some vendors it may focus purely on tool output, where others will do more manual work to verify and extend those findings.

If that's your goal I'd focus on the "security assessment" type of approach mentioned in @RoryAlsop's answer.

A Penetration test is, classically, designed to replicate the actions of an attacker in attempting to compromise the security of a system or organisation. So this is likely to be more appropriate for orgaisations who are confident about the security controls in place for a given system and want to prove or disprove their efficacy.

There are problems with this approach however, depending on who your attackers are. If you're looking to design controls that will defend against skilled targeted attackers (eg, organised crime) then it's very hard to effectively use a penetration test to check those, as it won't include some techniques that the attackers can use.

Some examples of areas that penetration testing will have difficulty covering due to legal issues would be things like targeting staff home PCs to compormise their remote access facilities, attacking 3rd party partners who may have access to the target systems for support purposes, or buying botnets which may have zombies already present in the target environment.

So it's worth being aware of the limitations of both types of testing, but a general rule of thumb is that VA and security assessments are good when you're not sure how secure you are, and penetration tests are good for proving it once you do know.

Answer 3

The trouble with this question is that there is not a strict/followed definition for what "penetration test" means in the industry. Some brilliant people from across the community got together to solve that problem, forming the "Penetration Testing Execution Standard."

It attempts to define each stage of a penetration test in order to establish a reasonable expectation on top of which business executives and Penetration Testers can base their interactions.

The stages they list are

1. Pre-engagement Interactions
2. Intelligence Gathering
3. Threat Modeling
4. Vulnerability Analysis
5. Exploitation
6. Post Exploitation
7. Reporting

Here is a watered down definition of each. To get a clear picture, you should read the linked wiki pages.

Pre-engagement Interactions involves establishing scope and the rules of engagement, along with many business-oriented aspects.

Intelligence Gathering is the reconnaissance phase wherein the attacker learns as much as it can about the target (both human and technical aspects) for use during vulnerability analysis and exploitation.

Threat Modeling involves identifying assets and threats, categorizing them, and finally associating them.

Vulnerability Analysis "is the process of discovering flaws in systems and applications which can be leveraged by an attacker." People often mistakenly assume that this and exploitation is all that penetration testing is about.

Exploitation "focuses solely on establishing access to a system or resource by bypassing security restrictions."

Post-Exploitation is the determination of the compromised machines value and the maintaining of control for later use. In this phase you may gather information that allows you to go deeper (the obvious being credentials).

Reporting "[communicates] the objectives, methods, and results of the testing conducted to various audiences."

Whether they are succeeding in their mission or not is up for debate, but I believe this is a good place to develop start for a thorough understanding.

There is also the "PTES Technical Guidelines" which goes over tactics and tools that might be used during a penetration test.

Answer 4

To most people -- they are the same. This is why efforts like PTES will fail. You cannot change people who do not want to change.

The correct question is, "What is the difference between penetration testing and ethical hacking?".

The answer to this question is that penetration-testing has a specific prize in mind, with no time limit and usually a long list of rules of engagement (the short list ones ask that nobody be physically harmed or threatened). Ethical hacking is time-boxed activity where as many flaws as possible are uncovered -- typically only using non-physical methods.

"Assessments", "audits", and "tests" are typically interchangeable words in the field of information security. The words "vulnerability", "threat", and a few others are typically misunderstood and misinterpreted. Professionals with 20 years of experience, the same background, and the same certifications will typically argue over these basic terms. It is best to ignore what someone "says" or "thinks" is the right answer -- and instead go with your gut instinct and common sense when applying the term in a conversation with the uninitiated.

Answer 5

I’m not game to write something long for this, but here’s a fun one that most pen-testers share:

• I have a client that has received passing PCI ASV scans for years (i.e. an external vulnerability assessment with no severe, high, or critical vulnerabilities found). A “clean” scan according to PCI.
• And for the last several years all of their internal databases can be exfiltrated, undetected, via skilled pen-testing (not to even mention client-side attacks, social engineering, physical pen-testing, phishing)

Pen-testing, at least in the corner I hang around in, is intended to simulate an attack by a motivated adversary. Vulnerability assessments are typically something much less.

Answer 6

Vulnerability Analysis is the process of identifying vulnerabilities on a network, whereas a Penetration Testing is focused on actually gaining unauthorized access to the tested systems and using that access to the network or data, as directed by the client. A Vulnerability Analysis provides an overview of the flaws that exist on the system while a Penetration Testing goes on to provide an impact analysis of the flaws identifies the possible impact of the flaw on the underlying network, operating system, database etc. Vulnerability Analysis is more of a passive process. In Vulnerability Analysis you use software tools that analyze both network traffic and systems to identify any exposures that increase vulnerability to attacks. Penetration Testing is an active practice wherein ethical hackers are employed to simulate an attack and test the network and systems’ resistance.

I have written a full post covering this. Read it here: http://www.ivizsecurity.com/blog/penetration-testing/difference-vulnerability-penetration-testing/