What are some metrics to be used to evaluate a SaaS app's security?
Some examples:
In case it isn't obvious, the code is available to audit and measure.
In a SaaS model, you will likely not be able to perform code audits. This is because as stated, it is Software as a Service. Not software where you would be able to download the software itself, let alone the source code for the software. That task would be done at the discretion of the software vendor, and or provider. Please see "5 Problems with SaaS Security"
So I will document issues I have with SaaS, and SaaS Security.
Security when it comes down to software you purchase is never your responsibility. When you buy a car, and the automaker tells you they went through (in the US at least) IIHS (Insurance Institute for Highway Safety) testing that established a baseline of safety, have you ever said: "I don't believe them, I need to crash my car for myself to make sure the airbag deploys." Many larger SaaS vendors have security measures, and testing in place. (Google, Microsoft, Salesforce, etc)
So let's put this into an alternative view now. You test the software, find vulnerabilities. So what. Now what. Do you fix them on your own? What makes you think a SaaS provider would allow you to randomly test their software "pre purchase." There has to be some level of trust, and while it can be said: "Trust but verify" in the business world, this is where Service Level Agreements, and Terms of Service agreements come into play to mitigate/transfer the risk.
Many SaaS providers do take security serious for example see Microsoft's stance on this:
Office 365 and Microsoft Dynamics CRM Online internal monitoring includes automated compliance monitoring of infrastructure (e.g., vulnerability scans, penetration testing and testing of process and people controls). The Office 365 and Microsoft Dynamics CRM Online third-party validation program includes independent audits that are conducted on an annual basis to provide verification of Office 365 and Microsoft Dynamics CRM Online's security posture.
What exactly is it you are trying to accomplish? Trying to be the security team for a vendor?
I can think of following scenarios:
Safety against SQL injection and login related attacks if your software has user accounts
The accuracy and robustness of your access control if your software has users
The SaaS offerors are not likely to agree to let you pen test their services. They might agree to independent testing but the testing frameworks are not likely to be comparable. So you will be hard-pressed to compare the findings by one to that of another.
Unless you are soliciting $1B in business, perhaps FedRAMP certification is the best you can hope for. Take a look at the FedRAMP Templates for the evaluation criteria GSA uses for cloud services providers (CSP). Program Overview Documents are heavily influenced by FISMA and NIST (this is a US Government Program) so you may not be comfortable with the methodology. However, given the thought and energy put into FedRAMP, you'd be hard pressed to match it in scope and thoroughness. Moreover, the FedRAMP methodology and process has been extensively road-tested over the program's five year history.
There are downsides to using FedRAMP. Many security practitioners point out that the USG has a less than stellar track record for cybersecurity. (Think OPM breach.) Also they argue that the FISMA methodology is overly complex and fails to place due emphasis and priority on contemporary threat vectors. As a general cloud services solution, you may find it not well suited to assessing specific SaaS applications. For example, there are likely to be more suitable approaches for comparing SAP to PeopleSoft than FedRAMP. Lastly, the assessments by Third-Party Independent Assessors (3PAOs) are paid for by the respective CSPs so the assessments may not be consistently excellent.
If you should choose to roll your own evaluation, nevertheless you would do well to consider the Security Assessment Plan (SAP) Template. This document is designed for FedRAMPS's 3PAOs to use for planning security testing of CSPs. Once filled out, this document constitutes a plan for testing. Actual findings from the tests are recorded in FedRAMP security test procedure workbooks and a Security Assessment Report (SAR), also available on the FedRAMP Templates and Documents pages.
