Should we build a generic security unit test suite

Question

As part of our secure SDLC, we are delivering security as security requirements directly into the backlog of an application team. The security requirements will directly correlate to the functional features being delivered in the iteration. I am predominantly talking about application-based controls (code level) here.

As part of that, we are looking to encourage unit testing for those requirements as a mechanism for fully validating their implementation and going one step further we are hoping to foster the practice of TDD - i.e. the developer will develop a unit test for security functionality e.g. input validation for path traversal, etc.

I have read that it would be a good idea for us to develop a generic unit test suite for this.

Now I have to ask - Would it?

Would it really be a good idea for us to develop a bunch of JUNIT tests to integrate into development teams projects? Is there not an issue around flexibility here in that we would pretty much be standardising the how, the parameters, and the naming of security functions for development teams?

Maybe there is another way we could develop generic unit tests that would allow developers to call their bespoke methods without them following the method signatures of our test frameworks that I am not aware of? I imagine the overhead for integrating this would be huge!

Does developing the test not fly in the face of TDD in that the developer doesn't need to "think about what he is delivering and write a test first"?

My gut says the test suite should be sudo code for each requirement with outliers and negative and positive scenarios that the developers must implement - this way it wouldn't matter if we are writing it in C, C# Java or python etc.

Is this not a better approach?

Answer 1

Yeah, I'm having the same debate with my teams and I am equally conflicted about it.

Arguments for security unit tests:

• Reusability / automation of security testing (ie you can re-use security unit tests against new products / features without needing time from the appsec team for manual testing)
• Regression testing: continue testing for these security defects in the future, not a point-in-time manual test.

Arguments against security unit tests:

• pen testing is often highly intuition-based relying on experience and training of the security tester, and typically not things you would think of if you just sat down to define the positive and negative test cases, and/or not even possible to test via unit tests. Pen test cases are also often complex. For example, is someone likely to write a junit test for "Do the login flow up to but not including MFA; take that mid-flow JWT and try calling APIs with it". Or "Create a JWT with sub= for each value in this XSS payload list then navigate around the UI and see if any of the javascript payloads execute". In fact, I'm not even sure how you do the later in junit.
• junit tests are usually testing individual methods. You may find real security issues there, but the high-value security issues typically happen at the boundary of software layers; ex.: you have a GET controller that assumes CSRF protection is being applied to it, but Spring is only configured to do CSRF on POSTS. Or tomcat is listening for /admin reqs on :666 and the firewall had better be blocking that externally. (plus the ones in the previous bullet; mismatch between login and access control parts of the code; a mismatch between JWT and UI).

I guess where I've landed is that some of your manual exploratory appsec testing will be automatable. Like, at the end of a Burp or ZAP session, go through your history, find things that would be good automation candidates, export as curl, throw them into unit tests. Great, value added. But recognize that the high-value security issues will likely not show up in unit tests, so question whether this is the best use of your appsec time budget.