Using OpenSSL as a library versus calling output from standalone command in a program being written

Question

In another question about how to use OpenSSL programmatically a community member was writing, one suggestion was to use the system() command whereas another was to include the C headers and work with the library using functions directly. Also, if using system() is there a security benefit to using popen() or another function instead?

What would be the security implications of running OpenSSL as a library versus calling a compiled executable from the system? I imagine there might be issues about integrity and authenticity relying upon openssl installed on a system in some cases, but its also possible an application might use an outdated library. Are there certain considerations for this scenario, or can this be answered more generally (If so, please feel free to edit the question).

Answer 1

As a rough summary, OpenSSL has been designed as a library and was meant to be used as such. The command-line utility was, initially, a tool for testing and for manual tasks. Potential issues with invoking the command-line utility from a C application, with system() or popen(), include the following:

• The tool expects its inputs as files, which must be produced and managed. Some of them are sensitive, e.g. files which contain private keys; you need to exercise great care when copying them.

• The tool produces its outputs as files or text on its standard output; the text is meant for human consumption, not for automatic parsing. A number of corner cases can be troublesome (e.g. if you use openssl as a way to analyse the contents of X.509 certificates, what does happen if a string within the certificate contains a newline character ?). The format of text output might vary with details of the system configuration (e.g. system-wide language or encoding).

• Error reporting will be hard. With the C API, you can know exactly which step failed and why; with the command-line tool, you will get, at best, a synthetic error status ("something failed") and a text message.

• system() and its brother popen() are a definite no-go in setuid applications (that's the king of "get root shell" holes of the 1980s).

To sum up, calling openssl (the command-line utility) from C code is cumbersome, inefficient, and complex. Complexity alone almost mechanically implies security issues.