Basic question regarding OpenSSL and AES-GCM

Question

While researching how to encrypt private keys for SSH connections as securely as possible, I have run into the following very basic understanding problems (Note: I have used the newest stable release 1.0.2h of OpenSSL for my research):

1. man enc states that the enc utility does not support authenticated encryption modes like GCM. On the other hand, I am using TLS 1.2 ciphers like ECDHE-RSA-AES128-GCM-SHA256 on my web server which uses OpenSSL for encrypting connections, and I have checked that those ciphers indeed are negotiated when connecting via SSL / TLS to the respective web site.

   So OpenSSL obviously is able to encrypt data in AESxxx-GCM mode. I would have expected that OpenSSL uses the same code to encrypt data when called via openssl enc as it uses to encrypt data while providing SSL / TLS functionality to web servers.

   Did I misunderstand something? What's the reason that OpenSSL supports AESxxx-GCM while doing TLS / SSL, but can't encrypt data directly that way when called via command line?

2. A common method to view the available cipher names (not cipher suites) is to call openssl enc with a wrong parameter, e.g. openssl enc --help. This makes openssl print a short help, including the available cipher names. In my case, the output includes multiple AES-GCM ciphers. Why does openssl enc claim to support those ciphers while the respective man page claims it does not?

Answer 1

OpenSSL implements almost a dozen symmetric ciphers, and several dozen cipher-mode combinations, but provides a (nearly) single interface to all of them in the EVP module (i.e. external function and type names beginning EVP_) documented here online or in the (crosslinked) man page for EVP_{Cipher,Encrypt,Decrypt}* and EVP_CIPHER_* and EVP_CIPHER_CTX_* on any Unixy system with OpenSSL installed. (If you built/installed it yourself rather than using the package manager or equivalent for your OS or distro, you may need to use MANPATH or other man options to find the man pages.) Similarly numerous digest/hash algorithms and public-key (and hybrid) encryption and signature algorithms are accessed through generic interfaces; see man evp. But the AEAD modes (GCM and CCM, plus OCB planned in 1.1.0) do not exactly fit the generic API and require additional 'control' calls, see the sections 'GCM and OCB modes' and 'CCM mode' in the above man page.

The SSL-and-TLS module (top-level directory ssl/ in the source) contains code in t1_enc.c that does different EVP calls for (implemented) AEAD suites, in the same way it also handles variations for CBC and stream ciphers, obsolete but still coded 'export' ciphers, TLS1.1 vs earlier IV handling, and other protocol options and variations.

But commandline enc in apps/enc.c only uses the generic interface and not the AEAD specials, although there is an unassigned entry in the request tracker (login guest/guest) to add this. The commandline utilities in general are fairly minimal wrappers around functionality in libssl and libcrypto, and if you want something complete, polished, convenient, etc. the idea is you should either modify or replace them. For this case you (would) need to define how the tag, and possibly AAD, is handled in the ciphertext file format, which is currently simple to the point of being trivial -- and remember any change that doesn't work with the probably millions of enc files users have stored over the past two decades won't be accepted by anyone but you. (update) The bugtracker moved to https://github.com/openssl/openssl/issues/471 where it is 'resolved' in 1.1.1 -- by documenting that enc does not and will not support AEAD. Also note that the early 1.0.1 releases, through patch g (2012-03 to 2014-04) failed to give an error message for this case; you could run the enc command with a GCM or CCM cipher, but it discarded the tag on encryption and gave an error on decryption.

Remember also that enc with a password (not actual key and IV using -K uppercase and -iv) uses a very poor PBKDF, a variant of PBKDF1 see EVP_BytesToKey with only one iteration.
See openssl: recover key and IV by passphrase
and openssl enc uses md5 to hash the password and the salt
and https://crypto.stackexchange.com/a/35614/12642 (disclosure: mine). Worrying about using best-practice ciphers like AES256-GCM with this PBKDF is like the proverbial gilding a cow turd.

The usage message for openssl enc -invalid lists all symmetric ciphers/modes in EVP, even those enc doesn't support. If you care, you could report this as a bug. openssl list-cipher-algorithms (planned space instead of first hyphen in 1.1.0) does the same, but openssl list-cipher-commands (ditto) lists only those usable as commands, which excludes the AEAD ones. (updates) In 1.1.0+ all command parsing is rewritten and the usage messages are replaced by -help which for enc does not list ciphers; the explicit commands are now list -cipher-{algorithms,commands} i.e. space and hyphen.

Finally, you mention but don't actually ask about SSH. If you mean OpenSSH (which is not the only SSH), FYI OpenSSH before 6.5 ssh-keygen actually uses OpenSSL libcrypto to write privatekeys in OpenSSL's 'legacy' formats (PEM types RSA PRIVATE KEY, DSA PRIVATE KEY, EC PRIVATE KEY) which also use EVP_BytesToKey with one iteration. But ssh-keygen and ssh and sshd using OpenSSL read routines can also handle OpenSSL's 'new' (circa 2000?) PKCS#8-encrypted format, PEM type ENCRYPTED PRIVATE KEY, which can use PBKDF2 with 2048 iterations through 1.0.2 (good around 2000, barely adequate now) and planned configurable up to INT_MAX in 1.1.0. OpenSSH beginning 6.5 (2014-01) has an option -o for its own (non-ASN.1 but still PEM) format using bcrypt, and forces that option for key type ed25519 (which OpenSSL doesn't support, at least not yet). (updates) OpenSSL 1.1.0 (2016-08) did add -iter N and optional -scrypt* for PKCS8 as expected; OTOH OpenSSH 7.8 (2018-08) made its own 'new format' the default; -o is no longer needed, and if you want to get the old and bad legacy formats you use -m pem.

Answer 2

Because until now, openssl enc does not support AES-256-GCM, I've written the following C source code to do what openssl enc would do:

(compile this way gcc -Wall -lcrypto -o aes256gcm aes256gcm.c)

// AES-256-GCM with libcrypto
// gcc -Wall -lcrypto -o aes256gcm aes256gcm.c

// tag is 16 bytes long
// no AAD (Additional Associated Data)
// output format: tag is written just after cipher text (see RFC-5116, sections 5.1 and 5.2)

// KEY=a6a7ee7abe681c9c4cede8e3366a9ded96b92668ea5e26a31a4b0856341ed224
// IV=87b7225d16ea2ae1f41d0b13fdce9bba
// echo -n 'plain text' | ./aes256gcm $KEY $IV | od -t x1

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <openssl/conf.h>
#include <openssl/evp.h>
#include <openssl/err.h>

EVP_CIPHER_CTX *ctx = NULL;
unsigned char *iv = NULL;
unsigned char *buf_plain = NULL;
unsigned char *buf_cipher = NULL;

typedef enum { false, true } bool;

void freeCrypto() {
  if (ctx) {
    EVP_CIPHER_CTX_free(ctx);
    ctx = NULL;
  }
  CRYPTO_cleanup_all_ex_data();
  ERR_free_strings();

  if (iv) {
    free(iv);
    iv = NULL;
  }
  if (buf_plain) {
    free(buf_plain);
    buf_plain = NULL;
  }
  if (buf_cipher) {
    free(buf_cipher);
    buf_cipher = NULL;
  }
}

void handleCryptoError() {
  fprintf(stderr, "ERROR\n");
  ERR_print_errors_fp(stderr);
  freeCrypto();
  exit(1);
}

bool isValidHexChar(char c) {
  return (c >= 'a' && c <= 'f') || (c >= '0' && c <= '9');
}

unsigned char hex2uchar(char *hex) {
  unsigned char ret;

  if (hex[0] >= 'a' && hex[0] <= 'f') ret = (hex[0] - 'a' + 10) * 16;
  else ret = (hex[0] - '0') * 16;
  if (hex[1] >= 'a' && hex[1] <= 'f') ret += hex[1] - 'a' + 10;
  else ret += hex[1] - '0';
  return ret;
}

int main(int ac, char **av, char **ae)
{
  const EVP_CIPHER *cipher;
  unsigned char key[32];
  int iv_len, len, i;
  unsigned char tag[16];

  if (ac != 3) {
    fprintf(stderr, "usage: %s KEY IV\n", av[0]);
    return 1;
  }

  char *key_txt = av[1];
  char *iv_txt = av[2];

  ERR_load_crypto_strings();

  if (strlen(key_txt) != 2 * sizeof key) {
    fprintf(stderr, "invalid key size\n");
    freeCrypto();
    return 1;
  }

  if (strlen(iv_txt) < 2 || strlen(iv_txt) % 2) {
    fprintf(stderr, "invalid IV size\n");
    freeCrypto();
    return 1;
  }
  iv_len = strlen(iv_txt) / 2;

  if (!(iv = malloc(iv_len))) {
    perror("malloc");
    freeCrypto();
    return 1;
  }

  if (!(buf_plain = malloc(iv_len))) {
    perror("malloc");
    freeCrypto();
    return 1;
  }

  if (!(buf_cipher = malloc(iv_len))) {
    perror("malloc");
    freeCrypto();
    return 1;
  }

  for (i = 0; i < sizeof key; i++) {
    if (!isValidHexChar(key_txt[2*i]) || !isValidHexChar(key_txt[2*i+1])) handleCryptoError();
    key[i] = hex2uchar(key_txt + 2*i);
  }

  for (i = 0; i < iv_len; i++) {
    if (!isValidHexChar(iv_txt[2*i]) || !isValidHexChar(iv_txt[2*i+1])) handleCryptoError();
    iv[i] = hex2uchar(iv_txt + 2*i);
  }

  if (!(ctx = EVP_CIPHER_CTX_new())) handleCryptoError();
  if (!(cipher = EVP_aes_256_gcm())) handleCryptoError();
  if (1 != EVP_EncryptInit_ex(ctx, cipher, NULL, NULL, NULL)) handleCryptoError();
  if (1 != EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, iv_len, NULL)) handleCryptoError();
  if (1 != EVP_EncryptInit_ex(ctx, NULL, NULL, key, iv)) handleCryptoError();

  do {
    size_t ret = fread(buf_plain, 1, iv_len, stdin);
    if (!ret) {
      if (ferror(stdin)) {
    perror("fread");
        freeCrypto();
        return 1;
      }
      if (feof(stdin)) break;
    }

    if (1 != EVP_EncryptUpdate(ctx, buf_cipher, &len, buf_plain, ret)) handleCryptoError();

    if (len && !fwrite(buf_cipher, len, 1, stdout)) {
      if (feof(stderr)) fprintf(stderr, "EOF on output stream\n");
      else perror("fwrite");
      freeCrypto();
      return 1;
    }

  } while (1);

  if (1 != EVP_EncryptFinal_ex(ctx, buf_cipher, &len)) handleCryptoError();

  if (len && !fwrite(buf_cipher, len, 1, stdout)) {
    if (feof(stderr)) fprintf(stderr, "EOF on output stream\n");
    else perror("fwrite");
    freeCrypto();
    return 1;
  }

  if (1 != EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, sizeof tag, tag)) handleCryptoError();
  if (!fwrite(tag, sizeof tag, 1, stdout)) {
    if (feof(stderr)) fprintf(stderr, "EOF on output stream\n");
    else perror("fwrite");
    freeCrypto();
    return 1;
  }

  fflush(stdout);
  freeCrypto();
  return 0;
}

And here is how to decrypt what the above program has encrypted:

(compile this way gcc -Wall -lcrypto -o aes256gcm-decrypt aes256gcm-decrypt.c)

// AES-256-GCM with libcrypto
// gcc -Wall -lcrypto -o aes256gcm-decrypt aes256gcm-decrypt.c

// tag is 16 bytes long
// no AAD (Additional Associated Data)
// input format: tag is read just after cipher text (see RFC-5116, sections 5.1 and 5.2)

// KEY=a6a7ee7abe681c9c4cede8e3366a9ded96b92668ea5e26a31a4b0856341ed224
// IV=87b7225d16ea2ae1f41d0b13fdce9bba
// cat ciphertext | ./aes256gcm-decrypt $KEY $IV

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <openssl/conf.h>
#include <openssl/evp.h>
#include <openssl/err.h>

EVP_CIPHER_CTX *ctx = NULL;
unsigned char *iv = NULL;
unsigned char *buf_plain = NULL;
unsigned char *buf_cipher = NULL;
unsigned char *input = NULL;

typedef enum { false, true } bool;

void freeCrypto() {
  if (input) free(input);

  if (ctx) {
    EVP_CIPHER_CTX_free(ctx);
    ctx = NULL;
  }
  CRYPTO_cleanup_all_ex_data();
  ERR_free_strings();

  if (iv) {
    free(iv);
    iv = NULL;
  }
  if (buf_plain) {
    free(buf_plain);
    buf_plain = NULL;
  }
  if (buf_cipher) {
    free(buf_cipher);
    buf_cipher = NULL;
  }
}

void handleCryptoError() {
  fprintf(stderr, "ERROR\n");
  ERR_print_errors_fp(stderr);
  freeCrypto();
  exit(1);
}

bool isValidHexChar(char c) {
  return (c >= 'a' && c <= 'f') || (c >= '0' && c <= '9');
}

unsigned char hex2uchar(char *hex) {
  unsigned char ret;

  if (hex[0] >= 'a' && hex[0] <= 'f') ret = (hex[0] - 'a' + 10) * 16;
  else ret = (hex[0] - '0') * 16;
  if (hex[1] >= 'a' && hex[1] <= 'f') ret += hex[1] - 'a' + 10;
  else ret += hex[1] - '0';
  return ret;
}

unsigned char *loadInput(int *plen) {
  int len = 0;
  unsigned char *buf = NULL;
  unsigned char *old_buf;

  do {
    int c = fgetc(stdin);
    if (c == EOF) break;
    if (c < 0) {
      perror("fgetc");
      exit(1);
    }
    len++;
    old_buf = buf;
    buf = malloc(len);
    if (buf < 0) {
      perror("malloc");
      exit(1);
    }
    if (len > 1) bcopy(old_buf, buf, len - 1);
    buf[len - 1] = c;
    if (old_buf) free(old_buf);
  } while (1);

  *plen = len;
  return buf;
}

int main(int ac, char **av, char **ae)
{
  const EVP_CIPHER *cipher;
  unsigned char key[32];
  int iv_len, len, i;
  unsigned char *current;
  int input_len;

  if (ac != 3) {
    fprintf(stderr, "usage: %s KEY IV\n", av[0]);
    return 1;
  }

  char *key_txt = av[1];
  char *iv_txt = av[2];

  input = loadInput(&input_len);
  current = input;

  ERR_load_crypto_strings();

  if (strlen(key_txt) != 2 * sizeof key) {
    fprintf(stderr, "invalid key size\n");
    freeCrypto();
    return 1;
  }

  if (strlen(iv_txt) < 2 || strlen(iv_txt) % 2) {
    fprintf(stderr, "invalid IV size\n");
    freeCrypto();
    return 1;
  }
  iv_len = strlen(iv_txt) / 2;

  if (!(iv = malloc(iv_len))) {
    perror("malloc");
    freeCrypto();
    return 1;
  }

  if (!(buf_plain = malloc(iv_len))) {
    perror("malloc");
    freeCrypto();
    return 1;
  }

  if (!(buf_cipher = malloc(iv_len))) {
    perror("malloc");
    freeCrypto();
    return 1;
  }

  for (i = 0; i < sizeof key; i++) {
    if (!isValidHexChar(key_txt[2*i]) || !isValidHexChar(key_txt[2*i+1])) handleCryptoError();
    key[i] = hex2uchar(key_txt + 2*i);
  }

  for (i = 0; i < iv_len; i++) {
    if (!isValidHexChar(iv_txt[2*i]) || !isValidHexChar(iv_txt[2*i+1])) handleCryptoError();
    iv[i] = hex2uchar(iv_txt + 2*i);
  }

  if (!(ctx = EVP_CIPHER_CTX_new())) handleCryptoError();
  if (!(cipher = EVP_aes_256_gcm())) handleCryptoError();
  if (1 != EVP_DecryptInit_ex(ctx, cipher, NULL, NULL, NULL)) handleCryptoError();
  if (1 != EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, iv_len, NULL)) handleCryptoError();
  if (1 != EVP_DecryptInit_ex(ctx, NULL, NULL, key, iv)) handleCryptoError();
  if (1 != EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, 16, input + input_len - 16)) handleCryptoError();

  do {
    int nbytes = input + input_len - 16 - current;
    if (nbytes > iv_len) nbytes = iv_len;
    if (!nbytes) break;

    bcopy(current, buf_plain, nbytes);
    current += nbytes;

    if (1 != EVP_DecryptUpdate(ctx, buf_cipher, &len, buf_plain, nbytes)) handleCryptoError();

    if (len && !fwrite(buf_cipher, len, 1, stdout)) {
      if (feof(stderr)) fprintf(stderr, "EOF on output stream\n");
      else perror("fwrite");
      freeCrypto();
      return 1;
    }

  } while (1);

  // correct tag is checked here
  if (EVP_DecryptFinal_ex(ctx, buf_cipher, &len) <= 0) handleCryptoError();

  if (len && !fwrite(buf_cipher, len, 1, stdout)) {
    if (feof(stderr)) fprintf(stderr, "EOF on output stream\n");
    else perror("fwrite");
    freeCrypto();
    return 1;
  }

  fflush(stdout);
  freeCrypto();
  return 0;
}