4

I create a CSR using OpenSSL, so I have public and private keys. Now I want to sign this CSR and convert it into an X.509 certificate.

The only problem is, my signer is a custom HSM. I can send bytes and it signs them for me using SHA1. (The full algorithm is called: SHA1-RSA-PKCS.)

How can I manually extract the bytes that need to be signed, get them signed and then feed that signature to OpenSSL?

I had an idea of signing my CSR with a fake CSR, then replace signature bits. I tried some openssl stuff, but I wasn't able to do it (I learned mostly from this question: What part of the CSR is hashed in order to create its signature?) Any suggestions?

StackzOfZtuff
  • 17,783
  • 1
  • 50
  • 86
John McKean
  • 73
  • 1
  • 6
  • 1
    My guess is that you can't do it with the openssl command line application. You definitely can do it with libcrypto and possibly bindings for libcrypto. Why doesn't your HSM provide a nice API that accepts a CSR and returns a signed certificate? – Z.T. Oct 13 '15 at 20:29

1 Answers1

6

Creating a certificate is NOT signing the CSR, or even the CSR body. You need to create a cert body containing data that is partly the same as the CSR and partly different, and sign that.

First, do you have a CA cert (root or otherwise) matching the key in your HSM? If not, any cert(s) you do manage to create with that key won't be useful because software has no reason to trust it(them).

OpenSSL has an ENGINE API to allow HSMs or other "external" components to handle some crypto operations, including RSA. This API is intended to be simple enough that new engines can easily be added. If there is an engine module already compatible with your HSM or you can get one, you can just use the same openssl commandline operations openssl ca or openssl x509 -req -CA* with explicit -engine option or the equivalent in your config file. If your HSM supports a PKCS#11 interface, there is a generic but thirdparty pkcs11 engine for OpenSSL, see https://stackoverflow.com/questions/19456555/openssl-engine-pkcs11-libp11-opensc .

Or you can as @Z.T. comments write a program that calls libcrypto to do most of the same things the commandline apps do, except using your HSM to do the signature. I suggest looking at the reqfile case in apps/x509.c as a starting point; ca.c has so many more options it's a bit harder to extract what you actually need. You'll also need to replace X509_sign, which is really just ASN1_item_sign with X509_CINF as the encoded type, with an explicit encoding step i2d_x509_CINF(x->cert_info,...) followed by digest, encode/pad, and sign as per PKCS#1/rfc3447 (and rfc3279).

As a last resort you could:

  • create a fake issuer keypair with the same algorithm (RSA) and size, and fake issuer cert for that keypair but with the same names, and keyid if used, as the real CA cert; the keyid won't match the key as openssl sets automatically so you'll have to encode the explicit value
  • use openssl commandline as-is to process your CSR and issue a bastard cert under the fake issuer keypair and cert (add) with right hash, so it has all the correct data in the cert body (add) and algids but a wrong signature
  • split the DER encoded cert into the outer wrapper, "to-be-signed" body, algid, and wrong signature
  • have your HSM sign the encoded/padded digest of body with the real CA key giving a good signature
  • recombine the outer wrapper, body, and algid with the good signature

Expanded third option per comment. I realized x509 can copy an existing subjectkeyid without computing it on the actual (new) key, so that part is easier; note this feature requires the fairly-recent version 1.0.2 of OpenSSL.

Also on going through the steps I realized your device is signing with SHA-1. Because MD5 was broken several years ago for collision, and thus signature at least in a potentially adversarial situation like cert issuance, and the consensus of cryptography experts is that essentially the same techniques will work soon against SHA-1, cert signature using SHA-1 has been officially disapproved since 2013 by NIST, and prohibited by CA/Browser Forum for certs lasting beyond next year (2016) and some people are actively pushing to make it sooner. There are already dozens of questions here and some other SEs about this: "Why does Chrome warn my cert is about to be insecure?" "SHA-1 is obsolete, you need to get a new one." "How can I shut off all the warnings clogging my Firefox console about SHA-1 certs?" "You can't, that's a feature." Browsers may, with sufficient manual overrides, continue to accept SHA1 signatures from private in-house CAs like yours, where the collision risk is mitigated, but it may become more difficult, especially if the users of your system(s) are not crypto experts.

Those said, here's a worked example on Windows, which is the slightly harder case:

:: contents of conf.txt 

[req]
# to generate CSR (and also fake HSM)
distinguished_name = reqdn
[reqdn]
# dummy, actual values on commandline

[myca]
# to issue "bastard" cert which will be patched 
private_key = fakekey.pem
certificate = fakecert.pem 
default_days = 365 
database = index.txt 
serial = serial 
new_certs_dir = . 
policy = policy_anything 
x509_extensions = user 
[ policy_anything ]
countryName     = optional
stateOrProvinceName = optional
localityName        = optional
organizationName    = optional
organizationalUnitName  = optional
commonName      = supplied
emailAddress        = optional

[HSMCA]
# to fake HSM only 
basicConstraints=CA:true
keyUsage = keyCertSign
subjectKeyIdentifier = hash
#authorityKeyIdentifer not needed

[user]
# for issued certs 
keyUsage = digitalSignature, keyEncipherment
authorityKeyIdentifier = keyid 
# could be omitted or 'issuer' additionally or instead 
subjectAltName=DNS:example.com,DNS:*.example.com
# canned value for example, reality should be copy-from-req or dynamic 

:: simulate for example "real" (HSM) key and CA cert for it 

C:\temp>openssl req -newkey rsa:2048 -x509 -subj "/C=QQ/O=McKean/OU=HSMCA" -days 3650 ^
-config conf.txt -extensions HSMCA -nodes -keyout hsmkey.pem -out hsmcert.pem
Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
..........................+++
..........................+++
unable to write 'random state'
writing new private key to 'hsmkey.pem'

C:\temp>openssl x509 <hsmcert.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            c3:1e:19:f6:87:e0:8e:90
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=QQ, O=McKean, OU=HSMCA
        Validity
            Not Before: Oct 16 02:38:16 2015 GMT
            Not After : Oct 13 02:38:16 2025 GMT
        Subject: C=QQ, O=McKean, OU=HSMCA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ba:f1:da:4a:2b:24:8c:de:1b:cc:b8:e4:30:a5:
                    17:bc:d2:a9:93:8c:cc:ee:91:98:99:83:ee:81:d3:
                    80:e5:9c:ec:5d:97:c3:2f:8e:e6:db:52:e0:af:a1:
                    c6:f4:6e:78:90:32:3a:8c:09:a5:78:34:76:44:98:
                    04:83:03:41:59:f0:0b:7e:13:02:20:a0:70:cb:eb:
                    32:52:c8:62:ee:1c:97:a3:c9:64:14:0a:73:85:9d:
                    e3:16:90:96:3d:e1:e7:3a:39:a8:95:64:68:8f:e4:
                    eb:29:62:36:ff:e6:e7:89:d6:5a:8f:b0:03:cb:38:
                    e4:dc:6a:9d:5b:9c:5d:1a:4e:13:5c:22:5a:cf:e6:
                    92:a5:42:82:44:c7:07:46:f4:db:57:97:04:e6:b3:
                    7b:88:3b:14:ff:c7:dd:87:7a:e3:14:f6:84:ef:09:
                    9e:44:7b:e1:81:a9:3d:c6:a6:5f:77:ee:02:6e:74:
                    a6:b8:d9:30:88:8e:b4:48:b3:f0:ab:c9:87:5c:db:
                    5d:df:05:28:b1:34:b7:20:32:be:b7:a2:77:7d:bf:
                    7f:62:57:88:c3:c9:4c:4d:29:63:cc:a3:dc:b2:b3:
                    00:8c:69:17:f9:d1:6c:02:31:9d:ca:03:6f:c2:a0:
                    ef:61:14:66:07:57:e6:9c:44:7d:ed:22:e4:e9:f8:
                    7f:83
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:TRUE
            X509v3 Key Usage:
                Certificate Sign
            X509v3 Subject Key Identifier:
                F7:A0:68:EB:7F:28:BC:4C:80:03:CA:57:C7:F0:61:7B:D6:14:C3:EC
    Signature Algorithm: sha256WithRSAEncryption
         1c:78:16:42:f4:29:7c:1e:a9:fb:b6:7f:15:12:8e:d1:e6:c6:
         c3:11:dc:f2:fe:6b:3d:05:4b:0b:bc:cc:f8:29:ce:29:c1:a4:
         11:8d:c1:c1:83:b7:4c:6e:f6:51:86:3a:08:2b:e5:a2:2e:ca:
         23:09:d5:13:6d:c9:fe:b9:af:c4:ed:6f:75:d5:ef:a7:3e:4b:
         7d:09:5b:ad:0d:3b:85:c4:fb:99:dd:a3:b1:e6:f7:34:25:f9:
         c7:8e:37:fd:0d:20:85:cd:04:80:4b:b0:fc:46:4e:5b:a1:3a:
         4c:ce:a3:17:a0:f9:41:f1:64:2e:ac:96:5a:f1:72:5b:9d:57:
         0c:32:8e:cf:9b:ed:50:bf:88:28:c1:b2:bc:e1:87:3e:7b:ac:
         f1:f0:3c:c7:5c:25:0d:da:38:f0:33:e0:25:03:88:f8:97:2c:
         8c:a6:54:92:11:4d:36:9c:9f:3c:40:bb:3c:09:50:4b:f5:6e:
         85:10:bb:dc:d9:67:8f:40:03:d1:95:a5:85:05:c2:65:f5:52:
         18:a2:4d:59:7d:3f:5e:7a:f9:8c:e1:ca:ec:f8:05:4c:5c:74:
         b6:04:f0:18:27:c5:a5:c6:d4:55:56:f9:80:1a:01:78:ab:51:
         8c:c0:69:64:eb:09:ee:d8:f0:62:b9:f9:04:3c:d7:50:67:9e:
         23:a6:69:ea

:: create "fake" RSA key of same size, and fake CA cert for it 
:: with same subject, issuer/serial and keyid as the HSM cert
:: (validity period differs, but that doesn't matter)

C:\temp>openssl genrsa 2048 -nodes >fakekey.pem
Loading 'screen' into random state - done
Generating RSA private key, 2048 bit long modulus
.....................................+++
..........................................................................+++
unable to write 'random state'
e is 65537 (0x10001)

C:\temp>openssl rsa <fakekey.pem -noout -text
Private-Key: (2048 bit)
modulus:
    00:98:07:94:79:39:14:c5:c7:64:8b:51:d3:9b:e1:
[remainder snipped] 

C:\temp>openssl rsa <fakekey.pem -pubout >fakepub.pem
writing RSA key

C:\temp>openssl x509 <hsmcert.pem -force_pubkey fakepub.pem -signkey fakekey.pem >fakecert.pem
Loading 'screen' into random state - done
Getting Private key
unable to write 'random state'

C:\temp>openssl x509 <fakecert.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            c3:1e:19:f6:87:e0:8e:90
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=QQ, O=McKean, OU=HSMCA
        Validity
            Not Before: Oct 16 02:53:15 2015 GMT
            Not After : Nov 15 02:53:15 2015 GMT
        Subject: C=QQ, O=McKean, OU=HSMCA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:98:07:94:79:39:14:c5:c7:64:8b:51:d3:9b:e1:
                    1b:0c:e7:f8:10:15:c6:95:b2:7d:81:66:2b:0a:ac:
                    82:2d:d2:86:72:84:f2:8c:d3:09:8a:d6:01:f4:d5:
                    73:cc:4b:6a:8a:3f:4c:e0:78:41:98:74:26:ab:07:
                    db:06:66:eb:63:23:27:f6:fa:d8:cf:7f:93:c6:65:
                    72:61:3d:c8:f5:db:5d:45:ac:4f:4e:a9:91:9b:af:
                    5e:4b:d1:61:9a:6c:42:de:87:56:ec:11:d0:64:12:
                    23:24:6a:8b:4c:b9:b2:68:97:8e:43:2e:9b:ad:1c:
                    1e:f5:f9:fa:69:6a:99:d7:ab:db:3f:7b:a9:dc:44:
                    b4:49:4c:95:7a:9b:ad:a1:69:84:b5:bc:fb:66:be:
                    58:0a:df:2c:8a:01:a6:7c:38:a6:10:e7:48:ed:72:
                    37:09:ea:fc:bf:1b:9c:27:49:8b:43:32:01:9a:1d:
                    aa:00:ba:88:2e:d0:1b:14:93:9e:5b:33:f9:12:03:
                    08:9e:9f:22:06:be:c7:85:3f:39:ae:41:f8:63:0e:
                    92:08:55:9b:82:6f:a7:b1:d2:df:f6:3a:2f:97:88:
                    9f:be:17:df:40:f0:89:de:1a:94:dd:63:10:75:3c:
                    dc:a1:c4:15:ca:e9:3b:26:81:6e:7b:58:52:79:72:
                    09:47
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:TRUE
            X509v3 Key Usage:
                Certificate Sign
            X509v3 Subject Key Identifier:
                F7:A0:68:EB:7F:28:BC:4C:80:03:CA:57:C7:F0:61:7B:D6:14:C3:EC
    Signature Algorithm: sha256WithRSAEncryption
         97:5b:8b:4b:e4:e0:04:b0:b5:bd:68:09:a1:18:f2:19:68:76:
         ea:be:e8:01:42:04:0a:c1:4c:a9:4f:4f:54:31:46:6c:cd:d2:
         5a:4c:9d:af:9d:f3:99:8b:a5:ec:c3:38:7e:24:3a:6c:a0:4e:
         7b:96:c2:88:9f:31:fc:93:47:57:fa:7c:4e:79:40:e6:ed:89:
         b4:f5:42:17:d8:37:20:79:a6:2a:30:ce:f9:18:51:f0:a2:02:
         7b:cb:8b:b5:41:92:79:4f:ce:ec:99:e1:a5:82:24:ec:e5:3d:
         a1:ec:d1:6c:63:75:41:90:b6:af:15:17:f4:a1:8b:f7:8b:a5:
         09:99:f9:aa:57:f7:98:ae:bc:ef:39:aa:6c:96:66:66:8b:65:
         3a:b5:11:1f:d2:d6:b3:4d:95:0a:22:e7:3d:88:1f:ce:1c:3a:
         04:c3:4f:33:e3:53:27:1d:94:10:98:4b:f6:70:ed:48:f1:ac:
         ac:91:1a:5e:4a:37:9a:eb:58:39:69:89:d0:0e:1e:e5:7b:be:
         51:bc:54:ae:09:dd:e5:8f:1f:41:0e:11:b7:e0:87:94:ba:9e:
         d3:9b:69:04:02:fa:55:94:b0:43:bb:e9:15:ac:5c:a9:78:0d:
         ae:1c:34:b1:ca:a7:d0:d4:78:5b:ec:35:1a:07:95:68:48:58:
         fe:22:82:49

:: create user key and CSR for it, then issue bastard cert under fake key (with SHA1) 

C:\temp>openssl req -newkey rsa:2048 -subj "/C=QQ/O=McKean/OU=server/CN=example.com" ^
-config conf.txt -nodes -keyout userkey.pem -out userreq.pem
Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
................................................................................
.................................................................+++
...............+++
unable to write 'random state'
writing new private key to 'userkey.pem'
-----

C:\temp>echo 01 >serial

C:\temp>echo |set/p x= >index.txt

C:\temp>openssl ca -config conf.txt -name myca -in userreq.pem -md sha1 -notext -out bastard.pem
Using configuration from conf.txt
Loading 'screen' into random state - done
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'QQ'
organizationName      :ASN.1 12:'McKean'
organizationalUnitName:ASN.1 12:'server'
commonName            :ASN.1 12:'example.com'
Certificate is to be certified until Oct 15 03:52:16 2016 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
unable to write 'random state'

C:\temp>openssl x509 <bastard.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=QQ, O=McKean, OU=HSMCA
        Validity
            Not Before: Oct 16 03:52:16 2015 GMT
            Not After : Oct 15 03:52:16 2016 GMT
        Subject: C=QQ, O=McKean, OU=server, CN=example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ad:9b:ea:c9:f1:fa:80:b6:9f:b4:6e:47:aa:9b:
                    f7:32:cf:2b:b9:44:f2:41:f3:42:10:dc:0a:58:6f:
                    8a:6b:71:2b:5d:93:a3:74:72:20:95:1a:db:9e:9c:
                    ae:d0:af:b7:da:11:f7:26:af:36:13:54:8f:b9:45:
                    d8:dd:5c:1c:a7:c7:c8:8f:b9:4d:09:f5:30:c3:2b:
                    d7:2c:c1:fa:25:78:91:51:22:61:aa:eb:8c:f4:d3:
                    47:80:7e:33:c0:9c:91:14:7d:97:fa:4c:72:28:73:
                    b5:41:7a:cc:5c:4f:da:4f:4d:6c:72:1e:65:b4:47:
                    1a:31:ff:1f:b9:e2:55:c5:c4:2c:d0:28:c9:e1:0c:
                    60:87:2c:0a:0b:21:9c:23:ee:86:a1:57:06:62:68:
                    8b:46:3b:07:2a:08:5c:6f:42:74:5b:53:a6:3e:ba:
                    ed:16:4d:64:c5:17:4a:8b:45:7e:fb:23:e7:63:94:
                    01:b1:c0:45:0d:7d:45:d0:fe:2d:1e:b0:d6:14:25:
                    45:2a:f7:3a:28:14:fe:67:0b:9b:7e:67:90:a2:84:
                    90:f5:de:55:3a:94:ad:4f:fc:80:e4:60:6d:7a:7c:
                    06:7f:aa:69:e2:22:39:e3:13:ae:20:f8:14:f8:55:
                    bd:3a:e7:6b:23:a0:b1:54:9f:bf:af:56:cc:77:32:
                    7e:27
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Authority Key Identifier:
                keyid:F7:A0:68:EB:7F:28:BC:4C:80:03:CA:57:C7:F0:61:7B:D6:14:C3:EC

            X509v3 Subject Alternative Name:
                DNS:example.com, DNS:*.example.com
    Signature Algorithm: sha1WithRSAEncryption
         24:4c:01:bf:b3:73:8f:16:03:d5:fd:34:55:e8:7c:de:be:1f:
         99:92:88:b7:df:97:a3:a3:9e:2e:31:78:22:b8:ea:12:ff:3b:
         10:ff:d3:56:4f:5d:77:8c:21:d4:65:87:56:42:e5:91:e6:d7:
         68:b6:09:2e:22:b6:2e:d2:d7:6c:a4:3f:fa:98:57:f9:e3:61:
         fd:49:90:e7:9c:3d:42:36:19:68:8e:be:a3:77:3d:94:41:54:
         81:46:e4:bd:0b:5a:96:6e:de:28:00:36:39:a1:27:0a:84:2d:
         19:b2:3d:19:d7:97:c9:a7:3f:f2:0a:71:d8:21:4a:64:6b:25:
         85:c2:1e:f4:bf:e1:01:1c:96:64:6f:33:d6:8d:46:c2:85:3c:
         e7:fb:15:95:75:bb:cf:0c:a5:8f:6d:a3:a8:51:3b:fe:8d:cc:
         27:d1:b7:13:c8:90:60:6a:fa:ba:7e:47:53:51:f5:89:bc:f0:
         55:e7:bd:fc:29:59:0f:6e:99:e6:53:bc:87:9f:a1:23:6f:cb:
         8f:d3:58:53:a3:05:40:54:38:6e:df:34:83:86:34:9d:5d:21:
         9f:26:e9:53:e3:f1:fd:a9:8e:be:b8:64:48:d7:8d:80:a2:de:
         4a:86:fc:99:a0:08:b3:62:2b:ed:9d:38:05:ae:2a:88:75:71:
         cd:3c:89:c1

:: find parts of cert 

C:\temp>openssl asn1parse -i <bastard.pem
    0:d=0  hl=4 l= 837 cons: SEQUENCE
    4:d=1  hl=4 l= 557 cons:  SEQUENCE
    8:d=2  hl=2 l=   3 cons:   cont [ 0 ]
   10:d=3  hl=2 l=   1 prim:    INTEGER           :02
   13:d=2  hl=2 l=   1 prim:   INTEGER           :01
   16:d=2  hl=2 l=  13 cons:   SEQUENCE
   18:d=3  hl=2 l=   9 prim:    OBJECT            :sha1WithRSAEncryption
   29:d=3  hl=2 l=   0 prim:    NULL
   31:d=2  hl=2 l=  46 cons:   SEQUENCE
   33:d=3  hl=2 l=  11 cons:    SET
   35:d=4  hl=2 l=   9 cons:     SEQUENCE
   37:d=5  hl=2 l=   3 prim:      OBJECT            :countryName
   42:d=5  hl=2 l=   2 prim:      PRINTABLESTRING   :QQ
   46:d=3  hl=2 l=  15 cons:    SET
   48:d=4  hl=2 l=  13 cons:     SEQUENCE
   50:d=5  hl=2 l=   3 prim:      OBJECT            :organizationName
   55:d=5  hl=2 l=   6 prim:      UTF8STRING        :McKean
   63:d=3  hl=2 l=  14 cons:    SET
   65:d=4  hl=2 l=  12 cons:     SEQUENCE
   67:d=5  hl=2 l=   3 prim:      OBJECT            :organizationalUnitName
   72:d=5  hl=2 l=   5 prim:      UTF8STRING        :HSMCA
   79:d=2  hl=2 l=  30 cons:   SEQUENCE
   81:d=3  hl=2 l=  13 prim:    UTCTIME           :151016035216Z
   96:d=3  hl=2 l=  13 prim:    UTCTIME           :161015035216Z
  111:d=2  hl=2 l=  69 cons:   SEQUENCE
  113:d=3  hl=2 l=  11 cons:    SET
  115:d=4  hl=2 l=   9 cons:     SEQUENCE
  117:d=5  hl=2 l=   3 prim:      OBJECT            :countryName
  122:d=5  hl=2 l=   2 prim:      PRINTABLESTRING   :QQ
  126:d=3  hl=2 l=  15 cons:    SET
  128:d=4  hl=2 l=  13 cons:     SEQUENCE
  130:d=5  hl=2 l=   3 prim:      OBJECT            :organizationName
  135:d=5  hl=2 l=   6 prim:      UTF8STRING        :McKean
  143:d=3  hl=2 l=  15 cons:    SET
  145:d=4  hl=2 l=  13 cons:     SEQUENCE
  147:d=5  hl=2 l=   3 prim:      OBJECT            :organizationalUnitName
  152:d=5  hl=2 l=   6 prim:      UTF8STRING        :server
  160:d=3  hl=2 l=  20 cons:    SET
  162:d=4  hl=2 l=  18 cons:     SEQUENCE
  164:d=5  hl=2 l=   3 prim:      OBJECT            :commonName
  169:d=5  hl=2 l=  11 prim:      UTF8STRING        :example.com
  182:d=2  hl=4 l= 290 cons:   SEQUENCE
  186:d=3  hl=2 l=  13 cons:    SEQUENCE
  188:d=4  hl=2 l=   9 prim:     OBJECT            :rsaEncryption
  199:d=4  hl=2 l=   0 prim:     NULL
  201:d=3  hl=4 l= 271 prim:    BIT STRING
  476:d=2  hl=2 l=  87 cons:   cont [ 3 ]
  478:d=3  hl=2 l=  85 cons:    SEQUENCE
  480:d=4  hl=2 l=  11 cons:     SEQUENCE
  482:d=5  hl=2 l=   3 prim:      OBJECT            :X509v3 Key Usage
  487:d=5  hl=2 l=   4 prim:      OCTET STRING      [HEX DUMP]:030205A0
  493:d=4  hl=2 l=  31 cons:     SEQUENCE
  495:d=5  hl=2 l=   3 prim:      OBJECT            :X509v3 Authority Key Identfier
  500:d=5  hl=2 l=  24 prim:      OCTET STRING      [HEX DUMP]:30168014F7A068EBF28BC4C8003CA57C7F0617BD614C3EC
  526:d=4  hl=2 l=  37 cons:     SEQUENCE
  528:d=5  hl=2 l=   3 prim:      OBJECT            :X509v3 Subject AlternativeName
  533:d=5  hl=2 l=  30 prim:      OCTET STRING      [HEX DUMP]:301C820B6578616D06C652E636F6D820D2A2E6578616D706C652E636F6D
  565:d=1  hl=2 l=  13 cons:  SEQUENCE
  567:d=2  hl=2 l=   9 prim:   OBJECT            :sha1WithRSAEncryption
  578:d=2  hl=2 l=   0 prim:   NULL
  580:d=1  hl=4 l= 257 prim:  BIT STRING

:: For this cert: outer TL is 0 to 4, body is 4 to 565, algid is 565 to 580, 
:: BITSTRING TL+unused is 580 to 585, and actual signature is 585 to 841 

:: separate the pieces; on Unix could use a bunch of dd, 
:: but perl is easier and (nearly) the same on Windows 

C:\temp>openssl x509 <bastard.pem >bastard.der -outform der 

C:\temp>type bastard.pl
open F, '<bastard.der' or die $!; binmode F;
my $b; my $i=0; foreach my $n (@ARGV){
  ($n-=tell(F)) > 0 or die "out of order";
  open G, '>bastard.'.(++$i) or die $!; binmode G;
  read F,$b,$n; print G $b; close(G); }

C:\temp>perl bastard.pl 4 565 580 999

C:\temp>dir bastard.?
[skip]
2015-10-16  00:48                 4 bastard.1
2015-10-16  00:48               561 bastard.2
2015-10-16  00:48                20 bastard.3
2015-10-16  00:48               256 bastard.4
[skip]

:: simulate HSM signature on body with SHA1,RSASSA-PKCS1v1.5 
:: doing this on the real HSM remains your problem, 
:: and if you (need to) write a program 
:: it might easily handle the split and recombine too 

C:\temp>openssl dgst -sha1 <bastard.2 -sign hsmkey.pem >bastard.x 

C:\temp>dir bastard.x
[skip]
2015-10-16  00:52               256 bastard.x
[skip]

:: recombine substituting HSM signature (on Unix just cat) 

C:\temp>copy /b bastard.1+bastard.2+bastard.3+bastard.x fixed.der

:: convert back to PEM and check result

C:\temp>openssl x509 <fixed.der -inform der >fixed.pem

C:\temp>openssl verify -CAfile hsmcert.pem fixed.pem
fixed.pem: OK
dave_thompson_085
  • 9,759
  • 1
  • 24
  • 28
  • Thank you for detailed answer. I tried to do engine and pkcs11 interface etc. but I kept getting errors I just gave up. Errors like DSO support routines:WIN32_BIND_FUNC:could not bind to.... Invalid this, invalid that. I debugged, tried everything. So for the last part of your recommendation. Can you ellaborate a little bit more on that? Like how to specify keyid, how to split DER cert into "to-be-signed" body? Thanks again! – John McKean Oct 14 '15 at 13:44
  • @JohnMcKean see edit – dave_thompson_085 Oct 16 '15 at 05:34