How to split a PEM file

Question

Note : This is not really a question because I already found the answer but since I didn't find it easily here I will post it so that it can benefit others.

Question : How to read a concatenated PEM file as the one used by apache/mod_ssl directive SSLCACertificateFile ?

Answer (original) (source) :

cat $file|awk 'split_after==1{n++;split_after=0} /-----END CERTIFICATE-----/ {split_after=1} {print > "cert" n ".pem"}'

This can leave an empty file if there's a blank line at the end, such as with openssl pkcs7 -outform PEM -in my-chain-file -print_certs. To prevent that, check the length of the line before printing:

cat $file|awk 'split_after==1{n++;split_after=0}
   /-----END CERTIFICATE-----/ {split_after=1}
   {if(length($0) > 0) print > "cert" n ".pem"}'

Answer 29/03/2016 :

Following @slugchewer answer, csplit might be a clearer option with :

csplit -f cert- $file '/-----BEGIN CERTIFICATE-----/' '{*}'

This may be a dumb question, but why would I need to split my pem file? — Ashwani Agarwal, Mar 26 '16 at 08:06
@AshwaniAgarwal You want to split a PEM file when it contains several certificates and you wish to examine the certificates individually with tools such as `openssl` that take one certificate to analyze. — Law29, Mar 26 '16 at 09:13
Additionally, some tools or servers want a combined file with cert and key, while others want them separate. — captncraig, Jan 06 '17 at 17:55
I had to add '%-----BEGIN CERTIFICATE-----%' to the csplit command line to prevent an empty file. Seems to match what the man page specifies: csplit -f ./tmp/cert- $file '%-----BEGIN CERTIFICATE-----%' '/-----BEGIN CERTIFICATE-----/' '{*}' — Craig Hicks, Sep 20 '17 at 06:17
The question appears to assume that everyone uses the Unix/Linux operating systems. I doubt that this is the case. There is nothing wrong with giving information relevant to one OS, but the name of the OS ought to be stated for clarity. Other ways to specify algorithms would include programming languages available on almost all OSs, such as JavaScript, which is available locally, supported by almost all browsers (notable exception: Lynx). — David Spector, Oct 18 '19 at 11:56

score 36 · Answer 1 · edited Jun 22 '17 at 04:55

36

The awk snippet works for extracting the different parts, but you still need to know which section is the key / cert / chain. I needed to extract a specific section, and found this on the OpenSSL mailinglist: http://openssl.6102.n7.nabble.com/Convert-pem-to-crt-and-key-files-tp47681p47697.html

# Extract key
openssl pkey -in foo.pem -out foo-key.pem

# Extract all the certs
openssl crl2pkcs7 -nocrl -certfile foo.pem |
  openssl pkcs7 -print_certs -out foo-certs.pem

# Extract the textually first cert as DER
openssl x509 -in foo.pem -outform DER -out first-cert.der

edited Jun 22 '17 at 04:55

kubanczyk

13,502
5
40
55

answered Mar 20 '15 at 12:26

Johannes 'fish' Ziemke

1,348
12
11

1

nice command set :) I'll keep it for future use, but in my use case above, I'm working with a cert-only file containing 50+ CA certs ==> no pkey nor chain – Cerber Mar 20 '15 at 12:42
3

I think this is superior to awk solution, let the openssl do the parsing + you get the conversion. – Rusty Mar 24 '16 at 16:22
1

I'm sorry but only pkey command is correct. Second and third don't do what you advertise - they do something else. In some cases the result is good in some cases it can generate mysterious behaviors in consumers. Edited a bit. – kubanczyk Jun 22 '17 at 04:46
1

Any idea how to get the textually 3rd cert in this way? – flickerfly Sep 06 '19 at 13:36
or like this if you have a chain and want the cert: `openssl x509 -in chain.pem -outform pem -out cert.pem` – iRaS Dec 03 '20 at 07:52
@iRas Q: Assuming `chain.pem` contains multiple certificates in unknown order, which one the command will output? First? Last? Something else? – Jari Turkia Aug 11 '21 at 07:41

score 27 · Answer 2 · edited Jan 21 '21 at 22:58

27

The split command is available on most systems, and its invocation is likely easier to remember.

If you have a file collection.pem that you want to split into individual-* files, use:

split -p "-----BEGIN CERTIFICATE-----" collection.pem individual-

If you don't have split, you could try csplit:

csplit -s -z -f individual- collection.pem '/-----BEGIN CERTIFICATE-----/' '{*}'

-s skips printing output of file sizes

-z does not create empty files

edited Jan 21 '21 at 22:58

MDMoore313

5,531
6
34
73

answered Mar 24 '16 at 18:03

squidpickles

751
1
8
12

6

Sorry, none of my systems (busybox, fedora, centos) show a `-p` option (nor [the manpages I read](http://linux.die.net/man/1/split)) on split. Maybe you're using a special binary / package – Cerber Mar 25 '16 at 10:25
2

@Cerber Could try `csplit` instead... (see edit above) – squidpickles Mar 26 '16 at 05:33
1

works fine with `csplit` ! – Cerber Mar 29 '16 at 08:52
1

On FreeBSD I get from csplit: `csplit: *}: bad repetition count` (but split seems to work) – Gwyneth Llewelyn Aug 10 '17 at 22:11
This one worked for me, although the first file was size 0. – MDMoore313 Jan 21 '21 at 22:53

score 19 · Accepted Answer · edited May 23 '17 at 12:41

19

This was previously answered on StackOverflow :

awk '
  split_after == 1 {n++;split_after=0}
  /-----END CERTIFICATE-----/ {split_after=1}
  {print > "cert" n ".pem"}' < $file

Edit 29/03/2016 : See @slugchewer answer

edited May 23 '17 at 12:41

Community

1

answered May 22 '12 at 11:30

Cerber

1,101
1
10
23

Works on Linux only, fails on FreeBSD. – Michael-O Jan 07 '15 at 22:07
4

Inspired by this I've created an awk script that splits out certs and keys into separate files: https://gist.github.com/jinnko/d6867ce326e8b6e88975 – JinnKo Mar 23 '15 at 18:12

score 9 · Answer 4 · edited Apr 13 '17 at 12:14

9

If you want to get a single certificate out of a multi-certificate PEM bundle, try:

$ openssl crl2pkcs7 -nocrl -certfile INPUT.PEM | \
    openssl pkcs7 -print_certs | \
    awk '/subject.*CN=host.domain.com/,/END CERTIFICATE/'

The first two openssl commands will process a PEM file and and spit it back out with pre-pended "subject:" and "issuer:" lines before each cert. If your PEM is already formatted this way, all you need is the final awk command.
The awk command will spit out the individual PEM matching the CN (common name) string.

source1 , source2

edited Apr 13 '17 at 12:14

Community

1

answered May 10 '16 at 20:48

cmcginty

1,263
15
24

I don't see this in your source. Beside, PEM are Base64 encoded you won't find text like "subject", "CN", ... with awk – Cerber May 12 '16 at 09:29
1

Yes, this doesn't work for every type of PEM. If you extract a P7B to PEM using openssl, it will have a subject line listed before each certificate. Or you can modify to any string you segment your PEM file with. – cmcginty May 12 '16 at 09:54
Updated answer to handle when PEM does not contain "subject" – cmcginty May 13 '16 at 01:22

score 5 · Answer 5 · answered Mar 12 '19 at 22:44

If you are handling full chain certificates (i.e. the ones generated by letsencrypt / certbot etc), which are a concatenation of the certificate and the certificate authority chain, you can use bash string manipulation.

For example:

# content of /path/to/fullchain.pem
-----BEGIN CERTIFICATE-----
some long base64 string containing
the certificate
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
another base64 string
containing the first certificate
in the authority chain
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
another base64 string
containing the second certificate
in the authority chain
(there might be more...)
-----END CERTIFICATE-----

To extract the certificate and the certificate authority chain into variables:

# load the certificate into a variable
FULLCHAIN=$(</path/to/fullchain.pem)
CERTIFICATE="${FULLCHAIN%%-----END CERTIFICATE-----*}-----END CERTIFICATE-----"
CHAIN=$(echo -e "${FULLCHAIN#*-----END CERTIFICATE-----}" | sed '/./,$!d')

Explanation:

Instead of using awk or openssl (which are powerful tools but not always available, i.e. in Docker Alpine images), you can use bash string manipulation.

"${FULLCHAIN%%-----END CERTIFICATE-----*}-----END CERTIFICATE-----": from end of the content of FULLCHAIN, return the longest substring match, then concat -----END CERTIFICATE----- as it get stripped away. The * matches all the characters after -----END CERTIFICATE-----.

$(echo -e "${FULLCHAIN#*-----END CERTIFICATE-----}" | sed '/./,$!d'): from the beginning of the content of FULLCHAIN, return the shortest substring match, then strip leading new lines. Likewise, the * matches all the characters before -----END CERTIFICATE-----.

For a quick reference (while you can find more about string manipulation in bash here):

${VAR#substring}= the shortest substring from the beginning of the content of VAR

${VAR%substring}= the shortest substring from the end of the content of VAR

${VAR##substring}= the longest substring from the beginning of the content of VAR

${VAR%%substring}= the longest substring from the end of the content of VAR

For the less bash savvy, when you echo these variables out surround the variable with quotes to preserve the line breaks the way you are used to seeing them. I remember when that wasn't so obvious to me. Fabio, sweet use of bash string manipulation! — flickerfly, Sep 06 '19 at 13:46

score 3 · Answer 6 · answered May 03 '13 at 00:46

3

Also worth noting that PEM files are just a collection of keys/certificates inside BEGIN/END blocks, so it's pretty easy to just cut/paste if it's just a single file with one or two interesting entities...

answered May 03 '13 at 00:46

mgalgs

345
2
9

score 0 · Answer 7 · answered Feb 21 '22 at 05:31

0

You can use awk as follows:

awk '/-----BEGIN CERTIFICATE-----/ {f=1} /-----END CERTIFICATE-----/ {print; f=0} f' cert_and_key.pem

awk '/-----BEGIN PRIVATE KEY-----/ {f=1} /-----END PRIVATE KEY-----/ {print; f=0} f' cert_and_key.pem

The command awk '/hello/ {f=1} /world/ {print; f=0} f' some_file will print all lines in some_file that between the lines hello and world, inclusive.

answered Feb 21 '22 at 05:31

Conway

1

1

As already shown in several other answers, `awk '/matchbegin/,/matchend/'` is an easier way to print such a block. But your method only separates (all) certs from privatekey(s), which was not the question; this question wants to separate multiple certs from each other and your answer doesn't help at all for that. – dave_thompson_085 Feb 21 '22 at 07:06

Ashish K Srivastava · Answer 8 · 2019-09-13T16:44:47.270

Hmmm...almost same way I prepared the solution ( as suggested y @Cerber ) without realizing that this situation seems many people have. My solution follow almost same logic but use some more basic commands:

My all certs are in file: certin.pem

c=0
while read line
  do
    if echo $line | grep END; then
    echo $line >> certout$c.pem
    c=`expr $c + 1`
    else
     echo $line
     echo $line >> certout$c.pem
    fi
done < /tmp/certin.pem

This basically keep writing in a file until encounters "END" and then start writing to another file in incremented way. This way you will have "N" number of output files (certout0.pem, certout1.pem and so on..) files depending on how many certificates are there in your input pem file ( certin.pem ).

score -1 · Answer 9 · answered Sep 14 '22 at 14:16

full.pem can be easily splited into cert.pem and chain.pem using grep and tail:

grep -B 1000 -m 1 -F -e "-----END CERTIFICATE-----" full.pem > cert.pem
tail -n +2 full.pem | grep -A 1000 -m 1 -F -e "-----BEGIN CERTIFICATE-----" > chain.pem

For cert.pem: search the first string -----END CERTIFICATE----- and take all before -B 1000 then output in cert.pem.

For chain.pem: skip the first line with tail -n +2 then search the first string -----BEGIN CERTIFICATE----- and take all after -A 1000 then output in chain.pem.

Sorry Nicolas, this is not about "full chain file" but more about a stack of unrelated PEM like the one required by Apache's SSLCaCertificateFile — Cerber, Sep 15 '22 at 14:44

How to split a PEM file

9 Answers9

Linked