Comparing allegedly identical OpenPGP keys published through different channels

Question

tldr version: When I try to compare public keys that are supposed to be for the same entity the versions I get through different channels are NOT the same and the ones that I download as a file like somekey.sig.asc are smaller files.

---

I'm trying to learn the whole gpg thing and practice using it correctly. A crucial step is verifying that a public key does in fact belong to the party it is supposed to. In the absence of a chain of personal contacts, the obvious thing is to get multiple copies from different sources and verify that they are identical. I never can seem to do that. I suspect that part of the problem lies in newlines, carriagereturns, spaces, tabs, and other characters invented by the SCOPTDUI (Secret Conspiracy Of Programmers To Drive Users Insane). Here is an example where I'm fairly sure the keys ARE identical, but I can't come up with a procedure to prove it.

Veracrypt team key from the MIT keyserver: https://pgp.mit.edu/pks/lookup?op=get&search=0xEB559C7C54DDD393

Veracrypt team key from the Veracrypt site: https://www.idrix.fr/VeraCrypt/VeraCrypt_PGP_public_key.asc

I've downloaded both directly. I've copied and pasted both into text files. I've stripped away the preambles and the endings, leaving blocks that each begin with the same gibberish and end with the same gibberish.

I've run them through all sorts of tr filters, like:

cat file | tr -d '\040\011\012\015' > file-tred

The one from MIT is still a lot bigger file. I've loaded them into gedit and I can copy substantial chunks of one and search for that chunk in the other and find an identical substring, but not with the whole file.

This isn't an isolated example. MIT isn't unique. The second keyserver I checked (in New Zealand, but I'm not allowed to post a third link here) has the same issue. Nor is Veracrypt a special case. I've run into this every time I've resolved to learn to do gpg the "right way" and beat my head against the same wall for a few days before giving up. Is there some reason these things aren't published in a standardised form? Is there some way to make this easy? Am I missing something obvious? What makes that MIT file so big?

Addendum: As a practical matter:

gpg --keyserver address.like-this-with-no-protocol-prefix-and-no-trailing-slash.net --search SOMETHING

seems to be checking duplicates of keys that I already have and reporting "no change", provided I choose SOMETHING well. So, functionally, I guess this is ok. I'd still like to know why my manual comparisons fail.

Answer 1

A PGP keyfile is not a single key string, but contains several entries (packets). Instead of trying to compare the ASCII representation of both files, you should use appropriate tools such as gpg(1) and compare the fingerprints.

Is there some way to make this easy?

Yes, like this:

$ wget https://www.idrix.fr/VeraCrypt/VeraCrypt_PGP_public_key.asc
$ gpg --with-fingerprint VeraCrypt_PGP_public_key.asc
pub  rsa4096/54DDD393 2014-06-27
      Key fingerprint = 993B 7D7E 8E41 3809 828F  0F29 EB55 9C7C 54DD D393
uid                   VeraCrypt Team <veracrypt@idrix.fr>

The displayed fingerprint should be equal for both files. Note, how the lower few bits of the hash represent the key ID (EB559C7C54DDD393).

Is there some reason these things aren't published in a standardised form?

They are, just check out RFC 4880 for the OpenPGP Message Format specification.

What makes that MIT file so big?

It contains a lot more signature packets. Again, from the RFC:

An OpenPGP message is constructed from a number of records that are traditionally called packets. An OpenPGP message, keyring, certificate, and so forth consists of a number of packets.

You can list all packets in a file via --list-packets (or use pgpdump(1) for a more readable version).

$ gpg --list-packets VeraCrypt_PGP_public_key.asc

This will yield an enumeration like this:

# off=0 ctb=99 tag=6 hlen=3 plen=525
:public key packet:
    version 4, algo 1, created 1403892630, expires 0
    pkey[0]: [4096 bits]
    pkey[1]: [17 bits]
    keyid: EB559C7C54DDD393
# off=528 ctb=b4 tag=13 hlen=2 plen=35
:user ID packet: "VeraCrypt Team <veracrypt@idrix.fr>"
# off=1137 ctb=89 tag=2 hlen=3 plen=540
:signature packet: algo 1, keyid D6BE7DAF738161CE
    version 4, created 1403950017, md5len 0, sigclass 0x10
    digest algo 2, begin of digest 08 92
    hashed subpkt 2 len 4 (sig created 2014-06-28)
    subpkt 16 len 8 (issuer key ID D6BE7DAF738161CE)
...

As you can see, the public key packet has the familiar key ID EB559C7C54DDD393, while the signature packets come with their respective issuer's key ID (which you can also look up at the keyserver).

Side note: What you call "preamble" is called ASCII armor and used to transfer PGP messages over channels that have difficulties with arbitrary binary data (e.g. e-mails). It's part of the format and you can leave it in the file.

Answer 2

tl;dr: both dumps contain the same keys, but you get some additional certifications from the key server not included in the minimal export of the VeraCrypt website.

OpenPGP Packets

OpenPGP keys are composed by a set of OpenPGP packets, which can be listed by gpg --list-packets and pgpdump. For exported keys, there's one required packet, the public key packet. Sneak peak (explanation following below): this is the same for both keys.

:public key packet:
  version 4, algo 1, created 1403892630, expires 0
  pkey[0]: [4096 bits]
  pkey[1]: [17 bits]
  keyid: EB559C7C54DDD393

A key is hardly usable without any user ID packets. It cannot be queried by a name, and it cannot even receive certifications (signatures on keys point to tuples of public key fingerprint and user ID).

:user ID packet: "VeraCrypt Team <veracrypt@idrix.fr>"

Finally, a self-signature is usually issued. Key servers often deny loading keys without self-signatures.

:signature packet: algo 1, keyid EB559C7C54DDD393
  version 4, created 1403892630, md5len 0, sigclass 0x13
  digest algo 2, begin of digest 7f 33
  hashed subpkt 2 len 4 (sig created 2014-06-27)
  hashed subpkt 27 len 1 (key flags: 0F) 
  hashed subpkt 11 len 6 (pref-sym-algos: 9 8 7 3 2 1)
  hashed subpkt 21 len 5 (pref-hash-algos: 8 2 9 10 11) 
  hashed subpkt 22 len 3 (pref-zip-algos: 2 3 1)
  hashed subpkt 30 len 1 (features: 01) 
  hashed subpkt 23 len 1 (key server preferences: 80) 
  subpkt 16 len 8 (issuer key ID EB559C7C54DDD393)
  data: [4096 bits]

Minimal Exports

This is the more or less required minimal part of keys for the key to be usable. For each of the user IDs, optionally a list of incoming certifications could be appended, to help others at verifying the key. This is the most likely reason of different outputs.

Comparing the Keys

Now, how to compare the keys? First of all, lets fetch both of them into local files mit.asc and veracrypt.asc. diffing those would be the general way of comparing files, but OpenPGP files are hardly readable, so let's first list their contents and diff those:

gpg --list-packets mit.asc >mit.asc.list
gpg --list-packets veracrypt.asc >veracrypt.asc.list
diff mit.asc.list veracrypt.asc.list

The diff reveals lots of additional signatures in the key server dump. Looking at veracraypt.asc.list, it indeed seems that veracraypt.asc is a minimal version, so lets make user of GnuPG's export options to create a minimal copy of mit.asc:

gpg --dearmor mit.asc # Dearmor first, to use it as keyring file
gpg --export-options export-minimal --no-default-keyring --keyring ./mit.asc.gpg \
    --export 0xEB559C7C54DDD393 >mit-minimal.asc
gpg --list-packets mit-minimal.asc >mit-minimal.asc.list

Diffing now reveals that veracrypt.asc holds exactly one additional signature:

:signature packet: algo 1, keyid D6BE7DAF738161CE
  version 4, created 1403950017, md5len 0, sigclass 0x10
  digest algo 2, begin of digest 08 92
  hashed subpkt 2 len 4 (sig created 2014-06-28)
  subpkt 16 len 8 (issuer key ID D6BE7DAF738161CE)
  data: [4091 bits]

A signature at least claiming to belong to the author of VeraCrypt (I didn't validate the key, though).