Pearson hashing

Pearson hashing is a hash function designed for fast execution on processors with 8-bit registers. Given an input consisting of any number of bytes, it produces as output a single byte that is strongly dependent on every byte of the input. Its implementation requires only a few instructions, plus a 256-byte lookup table containing a permutation of the values 0 through 255.[1]

This hash function is a CBC-MAC that uses an 8-bit substitution cipher implemented via the substitution table. An 8-bit cipher has negligible cryptographic security, so the Pearson hash function is not cryptographically strong, but it is useful for implementing hash tables or as a data integrity check code, for which purposes it offers these benefits:

• It is extremely simple.
• It executes quickly on resource-limited processors.
• There is no simple class of inputs for which collisions (identical outputs) are especially likely.
• Given a small, privileged set of inputs (e.g., reserved words for a compiler), the permutation table can be adjusted so that those inputs yield distinct hash values, producing what is called a perfect hash function.
• Two input strings differing by exactly one character never collide.[2] E.g., applying the algorithm on the strings ABC and AEC will never produce the same value.

One of its drawbacks when compared with other hashing algorithms designed for 8-bit processors is the suggested 256 byte lookup table, which can be prohibitively large for a small microcontroller with a program memory size on the order of hundreds of bytes. A workaround to this is to use a simple permutation function instead of a table stored in program memory. However, using a too simple function, such as T[i] = 255-i, partly defeats the usability as a hash function as anagrams will result in the same hash value; using a too complex function, on the other hand, will affect speed negatively. Using a function rather than a table also allows extending the block size. Such functions naturally have to be bijective, like their table variants.

The algorithm can be described by the following pseudocode, which computes the hash of message C using the permutation table T:

algorithm pearson hashing is
    h := 0

    for each c in C loop
        h := T[ h xor c ]
    end loop

    return h

The hash variable (h) may be initialized differently, e.g. to the length of the data (C) modulo 256; this particular choice is used in the Python implementation example below.

Example implementations

Python, 8-bit output

The 'table' parameter requires a pseudo-randomly shuffled list of range [0..255]. This may easily be generated by using python's builtin range function and using random.shuffle to permutate it:

 1 from random import shuffle
 2 
 3 example_table = list(range(0, 256))
 4 shuffle(example_table)
 5 
 6 def hash8(message: str, table) -> int:
 7     """Pearson hashing."""
 8     hash = len(message) % 256
 9     for i in message:
10         hash = table[hash ^ ord(i)]
11     return hash

C, 64-bit

 1 #include <stdint.h>
 2 static const unsigned char T[256] = {
 3     // TODO: Add 0-255 shuffled in any (random) order
 4 };
 5 
 6 uint64_t Pearson64(const unsigned char *x, size_t len, char *hex, size_t hexlen) {
 7   size_t i;
 8   size_t j;
 9   unsigned char h;
10   unsigned char hh[8];
11 
12   for (j = 0; j < 8; ++j) {
13     // Change the first byte
14     h = T[(x[0] + j) % 256];
15     for (i = 1; i < len; ++i) {
16       h = T[h ^ x[i]];
17     }
18     hh[j] = h;
19   }
20 
21   return (hh[0] << 56) | (hh[1] << 48) | (hh[2] << 40) | (hh[3] << 32) | 
22          (hh[4] << 24) | (hh[5] << 16) | (hh[6] << 8)  | (hh[7]);
23 }

The scheme used above is a very straightforward implementation of the algorithm, with a simple extension to generate a hash longer than 8 bits. That extension comprises the outer loop (i.e. all statement lines that include the variable j) and the array hh.

For a given string or chunk of data, Pearson's original algorithm produces only an 8-bit byte or integer, 0–255. However, the algorithm makes it extremely easy to generate a hash of whatever length is desired. As Pearson noted, a change to any bit in the string causes his algorithm to create a completely different hash (0-255). In the code above, following every completion of the inner loop, the first byte of the string is effectively incremented by one (without modifying the string itself).

Every time that simple change to the first byte of the data is made, a different Pearson hash, h, is generated. The C function builds a 16 hex character hash by concatenating a series of 8-bit Pearson hashes (collected in hh). Instead of producing a value from 0 to 255, this function generates a value from 0 to 18,446,744,073,709,551,615 (= 2⁶⁴ - 1).

This shows that Pearson's algorithm can be made to generate hashes of any desired length by concatenating a sequence of 8-bit hash values, each of which is computed simply by slightly modifying the string each time the hash function is computed. Thus the same core logic can be made to generate 32-bit or 128-bit hashes.

C#, 8-bit

 1 public class PearsonHashing
 2 {
 3 	public byte hash(string input)
 4 	{
 5 		byte toRet = 0;
 6 		byte[] bytes = Encoding.UTF8.GetBytes(input);
 7 
 8 		foreach (var b in bytes)
 9 		{
10 			toRet = T[(byte)(toRet ^ b)];
11 		}
12 
13 		return toRet;
14 	}
15 }

See also

• Non-cryptographic hash functions

References

1. Pearson, Peter K. (June 1990), "Fast Hashing of Variable-Length Text Strings" (PDF), Communications of the ACM, 33 (6): 677, doi:10.1145/78973.78978, archived from the original (PDF) on 2012-07-04, retrieved 2013-07-13
2. Lemire, Daniel (2012), "The universality of iterated hashing over variable-length strings", Discrete Applied Mathematics, 160 (4–5): 604–617, arXiv:1008.1715, doi:10.1016/j.dam.2011.11.009