Choosing a Hashing Function

There are lots of hashing functions to choose from, all with different proprties.

We could have gone for one of the simple cryptographic hashes like MD5. The fact that it is no longer considered secure is not an issue as we are not using it for security. However we decided that our best option was probably to go for a simpler algorithm optimised for speed.

We benchmarked all the algorithms available without installing additional packages to double check our expectations. You can find the code in the "Experimental" directory of the repository.

Algorithm: <built-in function crc32> - 9.462833404541015e-05
Algorithm: <built-in function adler32> - 4.3964385986328126e-05
Algorithm: sha512-256 - 0.00020053386688232422
Algorithm:       mdc2 - 0.006865143775939941
Algorithm:     sha224 - 0.00030040740966796875
Algorithm:    blake2b - 0.0001709461212158203
Algorithm:   sha3-256 - 0.00035643577575683594
Algorithm:   sha3-224 - 0.0003325939178466797
Algorithm:   sha3_512 - 0.000725245475769043
Algorithm: blake2s256 - 0.000252223014831543
Algorithm:        md5 - 0.00018529891967773439
Algorithm:   sha3_384 - 0.0005069494247436524
Algorithm:    blake2s - 0.0002993345260620117
Algorithm:   sha3-512 - 0.0010445833206176758
Algorithm:  shake_256 - 0.0013045310974121095
Algorithm:  whirlpool - 0.0009180068969726563
Algorithm:   shake128 - 0.0003023624420166016
Algorithm:   shake256 - 0.00042994022369384765
Algorithm:  shake_128 - 0.0003332376480102539
Algorithm:   sha3_224 - 0.000599813461303711
Algorithm:     sha256 - 0.00030012130737304686
Algorithm: sha512-224 - 0.0002460002899169922
Algorithm:   md5-sha1 - 0.00041191577911376954
Algorithm:   sha3_256 - 0.0006462812423706054
Algorithm:   sha3-384 - 0.0007459640502929688
Algorithm:        sm3 - 0.0006976842880249024
Algorithm:  ripemd160 - 0.0004863262176513672
Algorithm:       sha1 - 0.00013475418090820314
Algorithm:     sha384 - 0.0002084970474243164
Algorithm: blake2b512 - 0.00018634796142578126
Algorithm:     sha512 - 0.0002222776412963867
Algorithm:        md4 - 0.00012958049774169922

This showed what we expected. The speed optimised hashes are up to 2 orders of magnitude faster. We went with Adler32 in the end. It has a slight advantage in speed.