There is an entity with an identifier and some information. The identifier of this entity is a string with a known scheme/pattern(Which means the total number of such possible entities is say 10-100 million).
Information about the entities is stored in two geographically distributed systems. A third system is responsible for aggregating information about these entities.
And here is where the security comes into play - The third system should not know the actual identifiers of the entities and should not be able to derive the actual identifiers.
Existing proposed solution:
Both of the sites storing information about the entities decide upon a one way hashing algorithm to use, say 'PBKDF-2' and also securely communicate a long random salt that they will use while hashing. Both the sites do the one way hashing and send the hash as identifier to the third entity. The third entity can still merge the entities based on the hash but does not know the actual identifier.
Problem with the solution:
Since the set of all possible entity identifiers can be recreated(about 10-100 million), someone can execute a dictionary attack fairly easily. The salt makes it a bit more difficult but that's just some more additional overhead and does not provide a lot of comfort.
- Use multiple hashing cycles with different salts
- Use multiple hashing cycles with different salts and ciphers(scrypt, bcrypt and PBKDF-2)
- Use mappings of identifiers to random strings (but that adds the overhead of always communicating the mappings between the two systems)
Please let me know if any of the three approaches listed above is acceptable. If there is some other more known/acceptable solution to scenarios like above(which I imagine must be plenty out there), I would gladly appreciate and accept them.