Hash transformer is too slow

[viniciuschiele]

I'm currently using RandomUuid for most of the columns but I was asked to hash the original values to maintain the same masked value.

I've replaced RandomUuid with Hash and what used to take less than a minute to dump/transform the data now takes 30 min.

This is what looks like the transformation config for 6 tables

    - schema: core
      name: users
      transformers:
        - name: Hash
          params:
            column: email
        - name: Hash
          params:
            column: first_name
        - name: Hash
          params:
            column: last_name

[wwoytenko]

We will review the implementation. But I suspect it is due to the hash function. I think we should provide hash function choice and hash function params. We will try to resolve it in the next release.

For now, I can suggest a temporal solution. You can use a simple shell script to implement any hashing function. For instance

#!/bin/bash

while read line
do
   printf "%s" "$line" | md5sum | awk '{print $1}'
done

And the config can be like:

    - schema: "humanresources"
      name: "employee"
      transformers:
        - name: "Cmd"
          params:
            driver:
              name: "text"
            expected_exit_code: -1
            skip_on_null_input: true
            executable: "/var/lib/playground/test.sh"
            columns:
              - name: "jobtitle"

The result

Read about Cmd transformer

[viniciuschiele]

I see that the current hash implementation is using an encryption algorithm, maybe a hash algorithm would be faster.

I'm going to push back this requirement for now until there is a built-in solution for it, thanks

[wwoytenko]

Agreed. We will try to deliver it soon, but any contribution is appreciated. Thank you!

[viniciuschiele]

Another finding about the Hash transformer, it seems to generate duplicate values even for a small set of values.

I have a table users with 1000 records and the email column is unique (UNIQUE INDEX), when I use the Hash I get an error during pg_restore: ERROR: could not create unique index \"users_email_ux\""

[wwoytenko]

Yeah, a collision was caused. I will rewrite the implementation with the possibility of choosing a hash function (md5, sha1, SHA224/256/384/512). The expected release date is 14 February. Thank you so much for reporting.

[viniciuschiele]

Never mind, it was my fault, the UNIQUE INDEX has a FILTER condition allowing duplicate emails for deleted users.
Sorry for the false alarm.

[wwoytenko]

FIxed in v0.1.5

[viniciuschiele]

I gave it a try, the new hash transformer is crazy fast now, thank you