Is it possible to create columns of varying distribution when using Pyfunctext?

[tscan89]

Hello! A big thank you for creating this, it is awesome! We are currently using this in conjunction with Mimesis. For some context, Mimesis is another 3rd party data generation library like faker. Is it possible to use the data generated Mimesis/Faker to create 1 to N relationships?

For example, if I wanted to generate a single column with the below:

spec.withColumn(colName="test_column", text=PyfuncText(lambda c, v: mimesis.address.postal_code()))

I can reference that column using the baseColumn attribute, but I'm not sure how to iterate through the values generated by that column and use each one 5 times, or something similar. What I'm looking to do is create related data via primary and foreign keys with something other than integers and be able to specify that I need some X distribution of foreign keys. I'm not sure if I'm making sense, but would appreciate any pointers/assistance!

[tscan89]

After looking at this more, I believe what I need is the ability to use a baseColumn as a value for seeding. This would let me create a column with a certain distribution of integers and then I can use each row value in that column to seed the text generators. There is currently no mechanism to do this from the documentation that I've read? The column value would have to be accessible via the lambda context (or some other way).

[ronanstokes-db]

Thanks for the the feedback.

When generating data using the native data generation capabilities, you can control the distribution with either the use of the distribution option or for discrete categorical values, you may use the weights option.

However when calling out to an external library such as faker, we enable the calls to be distributed but dont have any control over the faker random number generation, so we cannot control distribution of data or repeatability.

Any suggestions regarding this are welcome

The following doc illustrates how the text generation is integrated with Faker - https://databrickslabs.github.io/dbldatagen/public_docs/extending_text_generation.html

If you look at the code that performs the integration - you will see that the base value is passed to the plugin mechanism:

https://github.com/databrickslabs/dbldatagen/blob/master/dbldatagen/text_generator_plugins.py

[ronanstokes-db]

Also i was looking to add integrations for Mimemis - but if you have already implemented this and are interested in contributing it as a PR,feel free to submit it.

[tscan89]

It is possible to control text generation using 3rd party libraries. Mimesis/faker both have the ability to seed the generators, but I'm not sure how I would pass a value from another column to them. baseColumn seems to have no effect when using 3rd party libraries. What I would want to do is pass the current rowNumber or value from the id field as a seed to the generator so that the data generated is always deterministic.

[ronanstokes-db]

I believe we can seed the Faker instance at an instance level but not at an individual row level - so there doesnt seem to be a way to make this deterministic on a row by row basis (other than by reseeding the instance on every row - which may have a substantial perf impact)

We should be able to add fixes to at least make the instances seeded and if there is not a substantial perf impact, to make the row by row generation seeded

[tscan89]

Yes, you're absolutely right, you would need to reseed the generators on a row-by-row basis and it does have a performance impact, but the benefit is that you can generate data pulled from random data sources deterministically. I've built an internal application that takes advantage of dbldatagen with other custom code to accomplish just this. To generate 1 billion records across 256 partitions went from 10-15 minutes to about an hour. This was with 10 columns all using different mimesis types as a backend. Not terrible, and we're ok with it.