Doesn't work with Databricks Serverless Instances.

[danielp-db]

Expected Behavior

DBLDATAGEN should be able to run with Databricks Serverless Instances as spark.sql.execution.arrow.pyspark.enabled is set by default.

Current Behavior

Module fails as it checks for the value of the configuration.

Steps to Reproduce (for bugs)

Connect a notebook to a Databricks Serverless Instance.
Import dbldatagen
Try creating a spec with DataGenerator.

Context

N/A

Your Environment

Databricks Serverless Instance

• dbldatagen version used: 0.4.0
• Databricks Runtime version: Serverless
• Cloud environment used: Azure

[ronanstokes-db]

Thanks Daniel - this is a current open issue and will be in hotfix or next version

[ronanstokes-db]

See #292

[ronanstokes-db]

Fixed with most recent release