[CT-1812] [Feature] Adding extra Spark config for Thrift connection type

[vinhnemo]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Context

Currently, Thrift connection type does not support setting more spark properties. To assist with providing these settings for the connection, I would like to add a variable to the profile declaration.

Detail

• Implemented by add spark_conf_string to thrift connection type #591

A variable to the profile declaration

• Name: spark_conf_string
• Data type: String
• Example

your_profile_name:
  target: dev
  outputs:
    dev:
      type: spark
      method: thrift
      schema: [database/schema name]
      host: [hostname]
      
      # optional
      port: [port]              # default 10001
      user: [user]
      auth: [e.g. KERBEROS]
      kerberos_service_name: [e.g. hive]
      use_ssl: [true|false]
      spark_conf_string: 'spark.executor.memory=1g;spark.executor.cores=1'

• This variable is used String data type to make it easier to combine with environment variables.

spark_conf_string: "{{ env_var('SPARK_CONFIG_STRING') }}"

Describe alternatives you've considered

No response

Who will this benefit?

DBT users using dbt-spark with complex needs to adjust Spark Properties

Are you interested in contributing this feature?

I made it and want to push it up for review.

Anything else?

No response

[Fleid]

Duplicate of #387
Not closing until we decide which PR to move forward

[Fleid]

Hey @vinhnemo, could you please check this discussion, I'm trying to regroup all the threads on that topic in one place.
Please let me know if the plan works for you, before we can decide to move forward with this specific issue.

[JCZuurmond]

@Fleid : @Fokko and I prefer to close this one over the #387 one, as we prefer the parameters in a dictionary over the string.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.