[FEATURE] Databricks Secrets support

[riccamini]

Is your feature request related to a problem? Please describe.

I want to be able to retrieve secrets stored in an existing scope.

Describe the solution you'd like

I would like to create an implementation for the Secrets class using the databricks SDK

Describe alternatives you've considered

The alternative considered is to use directly the Databricks SDK

Additional context

...

[riccamini]

There are two ways to retrieve the secrets:

• dbutils
• databricks API

For dbutils, you are forced to run on a notebook, which is not always the case, and it is already supported in Brickflow. Using the API requires retrieving the token first, and that has to be stored somewhere else.

Recommendation is to use Brickflow to retrieve the token or plain dbutils if you are on a notebook environment without using brickflow.

Note: brickflow always run inside a notebook, hence it has access to dbutils.

For the above reasons we do not see a use case for dbutils support in Koheesio

[mikita-sakalouski]

I have a code for this