merge init_scripts instead of raising error, or let merge to be an option

[copdips]

With current version of dbx, when user specifies in its deployement.json, both policy_name and init_scripts, and if in the specifid policy, there're already some init scripts, both must be the same, this should be currently by design as dbx performs a deep dict update.

Our use case is that in the cluster policy, we only declare some common init scripts, for e.g. setup-pip.conf.sh, etc.
And we would like to enable users to add their own specific init scripts in addition to the common one. for e.g. some jobs need to install a sql driver with the script install-sql-driver.sh.

If we add install-sql-driver.sh in the cluster policy, all the jobs using the cluster policy will get the sql driver installed even they don't need it. Although for the moment, it doesn't raise any issue, we would like to keep the jobs environment as clean as possible, for those who don't need this sql driver, we don't want to install it. That's why I opened this discussion to see if we can merge the init_scripts part instead of update.

Concretly, a part of deployement.json could be:

"job_clusters": [
  {
    "new_cluster": {
      "policy_name": "pip.conf",
      "instance_pool_name": "foo",
      "init_scripts": [
        {
          "dbfs": {
            "destination": "dbfs:/bar/install-sql-driver.sh"
          }
        }
      ]
    }
  }

Our cluster policy pip.conf contains an init script setup-pip.conf.sh, and this job need to append a second init script install-sql-driver.sh.

Without this feature, we must declare all the scripts in the cluster policy, it's a paint point if we have many teams/users' scripts and it requires admin rights to manage the cluster policies.

If changing the current behavior raising error to merge is a breaking change, is it possible to add an option to enable the merge only for init_scripts key?

[copdips]

@renardeinside I've a local patch which seems worked, if you agree with this idea, I can make a PR.

[renardeinside]

hi @copdips ,
please wait for this PR to be merged, it brings a lot of refactoring to policy as well.

[renardeinside]

#522

[copdips]

sure, just would like to know the ETA of #522

[renardeinside]

in 2-3 days should be merged, doing my best on this :)

[copdips]

good luck:)