Artifact location in project.json

[BramLangelaar1]

Hi there!

I am in contact with one of the Databricks Architects about whether it is possible to configure the artifact_location var in the project.json to an Azure DevOps Artifact Feed, so that the whl file is stored in the Feed, and not in Databricks’ local dbfs.

Has this been done before, and if so, how to configure it (also taken into account authentication), so that DBX will write to the DevOps Artifact Feed, and read and installed onto the cluster when the actual deployed job is launched.

Kind regards,

Bram

[renardeinside]

hi @BramLangelaar1,

artifact_location means a slightly different thing in this context. dbx uses MLflow as an artifact location by default to store assets for dbx deploy, launch and execute.

If you would like to store your package in Azure Artifacts and use it in your jobs, a slightly different approach is required:

1. Deploy your package to Azure Artifacts using Azure DevOps or twine
2. Specify your extra index url in /etc/pip.conf during cluster start (using an init script) as described here.
3. In your job deployment configuration file specify this library in the cluster dependencies section
4. Deploy the job with the following parameters:

dbx deploy --no-rebuild --no-package