Can I use private repository as dependency?

[filiphagan]

Hi,
I couldn't find anything in docs on that topic. I'm trying to add a private repository as a dependency in databricks job (private PyPI server, private GitHub repo etc). I started with adding dependency_links to the setuptools setup() in setup.py but it doesn't seem to work.

[dbx][2022-08-18 10:18:40.736] [Run Id: 5608] Current run status info - result state: FAILED, lifecycle state: INTERNAL_ERROR, 
state message: Library installation failed for library due to user error for pypi {
  package: "my-secret-lib==0.1"
}
. Error messages:
Library installation attempted on the driver node of cluster 0000000000 and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(/databricks/python/bin/pip, install, my-secret-lib==0.1, --disable-pip-version-check) exited with code 1. ERROR: Could not find a version that satisfies the requirement my-secret-lib==0.1

Is it possible to add private pypi links and a safe way to pass access tokens using dbx?

[renardeinside]

hi @filiphagan ,
yes, you can. You'll need to setup it in the following way:

1. on the cluster level, configure a file called /etc/pip.conf. You can configure this file via an init script. Take a look at this example (it's Azure Artifacts specific, but you could get a grasp of the idea).
2. In your setup.py, simply reference the library by it's name, no dependency_links required.

[filiphagan]

hi @renardeinside,
Thanks for the response. Actually, I found better solution for that
To avoid adding extra file /etc/pip.conf we can add private pypi links in environmental variable PIP_EXTRA_INDEX_URL inside the deployment.json:

{
    "dev": {
        "jobs": [
            {
                "name": "hello-world-job",
                "new_cluster": {
                    "spark_version": "11.0.x-cpu-ml-scala2.12",
                    "node_type_id": "Standard_F4",
                    "num_workers": 1,
                    "spark_env_vars": {
                        "JF_USER": "{{secrets/test-scope-fh/JFROG_USER}}",
                        "JF_TOKEN": "{{secrets/test-scope-fh/JFROG_TOKEN}}",
                        "PIP_EXTRA_INDEX_URL": "https://${JF_USER}:${JF_TOKEN}@xyz.jfrog.io/artifactory/api/pypi/my-space/"
                    }
                },
                "spark_python_task": {
                    "python_file": "file://src/test_hello_world_jfrog/jobs/hello_world_job/entrypoint.py",
                    "parameters": [
                        "--times",
                        "3"
                    ]
                }
            }
        ]
    }
}

This way our user would only add secret JFrog credentials via Databricks CLI and run dbx deploy ... dbx launch ...

By the way. You've probably noticed that I'm doing

"JF_USER": "{{secrets/../JFROG_USER}}",
"JF_TOKEN": "{{secrets/../JFROG_TOKEN}}",
"PIP_EXTRA_INDEX_URL": "https://${JF_USER}:${JF_TOKEN}@xyz.jfrog.io..."

Instead of

"PIP_EXTRA_INDEX_URL": "https://{{secrets/../JFROG_USER}}:{{secrets/../JFROG_TOKEN}}@xyz.jfrog.io..."

For some reason the latter doesn't work. Or I miss something here

Have a nice day

[renardeinside]

this sounds nice to me as well. I'll add this method to the guiding document, thanks a lot!