Dbx installing libraries for a long time

[datascientistlin]

Hi,

I'm using dbx to launch my python job to databricks. However, the launch step always stuck at the "Installing libraries" state. Then after some time, the launch would time out.

Is there any way I could trouble shoot why it takes so long for dbx to install libraries?

Thanks in advance for any tips and ideas.

Here are the commands I had executed:

dbx deploy --job=E2E-test -e test --files-only
dbx launch --job=E2E-test -e test --as-run-submit --trace

The list of packages in reuirements.txt

black~=22.0
flake8>=3.7.9, <4.0
ipython>=7.31.1, <8.0
isort~=5.8.0
kedro[pandas.CSVDataSet, pandas.ExcelDataSet, pandas.ParquetDataSet, spark.SparkDataSet, plotly.PlotlyDataSet, plotly.JSONDataSet]==0.18.2
kedro-telemetry~=0.2.0
kedro-viz~=4.6.0
pytest-cov~=3.0
pytest-mock>=1.7.1, <2.0
pytest~=6.2
tqdm>=4.0, <5.0.0
xgboost>=1.6.0
lightgbm~=3.3.0
scikit-learn>=1.0.0
shap~=0.40
matplotlib~=3.5
numpy~=1.22
pandas~=1.4
seaborn~=0.11.0
PyPDF2>=1.26
markdown==3.3.6
nbformat==5.1.3
jupyter~=1.0.0
pyspark==3.1.3
boltons~=20.0
great_expectations==0.15.12
openpyxl==3.0.10
mlflow>=1.27
dbx~=0.6
sklearn

[renardeinside]

hi @datascientistlin ,
I would recommend specifying the dependencies in your setup.py instead of using requirements.txt.
Built-in pip installation from setup.py should run faster.

Also, please check that you're not running this on an all-purpose cluster - it's not possible to install one version over another when its a job which is running on all-purpose cluster.

[di-lin-mckinsey]

Hi @renardeinside , thanks for the advise. Here is what I have in setup.py for getting the package requirements.

with open("requirements.txt", encoding="utf-8") as f:
    requires = []
    for line in f:
        req = line.split("#", 1)[0].strip()
        if req and not req.startswith("--"):
            requires.append(req)

Then I would specify install_requires=requires as part of the setup argument.

May I also clarify with you on your second point? Do you mean if the all-purpose cluster already has some packages installed, this would conflict with the installation process from setup.py?

Thank you.

[renardeinside]

hi @di-lin-mckinsey ,

on the first point - if you use the requirements file from the setup.py, please disable the dbx default behaviour which parses requirements.txt separately by passing an argument:

dbx deploy --requirements-file=None # disables the default library parsing behaviour

On the point of all-purpose cluster library reinstallation:

1. Imagine you have a plain All purpose cluster, no cluster libraries installed
2. If you now run a job on this all purpose cluster, this job has a reference to install your package as a cluster library.
3. After installing this package as a cluster library, you won't be able to install a package with the same name again (even if versions are different). To do this you'll need to delete the existing library in the UI and restart the cluster.

To sum up - don't use all purpose clusters for jobs. Use job clusters for jobs. If the startup time is an issue, use instance pools.

Please read more on this behaviour here.

[datascientistlin]

Thanks @renardeinside . This is extremely helpful. On that note of cluster library, does setting --no-package flag during dbx deploy help with this situation?

If I were to interpre the job cluster correctly, I should specify new_cluster instead of existing_cluster_id in the job config in deployment.yml. Is that correct?

Thank you.

[renardeinside]

Thanks @renardeinside . This is extremely helpful. On that note of cluster library, does setting --no-package flag during dbx deploy help with this situation?

By disabling this property you will omit adding (and installation) the package code to the cluster, but it essentially means that you'll use the latest cluster library available, and if you do some updates locally they won't be applied.

If I were to interpre the job cluster correctly, I should specify new_cluster instead of existing_cluster_id in the job config in deployment.yml. Is that correct?

Exactly. Please take a look at the examples of new cluster configuration in the dbx docs. we fully support any payload which is valid for Jobs API.