Run databricks jobs for code in merged state

[datascientistlin]

Hi Team,

As far as I understood, I could choose from branch, commit and tag if my task source is GIT. However, the use case I'm trying to implement is to run the databricks jobs during the PR merging state. This could help us perform integration testing, to ensure the code is running fine after merging the PR (before the actual merging happens).

We used Azure Devops platform for CICD and Repo. Here is how I have implemented it so far.

In pipelines.yml file:

  - job: On_PR
    displayName: Build on new PR
    condition: eq(variables['Build.Reason'], 'PullRequest')

      steps:
          # Add DEPLOY_COMMIT variable to be used in dbx environment
          - script: |
              echo "##vso[task.setvariable variable=DEPLOY_COMMIT;]$(Build.SourceVersion)"
            displayName: Set DEPLOY_COMMIT variable
          # Deploy databricks job using dbx
          - script: |
              source /usr/share/miniconda/etc/profile.d/conda.sh
              conda activate ${{ parameters.CONDA_ENV }}
              dbx deploy --job=${{ parameters.JOB_NAME }} --files-only --requirements-file requirements.txt
            displayName: Deploy Databricks Job
          # Launch databricks job using dbx
          - script: |
              source /usr/share/miniconda/etc/profile.d/conda.sh
              conda activate ${{ parameters.CONDA_ENV }}
              dbx launch --job=${{ parameters.JOB_NAME }} --as-run-submit --trace 
            displayName: Launch Databricks Job

In deployment.yml file

Environments:
  default:
    strict_path_adjustment_policy: true
    jobs:
      - name: "Training-Job"
        new_cluster:
          spark_version: "10.4.x-scala2.12"
          num_workers: 1
          node_type_id: "Standard_E16d_v4"
        notebook_task:
          notebook_path: "notebooks/training"
          source: "GIT"
        git_source: 
          git_url: "https://[email protected]/_git/xxx"
          git_provider: "azureDevOpsServices"
          git_commit: {{ env['DEPLOY_COMMIT'] }}

The problem with this implementation is DEPLOY_COMMIT does not work in the PR merging state. This is because the commit during PR merging is a synthesized one which could not be checked out in the actual pipeline.

Thanks in advance for any tips and advises.

[renardeinside]

hi @datascientistlin ,

This is because the commit during PR merging is a synthesized one which could not be checked out in the actual pipeline.

I'm not sure how this is relevant to dbx to be honest, it seems to me like an issue on Azure DevOps side.
I would assume that inside CI pipeline you can create a real commit (or temp branch) from a synthesized one and then reference it in the deployment.yml.

[renardeinside]

Also could you please provide any documentation about this "syntesized commit"? I haven't found anything in the Azure DevOps docs relevant to the topic.

[datascientistlin]

Hi @renardeinside , I couldn't find a complete explaination about the "synthesized commit", but I think this post from SO can be an example. You would see 2 commits shown in the pipeline. The 2nd one triggering individual CI is the actual commit pushed to the PR while the 1st one triggering PR automation is the "synthesized commit" where the pipeline drives a state of the code base as if the PR has been merged.

[renardeinside]

hi @datascientistlin ,
We also consider that this might be a race condition. Could you please check if you can launch a job from the automated commit some time after (even manually)?

[datascientistlin]

Hi @renardeinside , it would not be possible to run it manually becuase the merged commit (if we use commit ID as reference in job) would not exist after the PR job is completed. And if an actual merge is performed, the merged commit ID would be different from the one generated by PR job as well.

[renardeinside]

hi @datascientistlin ,

unfortunately I cannot help much with this issue since I cannot even reproduce it.

I would recommend asking Azure Architects on the topic of Azure DevOps and the visibility / behaviour of this "synthesized commit". I've never run into such behaviour while making branches in Azure DevOps, therefore cannot comment much on this.

[copdips]

@datascientistlin
If I understand well your issue, maybe you can use this var: $(System.PullRequest.SourceCommitId) instead of $(Build.SourceVersion)

[datascientistlin]

Thanks for the help! I may not be reading it correctly, but I couldn't find $(System.PullRequest.SourceCommitId) as a pre-defined varialbe in Azure pipeline. My reference is here: https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml

[copdips]

yes, it's not officially documented yet, but it should be available, make a try and tell us if it works

[datascientistlin]

Thanks @copdips , I just tested this out. I think $(System.PullRequest.SourceCommitId) still gives me the commit that I pushed to the PR (i.e. "source" commit). However, it doesn't point to the synthesized commit that the PR generates.
The way that I resolve this problem currently is to change from notebook task to python task. And I could use dbx to build the package at PR state and run the python task there. Does that sound reasonable to you?

[copdips]

@datascientistlin
I'm sorry that I've never deployed notebooks, all my exp were in deploying python tasks with dbx.
$(System.PullRequest.SourceCommitId) is the last real commit you've pushed to git, not the one generated by the pull request, unfortunately, it's not what you need.
IMHO, notebooks are only for local POC, or quick test, they're great for developers, but there're some caveats in terms of operations. If I need something in production, I always use python tasks.

[copdips]

could you please enable the debug mode of your azure pipeline, and past the logs of the git checkout part here ?
from my test, pure bash command git checkout [commit_id] out of dbx works for me, I would like to know what it does when used in git_source with the jobs api.