Deploying Infrastructure via Azure DevOps

[corticalstack]

Describe the bug or the issue that you are facing

Error when following the deploy with terraform guide, outer loop step 8
https://github.com/Azure/mlops-v2/blob/main/documentation/deployguides/deployguide_ado.md

When running the pipeline to create the ML infra, I get the error:
Repository mlops-templates references endpoint github-connection which does not exist or is not authorized for use

I have seen a similar issue (now closed) which indicated to install the Terraform for ADO extension. I have this installed.

Steps/Code to Reproduce

As described earlier

Expected Output

Pipeline runs successfully, provisioning the required AML resources

Versions

As described earlier

Which platform are you using for deploying your infrastrucutre?

Azure DevOps (ADO)

If you mentioned Others, please mention which platformm are you using?

No response

What are you using for deploying your infrastrucutre?

Terraform

Are you using Azure ML CLI v2 or Azure ML Python SDK v2

Azure ML Python SDK v2

Describe the example that you are trying to run?

As described earlier

[corticalstack]

I thinnk the problem is that the step-by-step guide for MLOps V2 with ADO and Terraform misses the service connection creation to create github-connection, as documented in quickstart.md for mlopsv2 repo:

3.3.5 Select "Github", select "Next", select "Personal Access Token" and paste your Github SSO Token in the Personal Access token field, in the "Service connection name" field, enter "github-connection", grant pipeline security access, then select "Save".

  Repeat this step, and for the "Service connection name" enter YOUR GITHUB ORGANIZATION NAME or YOUR GITHUB NAME. Finishing this step, your conection should look like this.

[setuc]

@corticalstack What is the error that you are getting. The comment you made on Issue #72 was related to the terraform installer that wasn't installed.

The github-connection is not required as your repositories are in ADO. In case you still want to store the repository in GitHub and run the pipelines in Azure DevOps (ADO), then you can setup a GitHub connection. Here are the steps that you can follow this.

1. Go to your Azure DevOps project and navigate to the Project Settings page.
2. Select Service connections under the "Pipelines" section.
3. Click on the New service connection button and select "GitHub" (since the type of repository in your code is GitHub).
4. You will be prompted to enter the connection details, including the GitHub repository URL, a personal access token with the necessary permissions, and a connection name.
5. After entering the connection details, click on the Verify and Save button to create the connection.
6. Once the connection is created, you can refer to it by the name you gave it in your code, in this case github-connection.

You can now use this endpoint in your pipelines to connect to the specified GitHub repository. That's it! Once you have set up the endpoint, you can reference it in your Azure DevOps pipeline configurations, like the code you provided earlier.

[corticalstack]

@setuc Thanks for the detailed response, which is in fact what I did to get it working and move forward. However, as you say, I'm using ADO for repo management and running the pipelines, so the question would be, "why is github-connection in the yml config for the ADO-only path, and how should an ADO-pure yml look for the resources>repositories part of the yml?"

Perhaps for the ADO-only, no endpoint key-value pair is needed under repositories in the yml?

[tonyskidmore]

I also encountered this issue when performing an ADO only deployment. The problem is with https://github.com/Azure/mlops-project-template/blob/main/infrastructure/terraform/devops-pipelines/tf-ado-deploy-infra.yml, but also other files too, for example:

https://github.com/Azure/mlops-project-template/blob/main/classical/aml-cli-v2/mlops/devops-pipelines/deploy-batch-endpoint-pipeline.yml

https://github.com/Azure/mlops-project-template/blob/main/classical/aml-cli-v2/mlops/devops-pipelines/deploy-model-training-pipeline.yml

https://github.com/Azure/mlops-project-template/blob/main/classical/aml-cli-v2/mlops/devops-pipelines/deploy-online-endpoint-pipeline.yml

which all contain the hardcoded endpoint and type keys for a github connection that should not exist in an ADO only setup:

resources:
  repositories:
    - repository: mlops-templates  # Template Repo
      name: Azure/mlops-templates # need to change org name from "Azure" to your own org
      endpoint: github-connection # need to set up and hardcode
      type: github
      ref: main

This can be fixed in at least a couple of ways such as inline syntax similar to this reference or maybe a better approach from a resource consistency point of view by the following:

resources:
  repositories:
    - repository: mlops-templates
      name: mlops-templates
      type: git
      ref: main

I have only been testing on an ADO basis so I don't know whether the endpoint: github-connection is required for other scenarios (I presume so). I think if it is required, then the YAML files should be either patched based on the deployment scenario, i.e. ADO or GitHub or multiple copies of the file maintained (probably less desirable).

[corticalstack]

@tonyskidmore thanks for the follow-up and info, appreciated. Frustrating to see little movement on all these open issues

[bgawale]

This still continues to be an issue. thanks @tonyskidmore for the solution.