Convert pre-commit GitHub action to Azure Pipelines - Pipeline Creation & Testing

[ram-kishore-odi]

We currently drive CI through GitHub actions. We need to convert that to Azure Pipelines for the Azure DevOps flavored MDSA project repo. This is the second ticket related to pre-commit Github action conversion to Azure pipeline. This ticket takes care of adding a sample pipeline and testing this functionality.

The first ticket related to this is https://github.com/orgs/cagov/projects/31/views/39?sliceBy%5Bvalue%5D=ram-kishore-odi&pane=issue&itemId=84385198&issue=cagov%7Ccaldata-dse-internal-tracker%7C20. which takes of the conversion portion

[summer-mothwood]

Ran into this while researching dbt Cloud updates -- dbt Cloud has new merge jobs that are meant to remove the necessity for some Github actions. This couldn't take the place of all of our Github Actions as it's pretty limited, but might be worth a look:

Merge jobs for implementing continuous deployment (CD) workflows are now GA in dbt Cloud. Previously, you had to either set up a custom GitHub action or manually build the changes every time a pull request is merged.

[ian-r-rose]

Yeah, it may be that we can remove the pre-commit action all-together now that dbt cloud CI jobs can run sqlfluff.

We do other checks in the pre-commit job:

• linting/formatting YAML
• linting/formatting Python
• Validating JSON
• Checking for stray merge conflict markers
• Checking for AWS credentials
• Checking for common typos

That said, sqlfluff is the most important for a dbt-focused project, and the simplification gained for clients who don't want to maintain a YAML CI pipeline might be worth the trade-off

[ian-r-rose]

I would still like to have the option of an Azure pipeline CI job for clients who don't want to adopt dbt Cloud, but being able to run sqlfluff in dbt Cloud would take some of the time pressure off

[ram-kishore-odi]

Kevin has reached out to CDT to see if they have any guidance in response the request that was sent to him - Update from 12/11/24