This repo demonstrates how to deliver a Modern Data Warehouse using Azure and Terraform.
- An Azure Subscription
- An Azure DevOps Organisation
- Visual Studio Community Edition with SQL Server Data Tools (SSDT)
git
,az
, andterraform
installed in your local development environment.
The above diagram shows a high level architecture of the services that are defined in the Terraform scripts, and the CI/CD process that you can implement using the yaml scripts in the .ado
folder. Terraform is included for a development and test environment, but this could easily be replicated to build QA/Prod environments too.
- Import this repository into a new Azure DevOps repo in your organisation. Instructions for doing this are available in the docs.
- Open your terminal and clone the repo from your Azure DevOps repo to your local development environment using
git clone
. - Run
az login
to authenticate to Azure, and select the subscription you want to deploy these resources into withaz account set
cd
into themdw-Azure-terraform/iac/shared
directory- Open the
main.tf
file and update the shared storage account name (line 27) - Deploy the shared resources for the terraform state by running
terraform init
to initialize your terraform environment,terraform plan
to see what will be deployed, andterraform apply
to deploy the shared resources. - Open the
main.tf
file in../dev
and../test
, and update theAzure_rm
backendstorage_account_name
on line 10. - Update the
vsts_configuration
block indev/main.tf
to point to your new Azure Repo. This is the repository reference that will be attached to your Data Factory instance. - Deploy the development and test environments by running
tf init
,tf plan
,tf apply
in thedev
andtest
folders. - The ADF artefacts will need to be updated with details of the resources you have just deployed. In order to do this, in your local environment:
- Rename the
adf/factory/adf-mdw-dev-westeurope-01.json
and update thevalue
with the name of the ADF that was deployed in your dev environment - Also update the URLs in the
adf/linkedService/*.json
files to point to the services in your development environment - Commit and push these changes to your Azure DevOps repository.
- Rename the
Contains sample yaml pipelines for use in Azure DevOps for CI/CD of ADF and Synapse artefacts
Contains Terraform deployment for the environments.
Contains SSDT project that mananges and maintains Synapse data model.
Contains sample Azure Data Factory artefacts.
- Variables for the CI/CD pipelines currently need to be manually added to the pipeline instances on import.
- These templates do not implement best practice wrt network security. This is beyond the scope of this example.
- There is no automated deployment or update of the infrastructure.