Skip to content

Latest commit

 

History

History

adb-with-private-links-exfiltration-protection

README.md

Azure Databricks with Private Links (incl. web-auth PE) and Hub-Spoke Firewall structure (data exfiltration protection).

Include:

  1. Hub-Spoke networking with egress firewall to control all outbound traffic, e.g. to pypi.org.
  2. Private Link connection for backend traffic from data plane to control plane.
  3. Private Link connection from user client to webapp service.
  4. Private Link connection from data plane to dbfs storage.
  5. Private Endpoint for web-auth traffic.

Overall Architecture: alt text

With this deployment, traffic from user client to webapp (notebook UI), backend traffic from data plane to control plane will be through private endpoints. This terraform sample will create:

  • Resource group with random prefix
  • Tags, including Owner, which is taken from az account show --query user
  • VNet with public and private subnet and subnet to host private endpoints
  • Databricks workspace with private link to control plane, user to webapp and private link to dbfs

How to use

Note
You can customize this module by adding, deleting or updating the Azure resources to adapt the module to your requirements. A deployment example using this module can be found in examples/adb-with-private-links-exfiltration-protection

  1. Reference this module using one of the different module source types
  2. Add a variables.tf with the same content in variables.tf
  3. Add a terraform.tfvars file and provide values to each defined variable
  4. Add a output.tf file.
  5. (Optional) Configure your remote backend
  6. Run terraform init to initialize terraform and get provider ready.
  7. Run terraform apply to create the resources.

Requirements

Name Version
azurerm >=4.0.0

Providers

Name Version
azurerm >=4.0.0
external n/a
http n/a
random n/a

Modules

No modules.

Resources

Name Type
azurerm_databricks_workspace.this resource
azurerm_firewall.hubfw resource
azurerm_firewall_application_rule_collection.adbfqdn resource
azurerm_firewall_network_rule_collection.adbfnetwork resource
azurerm_network_interface.testvmnic resource
azurerm_network_interface_security_group_association.testvmnsgassoc resource
azurerm_network_security_group.testvm-nsg resource
azurerm_network_security_group.this resource
azurerm_network_security_rule.aad resource
azurerm_network_security_rule.azfrontdoor resource
azurerm_network_security_rule.test0 resource
azurerm_private_dns_zone.dnsdbfs_blob resource
azurerm_private_dns_zone.dnsdbfs_dfs resource
azurerm_private_dns_zone.dnsdpcp resource
azurerm_private_dns_zone_virtual_network_link.dbfsdnszonevnetlink_blob resource
azurerm_private_dns_zone_virtual_network_link.dbfsdnszonevnetlink_dfs resource
azurerm_private_dns_zone_virtual_network_link.dpcpdnszonevnetlink resource
azurerm_private_endpoint.auth resource
azurerm_private_endpoint.dbfspe_blob resource
azurerm_private_endpoint.dbfspe_dfs resource
azurerm_private_endpoint.dpcp resource
azurerm_public_ip.fwpublicip resource
azurerm_public_ip.testvmpublicip resource
azurerm_resource_group.this resource
azurerm_route_table.adbroute resource
azurerm_subnet.hubfw resource
azurerm_subnet.plsubnet resource
azurerm_subnet.private resource
azurerm_subnet.public resource
azurerm_subnet.testvmsubnet resource
azurerm_subnet_network_security_group_association.private resource
azurerm_subnet_network_security_group_association.public resource
azurerm_subnet_route_table_association.privateudr resource
azurerm_subnet_route_table_association.publicudr resource
azurerm_virtual_network.hubvnet resource
azurerm_virtual_network.this resource
azurerm_virtual_network_peering.hubvnet resource
azurerm_virtual_network_peering.spokevnet resource
azurerm_windows_virtual_machine.testvm resource
random_string.naming resource
azurerm_client_config.current data source
external_external.me data source
http_http.my_public_ip data source

Inputs

Name Description Type Default Required
dbfs_prefix Prefix for DBFS storage account name string "dbfs" no
firewallfqdn Additional list of fully qualified domain names to add to firewall rules list(any) n/a yes
hubcidr CIDR for Hub VNet string "10.178.0.0/20" no
metastoreip IP Address of built-in Hive Metastore in the target region string n/a yes
private_subnet_endpoints The list of Service endpoints to associate with the private subnet. list(string) [] no
rglocation Location of resource group to create string "southeastasia" no
spokecidr CIDR for Spoke VNet string "10.179.0.0/20" no
tags map of tags to add to all resources map(any) {} no
test_vm_password Password for Test VM string "TesTed567!!!" no
workspace_prefix Prefix to use for Workspace name string "adb" no

Outputs

Name Description
arm_client_id Depricated. Client ID for current user/service principal
arm_subscription_id Depricated. Azure Subscription ID for current user/service principal
arm_tenant_id Depricated. Azure Tenant ID for current user/service principal
azure_region Depricated. Geo location of created resources
azure_resource_group_id ID of the created Azure resource group
databricks_azure_workspace_resource_id Depricated. The ID of the Databricks Workspace in the Azure management plane.
my_ip_addr Depricated. IP address of caller
resource_group Name of created resource group
test_vm_public_ip Public IP of the created virtual machine
workspace_id The Databricks workspace ID
workspace_url The workspace URL which is of the format 'adb-{workspaceId}.{random}.azuredatabricks.net'