This project includes all necessary components to spin up the infrastructure for VM based GitHub self-hosted runners in Azure. This project was created with some inspiration from the Philips Lab AWS Solution with some opinionated changes on what our team at Liatrio has seen work well across different enterprises.
- Ephemeral Only
- Runners should only run one job to avoid interference from one workflow run to the next
- Warm Pool by Default
- Keeping idle runners on is a must to ensure quick feedback loops
- Custom Images
- Images should be able to build most apps in organization without additional tool installation (example)
- Including necessary tools in VM Image to reduce startup time for most builds
- Security
- Runner VMs are granted a single use registration token with no additional access to GitHub
- All components utilize Managed Identities for access to other resources, and are granted the least access required to function.
This Terraform module generates the infrastructure required to host the applications that will manage the self-hosted runners.
The event-handler will receive messages from the GitHub App during workflow run events. It will act as a filter to ensure they are from GitHub with labels that match what is provided in the module.
This application will act as the controller for the warm pool and ensure that the pool size adheres to the parameters specified in the Terraform module. It will consume events from the queue as necessary to create VMs and ensure a healthy number of VMs are always ready to process new workflow jobs.
- GitHub App for Organization (owner access)
- Azure
- Subscription
- Note: Subscription quota for "Total Regional Low-priority vCPUs" should be increased to allow multiple spot instances
- Resource Group
- Subnet with internet access
- KeyVault for GitHub App Credential
- Service Principal Roles/Permissions
Microsoft.AppConfiguration/locations/deletedConfigurationStores/read(Needed until closure of issue #19605 in hashicorp/terraform-provider-azurerm)
- Resource Provider Registration (See here for steps to register the resource provider)
Microsoft.AppConfiguration
- optional - Managed Image accessible by Runner-Controller
- Subscription
For convenience we have provided an image in our public Azure Community Gallery that can be used for quick setup, but you may want to build a custom image tailored to your use case. Referencing the Packer Template repo, create an image and publish it to Azure Compute Gallery that can be created by this Terraform module.
The GitHub App serves as the foundation for sending webhook events to App A and retrieving registration tokens to store in Azure Key Vault.
- Navigate: Settings → Developer Settings → GitHub Apps → New GitHub App
- Configure permissions
- Configure settings, webhook settings to be updated later
- Save App and take note of App ID, Client ID
| Permission | Access |
|---|---|
| Repository: Actions | Read-only |
| Repository: Checks | Read-only |
| Repository: Metadata | Read-only |
| Organization: Self-hosted runners | Read and write |
| Required Field | Value |
|---|---|
| GitHub App Name | {insert-name} |
| Homepage URL | {insert-any-url} |
| Webhook Active | False |
| Webhook URL | |
| Subscribe to events | Workflow job |
| Where can this GitHub App be installed? | Only on this account |
*Note: You will need one GitHub App per org. Allowing installation to "Any account" makes it difficult to change access if installed on orgs outside your control.
(optional) Set Key Vault name variable:
export KEYVAULT_NAME=<keyvault-name>Runner Password:
az keyvault secret set --name azure-runner-default-password --value $(uuidgen) --vault-name $KEYVAULT_NAMEGitHub Client Secret:
az keyvault secret set --name github-client-secret --vault-name $KEYVAULT_NAME --value <secret-value>GitHub Private Key:
az keyvault secret set --name github-private-key --encoding utf-8 --vault-name $KEYVAULT_NAME --file <location/pem>Webhook Secret:
az keyvault secret set --name github-webhook-secret --value $(uuidgen) --vault-name $KEYVAULT_NAME*Note: The private key must be added via the AZ CLI, all other secrets can be added manually via the portal if you choose to do so.
Consume this azure_github_runner module with inputs required for your GitHub Enterprise Cloud or GitHub Enterprise server configuration. Example of how to consume the module are coming soon.
Run terraform by using the following commands
terraform init
terraform applyThe terraform output displays the Azure Function endpoint and secret, which you need in the next step.
This terraform module is set up by default to use the latest version of both apps and deploy them on terraform apply. Specific versions found in our public GitHub Packages and set in the terraform module inputs. If you choose to publish your own images, functionality to do so will be implemented soon™.
Go back to the GitHub App and update the following settings
- Activate the webhook
- Provide the webhook url, which should be part of the terraform output
- Provide the webhook secret
- Save changes and navigate to the Install App tab
- Next to your GitHub App, select Install next to your org and select 'All Repositories'
Below are the required inputs required to get started with this module. Some may be marked with an asterisk (*) which indicates we recommend you pull this from a data source. Examples of usage are coming soon.
| Name | Description | Type |
|---|---|---|
| azure_tenant_id | Azure tenant ID | string |
| azure_subscription_id | Azure subscription ID | string |
| azure_resource_group_name | Resource Group that the components and runners will be created within | string |
| azure_subnet_id | Azure subnet ID | string |
| name_suffix | Identifying suffix that will be appended to all components created by this module (default: null) |
string |
| github_organization | GitHub organization | string |
| github_app_id | GitHub App ID | string |
| github_client_id | GitHub Client ID | string |
| github_installation_id | GitHub App installation ID | string |
| azure_secrets_key_vault_resource_id | Key Vault ID where GitHub secrets are stored | string |
| *azure_runner_default_password_key_vault_id | Key Vault ID for Azure runner default password (data source) | string |
| *github_client_secret_key_vault_id | Keyvault Vault ID for GitHub App client secret (data source) | string |
| *github_webhook_secret_key_vault_id | Keyvault Vault ID for GitHub App webhook secret (data source) | string |
| *github_private_key_key_vault_id | Keyvault Vault ID for GitHub App private key (data source) | string |
| *owners | The list of owners that will be assigned to all components (data source) | list(string) |
One goal of this module is to minimize the number of customizations needed in order to run autoscaling self-hosted runners. With this being said, this list of optional inputs will grow but hopefully not so much that it becomes difficult to manage and get started with this solution.
| Name | Description | Type | Default |
|---|---|---|---|
| log_level | Log level used across applications | string |
Information |
| azure_gallery_image_id | Azure Compute Gallery image ID to be used in runner creation, leave default to use latest Liatrio public image |
string |
/communityGalleries/liatrio-4e8ffc8d-5950-4137-b02c-df028384cdcd /images/ubuntu_gh_runner /versions/latest |
| azure_gallery_image_type | Azure Compute Gallery image type to be used in runner creation. Available options: 'community', 'direct-shared', 'rbac' | string |
community |
| event_handler_image_tag | Event-Handler image tag to use from GitHub Packages | string |
latest |
| runner_controller_image_tag | Runner-Controller image tag to use from GitHub Packages | string |
latest |
| github_runner_group | Runner Group to register runners to | string |
Default |
| tags | Map of tags that will be added to created resources | map(string) |
{} |
| azure_location | Azure location in which to create resources | string |
location of the resource group |