feature/google cloud provider

* feat: gcp implementation * feat: gcp implementation * docs: update documentation * feat: Enhance gcloud component checks, improve code documentation, and implement GCP storage security hardening - Fixed comments for clarity and accuracy. - Added logic to check the installation of additional gcloud components. - Introduced typing annotations for better code clarity and robustness. - Generated and updated docstrings for all methods using reST style. - Implemented logic to enforce strict access control policies on GCP storage buckets. * feat: Improved working with GCP Identities for IAM Policies --------- Co-authored-by: Alexander Ulyanov <[email protected]>
CloudGeometry · Aug 20, 2024 · a4820ec · a4820ec
1 parent 0d6362a
commit a4820ec
Show file tree

Hide file tree

Showing 45 changed files with 2,683 additions and 323 deletions.
diff --git a/QUICKSTART.md b/QUICKSTART.md
@@ -23,7 +23,7 @@ deploying the application in production, the following platforms are currently s
 
 - AWS - Supported
 - Azure - Supported
-- GCP - Will be added to a future release
+- GCP - Supported
 
 ## Prerequisites
 
@@ -93,45 +93,66 @@ Before deploying to AWS, ensure that you have:
 
 1. An AWS account with billing enabled. (Remember, deploying clusters will incur charges. Make sure to destroy
    resources when you're finished with them!)
-3. A public hosted zone with DNS routing.
+2. A public hosted zone with DNS routing.
    To set this up,
    you can follow [this guide](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/AboutHZWorkingWith.html).
-4. A user account with `AdministratorAccess`. We recommend that rather than using your root account, you set up a
+3. A user account with `AdministratorAccess`. We recommend that rather than using your root account, you set up a
    new IAM user, then grant it AdministratorAccess. You can
    use [this guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started.html)
    to set up an IAM account,
    and [this guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html) to grant it
    `AdministratorAccess`.
-5. The security credentials for this account, which enables CGDevX to use it.
+4. The security credentials for this account, which enables CGDevX to use it.
    Use [this guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html#access-keys-and-secret-access-keys)
    to
    get your access keys.
-6. The AWS CLI installed and configured to use this user.
+5. The AWS CLI installed and configured to use this user.
    You can use [this guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to install
    the CLI.
 
 #### Azure deployment
 
 Before deploying to Azure, ensure that you have:
 
-1. An Azure account with billing enabled.
+1. An Azure account with billing enabled. Subscription ID will be used
+   as cloud-profile during setup. 
    (Remember, deploying clusters will incur charges. Make sure to destroy
    resources when you're finished with them!)
-3. A public DNS zone hosted in Azure DNS.
+2. A public DNS zone hosted in Azure DNS.
    To set this up,
    you can follow [this guide](https://learn.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns).
-4. A user account with `Owner` access.
+3. A user account with `Owner` access.
    You can
    use [this guide](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal-subscription-admin)
    to set it up,
    or [this guide](https://learn.microsoft.com/en-us/azure/role-based-access-control/quickstart-assign-role-user-portal)
    to grant permissions to an existing user.
-5. The Azure CLI (**az**) and **[kubelogin](https://aka.ms/aks/kubelogin)** installed and configured to use this user.
+4. The Azure CLI (**az**) and **[kubelogin](https://aka.ms/aks/kubelogin)** installed and configured to use this user.
    You can
    use [this](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli)
    and [this](https://azure.github.io/kubelogin/install.html) guides
    to install the CLI.
 
+#### Google Cloud deployment
+
+Before deploying to Google Compute Platform, ensure that you have:
+
+1. An account with billing enabled.
+   (Remember, deploying clusters will incur charges. Make sure to destroy
+   resources when you're finished with them!)
+2. Google Cloud Project you will be deploying to. You could follow
+   this [guide]( https://developers.google.com/workspace/guides/create-project) to create one. Project ID will be used
+   as cloud-profile during setup.
+3. A public DNS zone hosted in Cloud DNS.
+   To set this up, you can follow [this guide](https://cloud.google.com/dns/docs/set-up-dns-records-domain-name).
+4. A user account with `Owner` permissions.
+   You can use [this guide](https://developers.google.com/apps-script/guides/admin/assign-cloud-permissions) to grant
+   permissions.
+5. The Google Cloud CLI (**gcloud**) and **google-cloud-cli-gke-gcloud-auth-plugin** plugin installed and configured to
+   use this user.
+   You can use [this](https://cloud.google.com/sdk/docs/install-sdk)
+   and [this](https://cloud.google.com/sdk/docs/authorizing) guides to install and configure the CLI.
+
 ## Installation process
 
 Once you have the prerequisites installed and configured, you are ready to install the CGDevX CLI.

diff --git a/...ps-pipelines/delivery/clusters/cc-cluster/core-services/components/vault/application.yaml b/...ps-pipelines/delivery/clusters/cc-cluster/core-services/components/vault/application.yaml
@@ -58,7 +58,7 @@ spec:
                     leader_api_addr = "http://vault-2.vault-internal:8200"
                   }
                 }
-                # <SECRET_MANAGER_SEAL>
+                # <SECRET_MANAGER_UNSEAL>
                 service_registration "kubernetes" {}
           ingress:
             enabled: true

diff --git a/platform/terraform/hosting_provider/output.tf b/platform/terraform/hosting_provider/output.tf
@@ -73,12 +73,16 @@ output "cluster_oidc_provider_arn" {
 ################################################################################
 # secret manager
 ################################################################################
-output "secret_manager_seal_key" {
+output "secret_manager_unseal_key" {
   value       = module.hosting-provider.secret_manager_unseal_key
   description = "Secret Manager seal key"
   sensitive   = true
 }
-
+output "secret_manager_unseal_key_ring" {
+  value       = module.hosting-provider.secret_manager_unseal_key_ring
+  description = "Secret Manager unseal key ring"
+  sensitive   = true
+}
 ################################################################################
 # artifact storage
 ################################################################################

diff --git a/platform/terraform/modules/cloud_aws/TERRAFORM-README.md b/platform/terraform/modules/cloud_aws/TERRAFORM-README.md
@@ -62,9 +62,9 @@
 | <a name="input_cluster_version"></a> [cluster\_version](#input\_cluster\_version) | (Optional) Specifies the EKS Kubernetes version | `string` | `"1.30"` | no |
 | <a name="input_domain_name"></a> [domain\_name](#input\_domain\_name) | Specifies the platform domain name | `string` | n/a | yes |
 | <a name="input_node_group_type"></a> [node\_group\_type](#input\_node\_group\_type) | n/a | `string` | `"EKS"` | no |
-| <a name="input_node_groups"></a> [node\_groups](#input\_node\_groups) | n/a | <pre>list(object({<br>    name           = optional(string, "default")<br>    instance_types = optional(list(string), ["m5.large"])<br>    capacity_type  = optional(string, "on_demand")<br>    min_size       = optional(number, 3)<br>    max_size       = optional(number, 6)<br>    desired_size   = optional(number, 4)<br>  }))</pre> | <pre>[<br>  {<br>    "capacity_type": "on_demand",<br>    "desired_size": 4,<br>    "instance_types": [<br>      "m5.large"<br>    ],<br>    "max_size": 6,<br>    "min_size": 3,<br>    "name": "default"<br>  }<br>]</pre> | no |
+| <a name="input_node_groups"></a> [node\_groups](#input\_node\_groups) | n/a | <pre>list(object({<br>    name           = optional(string, "default")<br>    instance_types = optional(list(string), ["m5.large"])<br>    capacity_type  = optional(string, "on_demand")<br>    min_size       = optional(number, 3)<br>    max_size       = optional(number, 6)<br>    desired_size   = optional(number, 4)<br>    disk_size      = optional(number, 50)<br>    gpu_enabled    = optional(bool, false)<br>  }))</pre> | <pre>[<br>  {<br>    "capacity_type": "on_demand",<br>    "desired_size": 4,<br>    "instance_types": [<br>      "m5.large"<br>    ],<br>    "max_size": 6,<br>    "min_size": 3,<br>    "name": "default"<br>  }<br>]</pre> | no |
 | <a name="input_region"></a> [region](#input\_region) | Specifies the regions | `string` | `"eu-west-1"` | no |
-| <a name="input_tags"></a> [tags](#input\_tags) | (Optional) Specifies the AWS resource tags | `map(string)` | <pre>{<br>  "ProvisionedBy": "CGDevX"<br>}</pre> | no |
+| <a name="input_tags"></a> [tags](#input\_tags) | (Optional) Specifies the AWS resource tags | `map(string)` | <pre>{<br>  "provisioned-by": "cg-devx"<br>}</pre> | no |
 | <a name="input_workloads"></a> [workloads](#input\_workloads) | Workloads configuration | <pre>map(object({<br>    description = optional(string, "")<br>  }))</pre> | `{}` | no |
 
 ## Outputs
@@ -118,6 +118,7 @@
 | <a name="output_public_subnet_id"></a> [public\_subnet\_id](#output\_public\_subnet\_id) | public\_subnet\_id |
 | <a name="output_secret_manager_irsa_role"></a> [secret\_manager\_irsa\_role](#output\_secret\_manager\_irsa\_role) | AWS Secretsmanager IAM Role ARN |
 | <a name="output_secret_manager_unseal_key"></a> [secret\_manager\_unseal\_key](#output\_secret\_manager\_unseal\_key) | The globally unique identifier for the secret manager key |
+| <a name="output_secret_manager_unseal_key_ring"></a> [secret\_manager\_unseal\_key\_ring](#output\_secret\_manager\_unseal\_key\_ring) | Secret Manager unseal key ring |
 | <a name="output_self_managed_node_groups"></a> [self\_managed\_node\_groups](#output\_self\_managed\_node\_groups) | Map of attribute maps for all self managed node groups created |
 | <a name="output_self_managed_node_groups_autoscaling_group_names"></a> [self\_managed\_node\_groups\_autoscaling\_group\_names](#output\_self\_managed\_node\_groups\_autoscaling\_group\_names) | List of the autoscaling group names created by self-managed node groups |
 | <a name="output_vpc_cni_irsa"></a> [vpc\_cni\_irsa](#output\_vpc\_cni\_irsa) | vpc\_cni role ARN |

diff --git a/platform/terraform/modules/cloud_aws/outputs.tf b/platform/terraform/modules/cloud_aws/outputs.tf
@@ -132,6 +132,12 @@ output "kms_key_policy" {
   value       = module.eks.kms_key_policy
 }
 
+output "secret_manager_unseal_key_ring" {
+  value       = ""
+  description = "Secret Manager unseal key ring"
+  sensitive   = true
+}
+
 ################################################################################
 # Security Group
 ################################################################################

diff --git a/platform/terraform/modules/cloud_aws/vpc.tf b/platform/terraform/modules/cloud_aws/vpc.tf
@@ -25,7 +25,7 @@ module "vpc" {
   }
 
   intra_subnet_tags = {
-    "Tier" = "intra"
+    "Tier" = "infra"
   }
 
 }
diff --git a/platform/terraform/modules/cloud_azure/TERRAFORM-README.md b/platform/terraform/modules/cloud_azure/TERRAFORM-README.md
@@ -72,9 +72,9 @@
 | <a name="input_cluster_version"></a> [cluster\_version](#input\_cluster\_version) | (Optional) Specifies the AKS Kubernetes version | `string` | `"1.30"` | no |
 | <a name="input_domain_name"></a> [domain\_name](#input\_domain\_name) | Specifies the platform domain name | `string` | n/a | yes |
 | <a name="input_enable_native_auto_scaling"></a> [enable\_native\_auto\_scaling](#input\_enable\_native\_auto\_scaling) | Enables AKS native autoscaling feature. | `bool` | `false` | no |
-| <a name="input_node_groups"></a> [node\_groups](#input\_node\_groups) | n/a | <pre>list(object({<br>    name           = optional(string, "default")<br>    instance_types = optional(list(string), ["Standard_B2ms"])<br>    capacity_type  = optional(string, "Regular")<br>    min_size       = optional(number, 3)<br>    max_size       = optional(number, 5)<br>    desired_size   = optional(number, 3)<br>  }))</pre> | <pre>[<br>  {<br>    "capacity_type": "on_demand",<br>    "desired_size": 3,<br>    "instance_types": [<br>      "Standard_B2ms"<br>    ],<br>    "max_size": 5,<br>    "min_size": 3,<br>    "name": "default"<br>  }<br>]</pre> | no |
+| <a name="input_node_groups"></a> [node\_groups](#input\_node\_groups) | n/a | <pre>list(object({<br>    name           = optional(string, "default")<br>    instance_types = optional(list(string), ["Standard_B2ms"])<br>    capacity_type  = optional(string, "Regular")<br>    min_size       = optional(number, 3)<br>    max_size       = optional(number, 5)<br>    desired_size   = optional(number, 3)<br>    disc_size      = optional(number, 50)<br>    gpu_enabled    = optional(bool, false)<br>  }))</pre> | <pre>[<br>  {<br>    "capacity_type": "on_demand",<br>    "desired_size": 3,<br>    "instance_types": [<br>      "Standard_B2ms"<br>    ],<br>    "max_size": 5,<br>    "min_size": 3,<br>    "name": "default"<br>  }<br>]</pre> | no |
 | <a name="input_region"></a> [region](#input\_region) | Specifies the location for the resource group and all the resources | `string` | `"westeurope"` | no |
-| <a name="input_tags"></a> [tags](#input\_tags) | (Optional) Specifies the Azure resource tags | `map(string)` | <pre>{<br>  "ProvisionedBy": "CGDevX"<br>}</pre> | no |
+| <a name="input_tags"></a> [tags](#input\_tags) | (Optional) Specifies the Azure resource tags | `map(string)` | <pre>{<br>  "provisioned-by": "cg-devx"<br>}</pre> | no |
 | <a name="input_workloads"></a> [workloads](#input\_workloads) | Workloads configuration | <pre>map(object({<br>    description = optional(string, "")<br>  }))</pre> | `{}` | no |
 
 ## Outputs
@@ -95,5 +95,6 @@
 | <a name="output_kube_config_raw"></a> [kube\_config\_raw](#output\_kube\_config\_raw) | Contains the Kubernetes config to be used by kubectl and other compatible tools. |
 | <a name="output_network_id"></a> [network\_id](#output\_network\_id) | Platform primary K8s cluster network ID |
 | <a name="output_secret_manager_irsa_role"></a> [secret\_manager\_irsa\_role](#output\_secret\_manager\_irsa\_role) | Secrets Manager IAM role for a K8s service account |
-| <a name="output_secret_manager_unseal_key"></a> [secret\_manager\_unseal\_key](#output\_secret\_manager\_unseal\_key) | Secret Manager seal key |
+| <a name="output_secret_manager_unseal_key"></a> [secret\_manager\_unseal\_key](#output\_secret\_manager\_unseal\_key) | Secret Manager unseal key |
+| <a name="output_secret_manager_unseal_key_ring"></a> [secret\_manager\_unseal\_key\_ring](#output\_secret\_manager\_unseal\_key\_ring) | Secret Manager unseal key ring |
 <!-- END_TF_DOCS -->
diff --git a/platform/terraform/modules/cloud_azure/outputs.tf b/platform/terraform/modules/cloud_azure/outputs.tf
@@ -72,7 +72,13 @@ output "cluster_node_groups" {
 ################################################################################
 output "secret_manager_unseal_key" {
   value       = azurerm_key_vault_key.secret_manager_unseal_kms_key.name
-  description = "Secret Manager seal key"
+  description = "Secret Manager unseal key"
+  sensitive   = true
+}
+
+output "secret_manager_unseal_key_ring" {
+  value       = ""
+  description = "Secret Manager unseal key ring"
   sensitive   = true
 }