Spot Ocean aks np k8s Terraform Module

Spotinst Terraform Module to integrate existing k8s with Ocean

Prerequisites

Installation of the Ocean controller is required by this resource. You can accomplish this by using the spotinst/terraform-ocean-kubernetes-controller module. The kubernetes provider will need to be initilaized before calling the ocean-controller module as follows:

provider "spotinst" {
  token   = "redacted"
  account = "redacted"
}

module "ocean-aks-np" {
  source  = "spotinst/ocean-aks-np-k8s/spotinst"
  ...
}
##################
## Option 1 to initialize kubernetes provider ##
provider "kubernetes" {
  config_path = "~/.kube/config"
}
##################

module "kubernetes-controller" {
  source = "spotinst/kubernetes-controller/ocean"

  # Credentials.
  spotinst_token   = "redacted"
  spotinst_account = "redacted"

  # Configuration.
  cluster_identifier = "cluster name"
}

~> You must configure the same cluster_identifier both for the Ocean controller and for the spotinst_ocean_aks_np resource. The ocean-aks-np module will use the cluster name as the identifier. Ensure this is also used in the controller config

Usage

### Create Azure Ocean on spot console ###
module "ocean-aks-np" {
  source = "spotinst/ocean-aks-np-k8s/spotinst"
  # Credentials.
  spotinst_token                          = var.spotinst_token
  spotinst_account                        = var.spotinst_account

  # Configuration
  ocean_cluster_name                       = "test"
  aks_region                               = "eastus"
  controller_cluster_id                    = "test-123124"
  aks_cluster_name                         = "test-cluster"
  aks_infrastructure_resource_group_name   = "MC_TestResourceGroup_test-cluster_eastus"
  aks_resource_group_name                  = "TestResourceGroup"
  autoscaler_is_enabled                    = true
  autoscaler_resource_limits_max_vcpu      = 750
  autoscaler_resource_limits_max_memory_gib= 1500
  autoscaler_headroom_automatic_is_enabled = true
  autoscaler_max_scale_down_percentage     = 30
  autoscaler_headroom_automatic_percentage = 5
  health_grace_period                      = 600
  max_pods_per_node                        = 30
  enable_node_public_ip                    = true
  os_disk_size_gb                          = 30
  os_disk_type                             = "Managed"
  os_type                                  = "Linux"
  os_sku                                   = "Ubuntu"
  kubernetes_version                       = "1.26"
  pod_subnet_ids                           = ["/subscriptions/123456-1234-1234-1234-123456789/resourceGroups/ExampleResourceGroup/providers/Microsoft.Network/virtualNetworks/ExampleVirtualNetwork/subnets/default"]
  vnet_subnet_ids                          = ["/subscriptions/123456-1234-1234-1234-123456789/resourceGroups/ExampleResourceGroup/providers/Microsoft.Network/virtualNetworks/ExampleVirtualNetwork/subnets/default"]
  linux_os_config                          = {
    vm_max_map_count                       = 79550}
  node_min_count                           = 1
  node_max_count                           = 100
  spot_percentage                          = 50
  fallback_to_ondemand                     = true
  autoscale_headrooms_cpu_per_unit         = 6
  autoscale_headrooms_memory_per_unit      = 10
  autoscale_headrooms_gpu_per_unit         = 4
  autoscale_headrooms_num_of_units         = 10
  availability_zones                       = [1, 2, 3]
  tags                                     = { "tagKey": "env", "tagValue": "staging" }
  labels                                   = { "key": "env","value": "test"}
  shutdown_hours                           = { is_enabled = false 
                                               time_windows = ["Fri:15:30-Sat:13:30", "Sun:15:30-Mon:13:30"] }
  taints                                   = [{"key":"taintKey","value":"taintValue", "effect" : "NoSchedule"}]
  vmsizes_filters_min_vcpu                 = 2
  vmsizes_filters_max_vcpu                 = 16
  vmsizes_filters_min_memory_gib           = 10
  vmsizes_filters_max_memory_gib           = 18
  vmsizes_filters_series                   = ["D v3", "Dds_v4", "Dsv2"]
  vmsizes_filters_architectures            = ["X86_64"]
  should_roll                              = true
  conditioned_roll                         = false
  batch_size_percentage                    = 25
  batch_min_healthy_percentage             = 100
  # Scheduling tasks parameters block (clusterRoll)
  tasks                                    = [
    {
      is_enabled                           = true
      cron_expression                      = "0 1 * * *"
      task_type                            = "clusterRoll"
      parameters_cluster_roll              = [{
        batch_min_healthy_percentage       = 80
        batch_size_percentage              = 20
        comment                            = "Scheduled cluster roll"
        respect_pdb                        = true
        respect_restrict_scale_down        = true
        vng_ids                            = ["vng123", "vng456"]
      }
      ]
    }
  ]
}

Providers

Name	Version
spotinst/spotinst	>= 1.102.0

Modules

• ocean-aks-np-k8s - Creates Ocean Cluster Doc
• ocean-controller - Create and installs Spot Ocean controller pod Doc
• ocean-aws-k8s-vng - (Optional) Add custom virtual node groups Doc

Documentation

If you're new to Spot and want to get started, please checkout our Getting Started guide, available on the Spot Documentation website.

Getting Help

We use GitHub issues for tracking bugs and feature requests. Please use these community resources for getting help:

• Ask a question on Stack Overflow and tag it with terraform-spotinst.
• Join our Spot community on Slack.
• Open an issue.

Community

• Slack
• Twitter

Contributing

Please see the contribution guidelines.

Modules

No modules.

Inputs

Name	Description	Type	Default	Required
ocean_cluster_name	The Ocean cluster.	string	null	yes
aks_region	The cluster's region	string	null	yes
controller_cluster_id	Enter a unique Ocean cluster identifier. Cannot be updated.	string	null	yes
aks_cluster_name	The name of the AKS Cluster	string	null	yes
aks_infrastructure_resource_group_name	The name of the cluster's infrastructure resource group.	string	null	yes
aks_resource_group_name	The name of the cluster's resource group.	string	null	yes
autoscaler_is_enabled	Enable the Ocean Kubernetes Autoscaler.	bool	true	yes
autoscaler_resource_limits_max_vcpu	The maximum cpu in vCpu units that can be allocated to the cluster.	number	null	no
autoscaler_resource_limits_max_memory_gib	The maximum memory in GiB units that can be allocated to the cluster.	number	null	no
autoscaler_max_scale_down_percentage	The maximum percentage allowed to scale down in a single scaling action.	number	null	no
autoscaler_headroom_automatic_is_enabled	Enable automatic headroom. When set to True, Ocean configures and optimizes headroom automatically.	bool	false	no
autoscaler_headroom_automatic_percentage	Optionally set a number between 0-100 to control the percentage of total cluster resources dedicated to headroom.	number	null	no
health_grace_period	The amount of time to wait, in seconds, from the moment the instance has launched until monitoring of its health checks begins.	number	600	no
availability_zones	An Array holding Availability Zones, this configures the availability zones the Ocean may launch instances in per VNG.	list(string)	null	yes
max_pods_per_node	he maximum number of pods per node in the node pools.	number	null	no
vm_max_map_count	Maximum number of memory map areas a process may have. Can be configured only if OS type is Linux.	number	null	no
enable_node_public_ip	Enable node public IP.	bool	null	no
os_disk_size_gb	The size of the OS disk in GB.	number	null	no
os_disk_type	The type of the OS disk.	string	null	no
os_type	The OS type of the OS disk.	string	null	no
os_sku	The OS SKU of the OS type. Must correlate with the os type.	string	null	no
kubernetes_version	The desired Kubernetes version of the launched nodes. In case the value is null, the Kubernetes version of the control plane is used.	string	null	no
pod_subnet_ids	The IDs of subnets in an existing VNet into which to assign pods in the cluster (requires azure network-plugin).	list(string)	null	no
vnet_subnet_ids	The IDs of subnets in an existing VNet into which to assign nodes in the cluster (requires azure network-plugin).	list(string)	null	no
node_min_count	Minimum node count limit.	number	null	no
node_max_count	Maximum node count limit.	number	null	no
spot_percentage	Percentage of spot VMs to maintain.	number	100	no
fallback_to_ondemand	If no spot instance markets are available, enable Ocean to launch on-demand instances instead.	bool	true	no
taints	Add taints to a virtual node group.	list(map(string))	null	no
tags	A maximum of 10 unique key-value pairs for VM tags in the virtual node group.	map(string)	null	no
labels	An array of labels to add to the virtual node group.Only custom user labels are allowed, and not Kubernetes built-in labels or Spot internal labels.	map(string)	null	no
scheduling_shutdown_hours_time_windows	The times that the shutdown hours will apply.	list(string)	null	no
scheduling_shutdown_hours_is_enabled	Flag to enable or disable the shutdown hours mechanism. When False, the mechanism is deactivated, and the cluster remains in its current state.	bool	null	no
shutdown_hours	shutdown_hours object.	object	null	no
vmsizes_filters_min_vcpu	Minimum number of vcpus available.	number	null	no
vmsizes_filters_max_vcpu	Maximum number of vcpus available.	number	null	no
vmsizes_filters_min_memory_gib	Minimum amount of Memory (GiB).	number	null	no
vmsizes_filters_max_memory_gib	Maximum amount of Memory (GiB).	number	null	no
vmsizes_filters_architectures	The filtered virtual machine types will support at least one of the architectures from this list.	list(string)	null	no
vmsizes_filters_series	Vm sizes belonging to a series from the list will be available for scaling.	list(string)	null	no
vmsizes_filters_exclude_series	Vm sizes belonging to a series from the list will not be available for scaling.	list(string)	null	no
vmsizes_filters_accelereated_networking	In case acceleratedNetworking is set to Enabled, accelerated networking applies only to the VM that enables it.	string	null	no
vmsizes_filters_disk_performance	The filtered vm sizes will support at least one of the classes from this list.	string	null	no
vmsizes_filters_min_gpu	Minimum number of GPUs available.	number	null	no
vmsizes_filters_max_gpu	Maximum number of GPUs available.	number	null	no
vmsizes_filters_min_nics	Minimum number of network interfaces.	number	null	no
vmsizes_filters_min_disk	Minimum number of data disks available.	number	null	no
vmsizes_filters_vm_types	The filtered vm types will belong to one of the vm types from this list.	list(string)	null	no
vmsizes_filters_gpu_types	The filtered gpu types will belong to one of the gpu types from this list. Supported GPU Types: nvidia-tesla-v100, amd-radeon-instinct-mi25, nvidia-a10, nvidia-tesla-a100, nvidia-tesla-k80, nvidia-tesla-m60, nvidia-tesla-p100, nvidia-tesla-p40, nvidia-tesla-t4, nvidia-tesla-h100.	list(string)	null	no
autoscale_headrooms_num_of_units	The number of units to retain as headroom, where each unit has the defined headroom CPU, memory.	number	null	no
autoscale_headrooms_cpu_per_unit	Configure the number of CPUs to allocate for each headroom unit. CPUs are denoted in millicores, where 1000 millicores = 1 vCPU.	number	null	no
autoscale_headrooms_gpu_per_unit	Amont of GPU to allocate for headroom unit.	number	null	no
autoscale_headrooms_memory_per_unit	Configure the amount of memory (MiB) to allocate the headroom	number	null	no
should_roll	If set to true along with the cluster update, roll will be triggered.	bool	false	no
conditioned_roll	Default: false. Spot will perform a cluster Roll in accordance with a relevant modification of the cluster’s settings. When set to true , only specific changes in the cluster’s configuration will trigger a cluster roll (such as availability_zones, max_pods_per_node, enable_node_public_ip, os_disk_size_gb, os_disk_type, os_sku, kubernetes_version, vnet_subnet_ids, pod_subnet_ids, labels, taints and tags).	bool	null	no
respect_pdb	During the roll, if the parameter is set to true we honor PDB during the nodes replacement	bool	true	no
batch_size_percentage	Value as a percent to set the size of a batch in a roll. Valid values are 0-100. In case of null as value, the default value in the backend will be 20%.	number	20	no
vng_ids	List of virtual node group identifiers to be rolled. Each identifier is a string. vngIds can be null, and cannot be used together with nodeNames and nodePoolNames.	list(string)	null	no
batch_min_healthy_percentage	Default: 50. Indicates the threshold of minimum healthy nodes in single batch. If the amount of healthy nodes in single batch is under the threshold, the roll will fail. If exists, the parameter value will be in range of 1-100. In case of null as value, the default value in the backend will be 50%. Value of param should represent the number in percentage (%) of the batch.	number	null	no
comment	Add a comment description for the roll. The comment is limited to 256 chars and optional.	string	null	no
node_pool_names	List of node pools to be rolled. Each node pool name is a string. nodePoolNames can be null, and cannot be used together with nodeNames and vngIds.	list(string)	null	no
respect_restrict_scale_down	During the roll, if the parameter is set to true we honor Restrict Scale Down label during the nodes replacement.	bool	false	no
node_names	List of node names to be rolled. Each identifier is a string. nodeNames can be null, and cannot be used together with nodePoolNames and vngIds.	list(string)	null	no
tasks	task object	list(object({ is_enabled = bool cron_expression = string task_type = string parameters_cluster_roll = set(object({ batch_min_healthy_percentage = number batch_size_percentage = number comment = string vng_ids = list(string) respect_restrict_scale_down = bool respect_pdb = bool })), []) }))	null	no

Outputs

Name	Description

| ocean_id | The Ocean cluster ID |

License

Code is licensed under the Apache License 2.0.