Skip to content

Commit

Permalink
Add e2e tests for RHEL 9 EFI on Baremetal
Browse files Browse the repository at this point in the history
- Add simple workflow test cases for RHEL 9 EFI on Baremetal.
- Introcude a new OSFamily for RHEL 9, the purpose of this is to
get the correct partition index when generating default tinkerbell
actions.
- Documentation improvement for RHEL 9 on Baremetal.
  • Loading branch information
2ez4szliu committed Aug 21, 2024
1 parent 75d5c92 commit 8465d8a
Show file tree
Hide file tree
Showing 8 changed files with 139 additions and 16 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,12 @@ env:
T_TINKERBELL_IMAGE_REDHAT_1_28: "tinkerbell_ci:image_redhat_1_28"
T_TINKERBELL_IMAGE_REDHAT_1_29: "tinkerbell_ci:image_redhat_1_29"
T_TINKERBELL_IMAGE_REDHAT_1_30: "tinkerbell_ci:image_redhat_1_30"
T_TINKERBELL_IMAGE_REDHAT_9_1_25: "tinkerbell_ci:image_redhat_9_1_25"
T_TINKERBELL_IMAGE_REDHAT_9_1_26: "tinkerbell_ci:image_redhat_9_1_26"
T_TINKERBELL_IMAGE_REDHAT_9_1_27: "tinkerbell_ci:image_redhat_9_1_27"
T_TINKERBELL_IMAGE_REDHAT_9_1_28: "tinkerbell_ci:image_redhat_9_1_28"
T_TINKERBELL_IMAGE_REDHAT_9_1_29: "tinkerbell_ci:image_redhat_9_1_29"
T_TINKERBELL_IMAGE_REDHAT_9_1_30: "tinkerbell_ci:image_redhat_9_1_30"
T_TINKERBELL_SSH_AUTHORIZED_KEY: "vsphere_ci_beta_connection:ssh_authorized_key"
T_TINKERBELL_CP_NETWORK_CIDR: "tinkerbell_ci:cp_network_cidr"
T_TINKERBELL_S3_INVENTORY_CSV_KEY: "tinkerbell_ci:s3_inventory_csv"
Expand Down
27 changes: 17 additions & 10 deletions docs/content/en/docs/getting-started/baremetal/bare-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ In the example, there are `TinkerbellMachineConfig` sections for control plane (
The following fields identify information needed to configure the nodes in each of those groups.
>**_NOTE:_** Currently, you can only have one machine group for all machines in the control plane, although you can have multiple machine groups for the workers.
>
### hardwareSelector (optional)
### hardwareSelector (Required)
Use fields under `hardwareSelector` to add key/value pair labels to match particular machines that you identified in the CSV file where you defined the machines in your cluster.
Choose any label name you like.
For example, if you had added the label `node=cp-machine` to the machines listed in your CSV file that you want to be control plane nodes, the following `hardwareSelector` field would cause those machines to be added to the control plane:
Expand All @@ -321,7 +321,7 @@ spec:
node: "cp-machine"
```
### osFamily (required)
Operating system on the machine. Permitted values: `bottlerocket`, `ubuntu`, `redhat` (Default: `bottlerocket`).
Operating system on the machine. Permitted values: `bottlerocket`, `ubuntu`, `redhat`, `redhat9` (Default: `bottlerocket`).

### osImageURL (required)
Required field to set the operating system. In order to use Ubuntu or RHEL see [building baremetal node images]({{< relref "../../osmgmt/artifacts/#build-bare-metal-node-images" >}}). This field is also useful if you want to provide a customized operating system image or simply host the standard image locally. To upgrade a node or group of nodes to a new operating system version (ie. RHEL 8.7 to RHEL 8.8), modify this field to point to the new operating system image URL and run [upgrade cluster command]({{< relref "../../clustermgmt/cluster-upgrades/baremetal-upgrades/#upgrade-cluster-command" >}}). The `osImageURL` must contain the `Cluster.Spec.KubernetesVersion` or `Cluster.Spec.WorkerNodeGroupConfiguration[].KubernetesVersion` version (in case of modular upgrade). For example, if the Kubernetes version is 1.24, the `osImageURL` name should include 1.24, 1_24, 1-24 or 124.
Expand Down Expand Up @@ -365,6 +365,11 @@ When you generate a Bare Metal cluster configuration, the `TinkerbellTemplateCon
Advanced users can override the default values set for `TinkerbellTemplateConfig`.
They can also add their own [Tinkerbell actions](https://docs.tinkerbell.org/actions/action-architecture/) to make personalized modifications to EKS Anywhere nodes.

A default `TinkerbellTemplateConfig` can be generated from cluster config template using eksctl:
```bash
eksctl anywhere generate tinkerbelltemplateconfig -f eksa-mgmt-cluster.yaml
```

The following shows three `TinkerbellTemplateConfig` examples that you can add to your cluster configuration file to override the values that EKS Anywhere sets: one for Ubuntu, one for RHEL and one for Bottlerocket.
Most actions used differ for different operating systems.

Expand Down Expand Up @@ -470,7 +475,7 @@ spec:
version: "0.1"
```

### RHEL TinkerbellTemplateConfig example
### RHEL9 TinkerbellTemplateConfig example

```yaml
---
Expand All @@ -487,13 +492,13 @@ spec:
- actions:
- environment:
COMPRESSED: "true"
DEST_DISK: /dev/sda
DEST_DISK: '{{ index .Hardware.Disks 0 }}'
IMG_URL: https://my-file-server/rhel-9-uefi-amd64.gz
image: public.ecr.aws/eks-anywhere/tinkerbell/hub/image2disk:6c0f0d437bde2c836d90b000312c8b25fa1b65e1-eks-a-15
name: stream-image
timeout: 360
- environment:
DEST_DISK: /dev/sda2
DEST_DISK: '{{ formatPartition ( index .Hardware.Disks 0 ) 3 }}'
DEST_PATH: /etc/netplan/config.yaml
STATIC_NETPLAN: true
DIRMODE: "0755"
Expand All @@ -503,17 +508,18 @@ spec:
UID: "0"
image: public.ecr.aws/eks-anywhere/tinkerbell/hub/writefile:6c0f0d437bde2c836d90b000312c8b25fa1b65e1-eks-a-15
name: write-netplan
pid: host
timeout: 90
- environment:
CONTENTS: |
datasource:
Ec2:
metadata_urls: [<admin-machine-ip>, <tinkerbell-ip-from-cluster-config>]
metadata_urls: [<admin-machine-ip>:50061, <tinkerbell-ip-from-cluster-config>:50061]
strict_id: false
manage_etc_hosts: localhost
warnings:
dsid_missing_source: off
DEST_DISK: /dev/sda2
DEST_DISK: '{{ formatPartition ( index .Hardware.Disks 0 ) 3 }}'
DEST_PATH: /etc/cloud/cloud.cfg.d/10_tinkerbell.cfg
DIRMODE: "0700"
FS_TYPE: ext4
Expand All @@ -527,7 +533,7 @@ spec:
CONTENTS: |
network:
config: disabled
DEST_DISK: /dev/sda2
DEST_DISK: '{{ formatPartition ( index .Hardware.Disks 0 ) 3 }}'
DEST_PATH: /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
DIRMODE: "0700"
FS_TYPE: ext4
Expand All @@ -540,7 +546,7 @@ spec:
- environment:
CONTENTS: |
datasource: Ec2
DEST_DISK: /dev/sda2
DEST_DISK: '{{ formatPartition ( index .Hardware.Disks 0 ) 3 }}'
DEST_PATH: /etc/cloud/ds-identify.cfg
DIRMODE: "0700"
FS_TYPE: ext4
Expand All @@ -550,8 +556,9 @@ spec:
image: public.ecr.aws/eks-anywhere/tinkerbell/hub/writefile:6c0f0d437bde2c836d90b000312c8b25fa1b65e1-eks-a-15
name: add-tink-cloud-init-ds-config
timeout: 90
- name: "reboot"
- name: "reboot-image"
image: public.ecr.aws/eks-anywhere/tinkerbell/hub/reboot:6c0f0d437bde2c836d90b000312c8b25fa1b65e1-eks-a-15
pid: host
timeout: 90
volumes:
- /worker:/worker
Expand Down
1 change: 1 addition & 0 deletions pkg/api/v1alpha1/machine_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ const (
Ubuntu OSFamily = "ubuntu"
Bottlerocket OSFamily = "bottlerocket"
RedHat OSFamily = "redhat"
RedHat9 OSFamily = "redhat9"
)

// UserConfiguration defines the configuration of the user to be added to the VM.
Expand Down
5 changes: 3 additions & 2 deletions pkg/api/v1alpha1/tinkerbellmachineconfig.go
Original file line number Diff line number Diff line change
Expand Up @@ -85,12 +85,13 @@ func validateTinkerbellMachineConfig(config *TinkerbellMachineConfig) error {
return fmt.Errorf("TinkerbellMachineConfig: missing spec.osFamily: %s", config.Name)
}

if config.Spec.OSFamily != Ubuntu && config.Spec.OSFamily != Bottlerocket && config.Spec.OSFamily != RedHat {
if config.Spec.OSFamily != Ubuntu && config.Spec.OSFamily != Bottlerocket && config.Spec.OSFamily != RedHat && config.Spec.OSFamily != RedHat9 {
return fmt.Errorf(
"TinkerbellMachineConfig: unsupported spec.osFamily (%v); Please use one of the following: %s, %s, %s",
"TinkerbellMachineConfig: unsupported spec.osFamily (%v); Please use one of the following: %s, %s, %s, %s",
config.Spec.OSFamily,
Ubuntu,
RedHat,
RedHat9,
Bottlerocket,
)
}
Expand Down
6 changes: 3 additions & 3 deletions test/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,9 +123,9 @@ T_TINKERBELL_SSH_AUTHORIZED_KEY # ssh public key for connectioning to machines
```
### Tinkerbell hardware-inventory.csv example
```csv
guid,ip_address,gateway,nameservers,netmask,mac,hostname,vendor,bmc_ip,bmc_username,bmc_password,labels,disk
bb341bc6-546f-4b38-s584-bb4f0e5f8934,10.24.32.110,10.24.32.1,8.8.8.8,255.255.255.0,3c:ec:ef:6e:a4:82,eksa-node01,supermicro,10.24.32.10,admin,password,type=cp,/dev/sda
cc5619b8-a894-4db0-bf1a-fd04d5964d54,10.24.32.111,10.24.32.1,8.8.8.8,,255.255.255.0,3c:ec:ef:6e:a5:7c,eksa-node02,supermicro,10.24.32.11,admin,password,type=worker,/dev/sda
ip_address,gateway,nameservers,netmask,mac,hostname,vendor,bmc_ip,bmc_username,bmc_password,labels,disk
10.24.32.110,10.24.32.1,8.8.8.8,255.255.255.0,3c:ec:ef:6e:a4:82,eksa-node01,supermicro,10.24.32.10,admin,password,type=cp,/dev/sda
10.24.32.111,10.24.32.1,8.8.8.8,,255.255.255.0,3c:ec:ef:6e:a5:7c,eksa-node02,supermicro,10.24.32.11,admin,password,type=worker,/dev/sda
```

## CloudStack tests requisites
Expand Down
66 changes: 66 additions & 0 deletions test/e2e/tinkerbell_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1405,6 +1405,72 @@ func TestTinkerbellKubernetes130RedHatSimpleFlow(t *testing.T) {
runTinkerbellSimpleFlow(test)
}

func TestTinkerbellKubernetes125RedHat9SimpleFlow(t *testing.T) {
test := framework.NewClusterE2ETest(
t,
framework.NewTinkerbell(t, framework.WithRedHat9125Tinkerbell()),
framework.WithClusterFiller(api.WithKubernetesVersion(v1alpha1.Kube125)),
framework.WithControlPlaneHardware(1),
framework.WithWorkerHardware(1),
)
runTinkerbellSimpleFlow(test)
}

func TestTinkerbellKubernetes126RedHat9SimpleFlow(t *testing.T) {
test := framework.NewClusterE2ETest(
t,
framework.NewTinkerbell(t, framework.WithRedHat9126Tinkerbell()),
framework.WithClusterFiller(api.WithKubernetesVersion(v1alpha1.Kube126)),
framework.WithControlPlaneHardware(1),
framework.WithWorkerHardware(1),
)
runTinkerbellSimpleFlow(test)
}

func TestTinkerbellKubernetes127RedHat9SimpleFlow(t *testing.T) {
test := framework.NewClusterE2ETest(
t,
framework.NewTinkerbell(t, framework.WithRedHat9127Tinkerbell()),
framework.WithClusterFiller(api.WithKubernetesVersion(v1alpha1.Kube127)),
framework.WithControlPlaneHardware(1),
framework.WithWorkerHardware(1),
)
runTinkerbellSimpleFlow(test)
}

func TestTinkerbellKubernetes128RedHat9SimpleFlow(t *testing.T) {
test := framework.NewClusterE2ETest(
t,
framework.NewTinkerbell(t, framework.WithRedHat9128Tinkerbell()),
framework.WithClusterFiller(api.WithKubernetesVersion(v1alpha1.Kube128)),
framework.WithControlPlaneHardware(1),
framework.WithWorkerHardware(1),
)
runTinkerbellSimpleFlow(test)
}

func TestTinkerbellKubernetes129RedHat9SimpleFlow(t *testing.T) {
test := framework.NewClusterE2ETest(
t,
framework.NewTinkerbell(t, framework.WithRedHat9129Tinkerbell()),
framework.WithClusterFiller(api.WithKubernetesVersion(v1alpha1.Kube129)),
framework.WithControlPlaneHardware(1),
framework.WithWorkerHardware(1),
)
runTinkerbellSimpleFlow(test)
}

func TestTinkerbellKubernetes130RedHat9SimpleFlow(t *testing.T) {
test := framework.NewClusterE2ETest(
t,
framework.NewTinkerbell(t, framework.WithRedHat9130Tinkerbell()),
framework.WithClusterFiller(api.WithKubernetesVersion(v1alpha1.Kube130)),
framework.WithControlPlaneHardware(1),
framework.WithWorkerHardware(1),
)
runTinkerbellSimpleFlow(test)
}

func TestTinkerbellKubernetes128BottleRocketSimpleFlow(t *testing.T) {
test := framework.NewClusterE2ETest(
t,
Expand Down
2 changes: 1 addition & 1 deletion test/framework/os_versions.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,5 @@ var osFamiliesForOS = map[OS]anywherev1.OSFamily{
Ubuntu2004: anywherev1.Ubuntu,
Bottlerocket1: anywherev1.Bottlerocket,
RedHat8: anywherev1.RedHat,
RedHat9: anywherev1.RedHat,
RedHat9: anywherev1.RedHat9,
}
42 changes: 42 additions & 0 deletions test/framework/tinkerbell.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,12 @@ const (
tinkerbellImageRedHat128EnvVar = "T_TINKERBELL_IMAGE_REDHAT_1_28"
tinkerbellImageRedHat129EnvVar = "T_TINKERBELL_IMAGE_REDHAT_1_29"
tinkerbellImageRedHat130EnvVar = "T_TINKERBELL_IMAGE_REDHAT_1_30"
tinkerbellImageRedHat9125EnvVar = "T_TINKERBELL_IMAGE_REDHAT_9_1_25"
tinkerbellImageRedHat9126EnvVar = "T_TINKERBELL_IMAGE_REDHAT_9_1_26"
tinkerbellImageRedHat9127EnvVar = "T_TINKERBELL_IMAGE_REDHAT_9_1_27"
tinkerbellImageRedHat9128EnvVar = "T_TINKERBELL_IMAGE_REDHAT_9_1_28"
tinkerbellImageRedHat9129EnvVar = "T_TINKERBELL_IMAGE_REDHAT_9_1_29"
tinkerbellImageRedHat9130EnvVar = "T_TINKERBELL_IMAGE_REDHAT_9_1_30"
tinkerbellInventoryCsvFilePathEnvVar = "T_TINKERBELL_INVENTORY_CSV"
tinkerbellSSHAuthorizedKey = "T_TINKERBELL_SSH_AUTHORIZED_KEY"
tinkerbellCIEnvironmentEnvVar = "T_TINKERBELL_CI_ENVIRONMENT"
Expand All @@ -59,6 +65,12 @@ var requiredTinkerbellEnvVars = []string{
tinkerbellImageRedHat128EnvVar,
tinkerbellImageRedHat129EnvVar,
tinkerbellImageRedHat130EnvVar,
tinkerbellImageRedHat9125EnvVar,
tinkerbellImageRedHat9126EnvVar,
tinkerbellImageRedHat9127EnvVar,
tinkerbellImageRedHat9128EnvVar,
tinkerbellImageRedHat9129EnvVar,
tinkerbellImageRedHat9130EnvVar,
tinkerbellInventoryCsvFilePathEnvVar,
tinkerbellSSHAuthorizedKey,
}
Expand Down Expand Up @@ -264,6 +276,36 @@ func WithRedHat130Tinkerbell() TinkerbellOpt {
return withKubeVersionAndOS(anywherev1.Kube130, RedHat8, "", nil)
}

// WithRedHat9125Tinkerbell tink test with redhat9 efi 1.25.
func WithRedHat9125Tinkerbell() TinkerbellOpt {
return withKubeVersionAndOS(anywherev1.Kube125, RedHat9, "", nil)
}

// WithRedHat9126Tinkerbell tink test with redhat9 efi 1.26.
func WithRedHat9126Tinkerbell() TinkerbellOpt {
return withKubeVersionAndOS(anywherev1.Kube126, RedHat9, "", nil)
}

// WithRedHat9127Tinkerbell tink test with redhat9 efi 1.27.
func WithRedHat9127Tinkerbell() TinkerbellOpt {
return withKubeVersionAndOS(anywherev1.Kube127, RedHat9, "", nil)
}

// WithRedHat9128Tinkerbell tink test with redhat9 efi 1.28.
func WithRedHat9128Tinkerbell() TinkerbellOpt {
return withKubeVersionAndOS(anywherev1.Kube128, RedHat9, "", nil)
}

// WithRedHat9129Tinkerbell tink test with redhat9 efi 1.29.
func WithRedHat9129Tinkerbell() TinkerbellOpt {
return withKubeVersionAndOS(anywherev1.Kube129, RedHat9, "", nil)
}

// WithRedHat9130Tinkerbell tink test with redhat9 efi 1.30.
func WithRedHat9130Tinkerbell() TinkerbellOpt {
return withKubeVersionAndOS(anywherev1.Kube130, RedHat9, "", nil)
}

func WithBottleRocketTinkerbell() TinkerbellOpt {
return func(t *Tinkerbell) {
t.fillers = append(t.fillers,
Expand Down

0 comments on commit 8465d8a

Please sign in to comment.