Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MULTIARCH-3164] add NBDE encryption for IBM Z #58373

Merged
merged 2 commits into from
May 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions installing/installing_ibm_z/installing-ibm-z-kvm.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,14 @@ include::modules/ibm-z-secure-execution.adoc[leveloffset=+2]

* link:https://www.ibm.com/docs/en/linux-on-systems?topic=ibmz-secure-execution[Linux as an IBM Secure Execution host or guest]

include::modules/ibmz-configure-nbde-with-static-ip.adoc[leveloffset=+2]

[role="_additional-resources"]
[id="additional-resources_configure-nbde-ibm-z-kvm"]
.Additional resources

* xref:../../installing/install_config/installing-customizing.adoc#installation-special-config-butane_installing-customizing[Creating machine configs with Butane]

include::modules/installation-ibm-z-kvm-user-infra-machines-iso.adoc[leveloffset=+2]

include::modules/installation-full-ibm-z-kvm-user-infra-machines-iso.adoc[leveloffset=+2]
Expand Down
8 changes: 8 additions & 0 deletions installing/installing_ibm_z/installing-ibm-z.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,14 @@ include::modules/nw-operator-cr.adoc[leveloffset=+1]

include::modules/installation-user-infra-generate-k8s-manifest-ignition.adoc[leveloffset=+1]

include::modules/ibmz-configure-nbde-with-static-ip.adoc[leveloffset=+1]

[role="_additional-resources"]
[id="additional-resources_configure-nbde-ibm-z"]
.Additional resources

* xref:../../installing/install_config/installing-customizing.adoc#installation-special-config-butane_installing-customizing[Creating machine configs with Butane]

include::modules/installation-ibm-z-user-infra-machines-iso.adoc[leveloffset=+1]

include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+2]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,14 @@ include::modules/ibm-z-secure-execution.adoc[leveloffset=+2]

* link:https://www.ibm.com/docs/en/linux-on-systems?topic=ibmz-secure-execution[Linux as an IBM Secure Execution host or guest]

include::modules/ibmz-configure-nbde-with-static-ip.adoc[leveloffset=+2]

[role="_additional-resources"]
[id="additional-resources_configure-nbde-ibm-z-kvm-restricted"]
.Additional resources

* xref:../../installing/install_config/installing-customizing.adoc#installation-special-config-butane_installing-customizing[Creating machine configs with Butane]

include::modules/installation-ibm-z-kvm-user-infra-machines-iso.adoc[leveloffset=+2]

include::modules/installation-full-ibm-z-kvm-user-infra-machines-iso.adoc[leveloffset=+2]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,14 @@ include::modules/nw-operator-cr.adoc[leveloffset=+1]

include::modules/installation-user-infra-generate-k8s-manifest-ignition.adoc[leveloffset=+1]

include::modules/ibmz-configure-nbde-with-static-ip.adoc[leveloffset=+1]

[role="_additional-resources"]
[id="additional-resources_Configure-nbde-ibm-z-restricted"]
.Additional resources

* xref:../../installing/install_config/installing-customizing.adoc#installation-special-config-butane_installing-customizing[Creating machine configs with Butane]

include::modules/installation-ibm-z-user-infra-machines-iso.adoc[leveloffset=+1]

include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+2]
Expand Down
162 changes: 162 additions & 0 deletions modules/ibmz-configure-nbde-with-static-ip.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
// Module included in the following assemblies:
//
// * installing/installing_ibm_z/installing-ibm-z.adoc
// * installing/installing_ibm_z/installing-restricted-networks-ibm-z.adoc
// * installing/installing_ibm_z/installing-ibm-z-kvm.adoc
// * installing/installing_ibm_z/installing-restricted-networks-ibm-z-kvm.adoc

ifeval::["{context}" == "installing-ibm-z"]
:ibm-z:
endif::[]
ifeval::["{context}" == "installing-ibm-z-kvm"]
:ibm-z-kvm:
endif::[]
ifeval::["{context}" == "installing-restricted-networks-ibm-z"]
:ibm-z:
endif::[]
ifeval::["{context}" == "installing-restricted-networks-ibm-z-kvm"]
:ibm-z-kvm:
endif::[]

:_content-type: PROCEDURE
[id="configuring-nbde-static-ip-ibmz-linuxone-environment_{context}"]
= Configuring NBDE with static IP in an {ibmzProductName} or {linuxoneProductName} environment

Enabling NBDE disk encryption in an {ibmzProductName} or {linuxoneProductName} environment requires additional steps, which are described in detail in this section.

.Prerequisites

* You have set up the External Tang Server. See link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/configuring-automated-unlocking-of-encrypted-volumes-using-policy-based-decryption_security-hardening#network-bound-disk-encryption_configuring-automated-unlocking-of-encrypted-volumes-using-policy-based-decryption[Network-bound disk encryption] for instructions.
* You have installed the `butane` utility.
* You have reviewed the instructions for how to create machine configs with Butane.

.Procedure

. Create Butane configuration files for the control plane and compute nodes.
+
The following example of a Butane configuration for a control plane node creates a file named `master-storage.bu` for disk encryption:
+
[source,yaml]
----
variant: openshift
Copy link

@techrustlings techrustlings Apr 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shortcut method works with s390x zfcp, but not works with dasd. So editing the ignition is not recommended. Define the storage configuration directly in the butane. Kindly remove the line from 41 to 52 and add the following config as example.

variant: openshift
version: 4.13.0
metadata:
  name: master-storage
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  filesystems:
    - device: /dev/mapper/root
      format: xfs
      label: root
      wipe_filesystem: true
  luks:
    - clevis:
        tang:
          - thumbprint: QcPr_NHFJammnRCA3fFMVdNBwjs
            url: http://12.23.21.58:7500
      device: /dev/disk/by-partlabel/root
      label: luks-root
      name: root
      wipe_volume: true

version: 4.13.0
metadata:
name: master-storage
labels:
machineconfiguration.openshift.io/role: master
storage:
luks:
- clevis:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the FIPS case, the luks entry should also include:

options:
  - --cipher
  - aes-cbc-essiv:sha256

We need this to ensure that we're using a FIPS 140-2 compatible cipher mode. The default cipher mode is too new for 140-2 (but is supported in 140-3). Normally Butane handles this if it sees boot_device.luks and also openshift.fips: true, but since we're not using boot_device we need to do this ourselves.

I'd be inclined to footnote it and say that it's only needed in FIPS mode.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed because I removed FIPS entry.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SNiemann15 the comment from @bgilbert catched my eyes.
in case a customer want to have fips mode & a change for LUKS is required to adjust the cipher, we might want to document that. Because encryption is kind of base for compliance rules which also includes FIPS.

tang:
- thumbprint: QcPr_NHFJammnRCA3fFMVdNBwjs
url: http://clevis.example.com:7500
options: <1>
- --cipher
- aes-cbc-essiv:sha256
ifndef::ibm-z-kvm[]
device: /dev/disk/by-partlabel/root <2>
endif::ibm-z-kvm[]
ifdef::ibm-z-kvm[]
device: /dev/disk/by-partlabel/root
endif::ibm-z-kvm[]
label: luks-root
name: root
wipe_volume: true
filesystems:
- device: /dev/mapper/root
format: xfs
label: root
wipe_filesystem: true
openshift:
ifndef::ibm-z-kvm[]
fips: true <3>
endif::ibm-z-kvm[]
ifdef::ibm-z-kvm[]
fips: true <2>
endif::ibm-z-kvm[]
----
ifdef::ibm-z-kvm[]
<1> The cipher option is only required if FIPS mode is enabled. Omit the entry if FIPS is disabled.
<2> Whether to enable or disable FIPS mode. By default, FIPS mode is not enabled. If FIPS mode is enabled, the {op-system-first} machines that {product-title} runs on bypass the default Kubernetes cryptography suite and use the cryptography modules that are provided with {op-system} instead.
endif::ibm-z-kvm[]
ifndef::ibm-z-kvm[]
<1> The cipher option is only required if FIPS mode is enabled. Omit the entry if FIPS is disabled.
<2> For installations on DASD-type disks, replace with `device: /dev/disk/by-label/root`.
<3> Whether to enable or disable FIPS mode. By default, FIPS mode is not enabled. If FIPS mode is enabled, the {op-system-first} machines that {product-title} runs on bypass the default Kubernetes cryptography suite and use the cryptography modules that are provided with {op-system} instead.
endif::ibm-z-kvm[]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The instructions don't say what to do with the Butane config once created. I assume we want to render it to a MachineConfig and add that to the install templates? We usually give explicit steps for that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a link to Creating machine configs with Butane in the additional resources.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but I think there's a risk that users will follow the steps verbatim and won't realize that additional actions are implied.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just below the line no 71 a Note section will remind the user to follow the Adding day 1 kernel arguments. Like below .

Generate the MachineConfig object from master-storage.bu by using butane and place it in the Openshift installation directory as described in "Adding day-1 kernel arguments"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets tackle this issue after 4.13 GA.

. Create a customized initramfs file to boot the machine, by running the following command:
+
[source,terminal]
----
$ coreos-installer pxe customize \
/root/rhcos-bootfiles/rhcos-<release>-live-initramfs.s390x.img \
--dest-device /dev/sda --dest-karg-append \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--dest-device does the same thing as coreos.inst.install_dev in the next step. I think it's better to only use --dest-device, and footnote the different values that should be used for virt/ECKD/FCP.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not require --dest-device in initramfs file for Dasda configuration. It has been mentioned in the call out section just below the example.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm suggesting that the instructions either consistently use only --dest-device or consistently use only coreos.inst.install_dev, rather than sometimes using both and having the latter override the former.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used --dest-device in coreos-installer command, and the coreos.inst.install_dev we used in the kernel param file which we punch during the installation. Are you suggesting we use --dest-device in kernel param file too?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I'm suggesting that we either use --dest-device in the coreos-installer pxe customize command or coreos.inst.install_dev in the kernel param file. It's unnecessary and confusing to use both.

ip=<ip-address>::<gateway-ip>:<subnet-mask>::<network-device>:none \
--dest-karg-append nameserver=<nameserver-ip> \
--dest-karg-append rd.neednet=1 -o \
/root/rhcos-bootfiles/<Node-name>-initramfs.s390x.img
----
Comment on lines +92 to +99
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not going to hold up the PR for this, but I would strongly recommend coming back later and adding annotations/callouts to describe the user-replaced values in this block.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll pick that up in the follow-up PR we are planning post GA.

+
[NOTE]
====
Before first boot, you must customize the initramfs for each node in the cluster, and add PXE kernel parameters.
====

. Create a parameter file that includes `ignition.platform.id=metal` and `ignition.firstboot`.
+
.Example kernel parameter file for the control plane machine:
+
ifndef::ibm-z-kvm[]
[source,terminal]
----
rd.neednet=1 \
console=ttysclp0 \
coreos.inst.install_dev=/dev/dasda \ <1>
ignition.firstboot ignition.platform.id=metal \
coreos.live.rootfs_url=http://10.19.17.25/redhat/ocp/rhcos-413.86.202302201445-0/rhcos-413.86.202302201445-0-live-rootfs.s390x.img \
coreos.inst.ignition_url=http://bastion.ocp-cluster1.example.com:8080/ignition/master.ign \
ip=10.19.17.2::10.19.17.1:255.255.255.0::enbdd0:none nameserver=10.19.17.1 \
zfcp.allow_lun_scan=0 \ <2>
rd.znet=qeth,0.0.bdd0,0.0.bdd1,0.0.bdd2,layer2=1 \
rd.zfcp=0.0.5677,0x600606680g7f0056,0x034F000000000000 \ <3>
zfcp.allow_lun_scan=0 \
rd.znet=qeth,0.0.bdd0,0.0.bdd1,0.0.bdd2,layer2=1 \
rd.zfcp=0.0.5677,0x600606680g7f0056,0x034F000000000000
----
<1> For installations on DASD-type disks, add `coreos.inst.install_dev=/dev/dasda`. Omit this value for FCP-type disks.
<2> For installations on FCP-type disks, add `zfcp.allow_lun_scan=0`. Omit this value for DASD-type disks.
<3> For installations on DASD-type disks, replace with `rd.dasd=0.0.3490` to specify the DASD device.
endif::ibm-z-kvm[]
ifdef::ibm-z-kvm[]
[source,terminal]
----
rd.neednet=1 \
console=ttysclp0 \
ignition.firstboot ignition.platform.id=metal \
coreos.live.rootfs_url=http://10.19.17.25/redhat/ocp/rhcos-413.86.202302201445-0/rhcos-413.86.202302201445-0-live-rootfs.s390x.img \
coreos.inst.ignition_url=http://bastion.ocp-cluster1.example.com:8080/ignition/master.ign \
ip=10.19.17.2::10.19.17.1:255.255.255.0::enbdd0:none nameserver=10.19.17.1 \
zfcp.allow_lun_scan=0 \
rd.znet=qeth,0.0.bdd0,0.0.bdd1,0.0.bdd2,layer2=1 \
rd.zfcp=0.0.5677,0x600606680g7f0056,0x034F000000000000
Comment on lines +134 to +142
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just mentioning that this block doesn't have callouts unlike the previous ifndef::ibm-z-kvm[] block, in case you think it's a miss. If it is intentional, then great.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional :-)

----
endif::ibm-z-kvm[]
+
[NOTE]
====
Write all options in the parameter file as a single line and make sure you have no newline characters.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be a note or added to the step above. I would prefer a note consideirng the structure and reowrding it to be something like:
Ensure that all options in the parameter file are in a single line, and have no newline characters.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to keep this active voice. But having it as a note is a good suggestion.

====

ifeval::["{context}" == "installing-ibm-z"]
:!ibm-z:
endif::[]
ifeval::["{context}" == "installing-ibm-z-kvm"]
:!ibm-z-kvm:
endif::[]
ifeval::["{context}" == "installing-restricted-networks-ibm-z"]
:!ibm-z:
endif::[]
ifeval::["{context}" == "installing-restricted-networks-ibm-z-kvm"]
:!ibm-z-kvm:
endif::[]
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ $ virt-install \
--network network={virt_network_parm} \
--boot hd \
--location {media_location},kernel={rhcos_kernel},initrd={rhcos_initrd} \
--extra-args "rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url={rhcos_liveos} ip={ip}::{default_gateway}:{subnet_mask_length}:{vn_name}:enc1:none:{MTU} nameserver={dns} coreos.inst.ignition_url={rhcos_ign}" \
--extra-args "rd.neednet=1 coreos.inst.install_dev=/dev/vda coreos.live.rootfs_url={rhcos_liveos} ip={ip}::{default_gateway}:{subnet_mask_length}:{vn_name}:enc1:none:{MTU} nameserver={dns} coreos.inst.ignition_url={rhcos_ign}" \
--noautoconsole \
--wait
----
8 changes: 4 additions & 4 deletions modules/installation-ibm-z-user-infra-machines-iso.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ The rootfs image is the same for FCP and DASD.
** For `coreos.live.rootfs_url=`, specify the matching rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported.

** For installations on DASD-type disks, complete the following tasks:
... For `coreos.inst.install_dev=`, specify `dasda`.
... For `coreos.inst.install_dev=`, specify `/dev/dasda`.
... Use `rd.dasd=` to specify the DASD where {op-system} is to be installed.
... Leave all other parameters unchanged.
+
Expand All @@ -61,7 +61,7 @@ Example parameter file, `bootstrap-0.parm`, for the bootstrap machine:
----
rd.neednet=1 \
console=ttysclp0 \
coreos.inst.install_dev=dasda \
coreos.inst.install_dev=/dev/dasda \
coreos.live.rootfs_url=http://cl1.provide.example.com:8080/assets/rhcos-live-rootfs.s390x.img \
coreos.inst.ignition_url=http://cl1.provide.example.com:8080/ignition/bootstrap.ign \
ip=172.18.78.2::172.18.78.1:255.255.255.0:::none nameserver=172.18.78.1 \
Expand All @@ -79,7 +79,7 @@ Write all options in the parameter file as a single line and make sure you have
====
When you install with multiple paths, you must enable multipathing directly after the installation, not at a later point in time, as this can cause problems.
====
... Set the install device as: `coreos.inst.install_dev=sda`.
... Set the install device as: `coreos.inst.install_dev=/dev/sda`.
+
[NOTE]
====
Expand All @@ -99,7 +99,7 @@ The following is an example parameter file `worker-1.parm` for a worker node wit
----
rd.neednet=1 \
console=ttysclp0 \
coreos.inst.install_dev=sda \
coreos.inst.install_dev=/dev/sda \
coreos.live.rootfs_url=http://cl1.provide.example.com:8080/assets/rhcos-live-rootfs.s390x.img \
coreos.inst.ignition_url=http://cl1.provide.example.com:8080/ignition/worker.ign \
ip=172.18.78.2::172.18.78.1:255.255.255.0:::none nameserver=172.18.78.1 \
Expand Down