From 9b0aa83e504b856f8886b4d1ce52538ce8ea46ff Mon Sep 17 00:00:00 2001 From: Dan With Date: Tue, 17 Dec 2024 08:51:13 -0600 Subject: [PATCH] Adding QoS examples for Cinder Volumes and in particular NetApp ONTAP (#622) * Adding QoS examples for Cinder Volumes and in particular NetApp ONTAP The following documentation had warning notes updated: openstack-cinder-lvmisci.md openstack-cinder-netapp-container.md openstack-cinder-netapp-worker.md openstack-cinder-volume-provisioning-specs.md openstack-cinder-volume-qos-policies.md openstack-cinder-volume-type-specs.md * newline fix * extra line fix * volume type specs edit * Add link to current NetApp ONTAP doc with QoS explanations --- ...eploy-cinder-netapp-volumes-reference.yaml | 10 +- docs/openstack-cinder-lvmisci.md | 29 +---- docs/openstack-cinder-netapp-container.md | 17 +-- docs/openstack-cinder-netapp-worker.md | 10 +- ...nstack-cinder-volume-provisioning-specs.md | 12 ++ docs/openstack-cinder-volume-qos-policies.md | 109 ++++++++++++++++++ docs/openstack-cinder-volume-type-specs.md | 16 +++ mkdocs.yml | 4 + 8 files changed, 157 insertions(+), 50 deletions(-) create mode 100644 docs/openstack-cinder-volume-provisioning-specs.md create mode 100644 docs/openstack-cinder-volume-qos-policies.md create mode 100644 docs/openstack-cinder-volume-type-specs.md diff --git a/ansible/playbooks/deploy-cinder-netapp-volumes-reference.yaml b/ansible/playbooks/deploy-cinder-netapp-volumes-reference.yaml index b01931b6..67cbf874 100644 --- a/ansible/playbooks/deploy-cinder-netapp-volumes-reference.yaml +++ b/ansible/playbooks/deploy-cinder-netapp-volumes-reference.yaml @@ -7,7 +7,7 @@ cinder_release: "2024.1" cinder_storage_network_interface: ansible_br_storage cinder_storage_network_interface_secondary: ansible_br_storage_secondary - cinder_backend_name: "block-ha-performance-at-rest-encrypted,block-ha-standard-at-rest-encrypted" + cinder_backend_name: "block-ha-performance-at-rest-encrypted,block-ha-standard-at-rest-encrypted,block-ha-performance-end-to-end-encrypted,block-ha-standard-end-to-end-encrypted" custom_multipath: false handlers: - name: Restart cinder-volume-netapp systemd services @@ -198,14 +198,6 @@ group: root mode: "0644" - - name: Replace yaml safe qos policy key in netapp-backends configuration - ansible.builtin.replace: - path: /etc/cinder/netapp-backends.conf - regexp: 'netapp_qos_policy_group' - replace: 'netapp:qos_policy_group' - notify: - - Restart cinder-volume-netapp systemd services - - name: Replace exec path in rootwrap community.general.ini_file: path: /etc/cinder/rootwrap.conf diff --git a/docs/openstack-cinder-lvmisci.md b/docs/openstack-cinder-lvmisci.md index 640f5dcc..300f1b44 100644 --- a/docs/openstack-cinder-lvmisci.md +++ b/docs/openstack-cinder-lvmisci.md @@ -119,35 +119,8 @@ root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin- !!! warning - You must define and apply the volume type QoS policies **before** creating a volume within your new volume type (`lvmdriver-1`). Also, any additional volume provisioning specifications like the minimum and maximum size for the volume type must be set **before** creating a volume within your new volume type (`lvmdriver-1`). + **Before** creating a volume, ensure that your volume type has been set up correctly and that you have applied QoS policies, provisioning specifications (min and max volume size), and any extra specs. See [Cinder Volume QoS Policies](openstack-cinder-volume-qos-policies.md), [Cinder Volume Provisioning Specs](openstack-cinder-volume-provisioning-specs.md), and [Cinder Volume Type Specs](openstack-cinder-volume-type-specs.md). -In order to apply a QoS policy to the `lvmdriver-1` volume type, you must first create the QoS policy. - -``` shell -root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume qos create --consumer "both" --property "read_iops_sec_per_gb=1" --property "write_iops_sec_per_gb=1" lvmdriver-1-iops -+------------+-----------------------------------------------------+ -| Field | Value | -+------------+-----------------------------------------------------+ -| consumer | both | -| id | b35fdf9c-d5bd-40f9-ae3a-8605c246ef2e | -| name | lvmdriver-1-iops | -| properties | read_iops_sec_per_gb='1', write_iops_sec_per_gb='1' | -+------------+-----------------------------------------------------+ -``` - -Once you have created the QoS policy, apply it to the `lvmdriver-1` volume type. -The command will utilize the `QOS_ID` and `VOLUME_TYPE_ID`. - -``` shell -root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume qos associate b35fdf9c-d5bd-40f9-ae3a-8605c246ef2e 6af6ade2-53ca-4260-8b79-1ba2f208c91d -``` - -Set any additional volume provisioning specifications like the minimum and maximum volume size. These specifications are set in the volume type. The following commands constrain the `lvmdriver-1` volume type to a size between 10 GB and 2 TB. - -``` shell -root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type set --property provisioning:min_vol_size=10 6af6ade2-53ca-4260-8b79-1ba2f208c91d -root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type set --property provisioning:max_vol_size=2048 6af6ade2-53ca-4260-8b79-1ba2f208c91d -``` ## Validate functionality diff --git a/docs/openstack-cinder-netapp-container.md b/docs/openstack-cinder-netapp-container.md index 3bfdb8c0..fd55b8f8 100644 --- a/docs/openstack-cinder-netapp-container.md +++ b/docs/openstack-cinder-netapp-container.md @@ -3,7 +3,8 @@ This document provides information on configuring NetApp backends for the isolated Cinder volume worker. Each backend is defined by a set of 11 comma-separated options, and multiple backends can be specified by separating them with semicolons. -!!! warning "The NetApp container is incompatible with iSCSI workloads" + +!!! warning "The NetApp container is incompatible with iSCSI workloads" The NetApp container is incompatible with iSCSI workloads. If the environment requires iSCSI support, review the [Cinder NetApp Worker ](openstack-cinder-netapp-worker.md) documentation instead. @@ -19,11 +20,10 @@ Below is a table detailing each option, its position in the backend configuratio | 3 | `netapp_server_hostname` | Hostname or IP address of the NetApp storage system. | String | | 4 | `netapp_server_port` | Port number to communicate with the NetApp storage system. | Integer | | 5 | `netapp_vserver` | The name of the Vserver on the NetApp storage system. | String | -| 6 | `netapp:qos_policy_group` | The name of the QoS policy group. | String | -| 7 | `netapp_dedup` | Enable (`True`) or disable (`False`) deduplication. | Boolean | -| 8 | `netapp_compression` | Enable (`True`) or disable (`False`) compression. | Boolean | -| 9 | `netapp_thick_provisioned` | Use thick (`True`) or thin (`False`) provisioning. | Boolean | -| 10 | `netapp_lun_space_reservation`| Enable (`enabled`) or disable (`disabled`) LUN space reservation. | String | +| 6 | `netapp_dedup` | Enable (`True`) or disable (`False`) deduplication. | Boolean | +| 7 | `netapp_compression` | Enable (`True`) or disable (`False`) compression. | Boolean | +| 8 | `netapp_thick_provisioned` | Use thick (`True`) or thin (`False`) provisioning. | Boolean | +| 9 | `netapp_lun_space_reservation`| Enable (`enabled`) or disable (`disabled`) LUN space reservation. | String | ### Detailed Option Descriptions @@ -33,7 +33,6 @@ Below is a table detailing each option, its position in the backend configuratio - **`netapp_server_hostname`**: The address of the NetApp storage system. This can be either an IP address or a fully qualified domain name (FQDN). - **`netapp_server_port`**: The port number used for communication with the NetApp storage system. Common ports are `80` for HTTP and `443` for HTTPS. - **`netapp_vserver`**: Specifies the virtual storage server (Vserver) on the NetApp storage system that will serve the volumes. -- **`netapp:qos_policy_group`**: The Quality of Service (QoS) policy group name that will be applied to volumes for this backend. - **`netapp_dedup`**: A boolean value to enable or disable deduplication on the storage volumes. Acceptable values are `True` or `False`. - **`netapp_compression`**: A boolean value to enable or disable compression on the storage volumes. Acceptable values are `True` or `False`. - **`netapp_thick_provisioned`**: Determines whether volumes are thick (`True`) or thin (`False`) provisioned. @@ -62,6 +61,10 @@ in a specific order. ## Run the deployment +!!! warning + + **Before** deploying a new backend, ensure that your volume type has been set up correctly and that you have applied QoS policies, provisioning specifications (min and max volume size), and any extra specs. See [Cinder Volume QoS Policies](openstack-cinder-volume-qos-policies.md), [Cinder Volume Provisioning Specs](openstack-cinder-volume-provisioning-specs.md), and [Cinder Volume Type Specs](openstack-cinder-volume-type-specs.md). + With your configuration defined, run the deployment with a standard `kubectl apply` command. ``` shell diff --git a/docs/openstack-cinder-netapp-worker.md b/docs/openstack-cinder-netapp-worker.md index b4abf9bf..d0e171a3 100644 --- a/docs/openstack-cinder-netapp-worker.md +++ b/docs/openstack-cinder-netapp-worker.md @@ -6,6 +6,10 @@ The NetApp Volume Worker is a cinder-volume service that is configured to use th The `deploy-cinder-volumes-netapp-reference.yaml` playbook is used to deploy the NetApp Volume Worker. The playbook will deploy the cinder-volume service on the specified nodes and configure the service to use the NetApp ONTAP driver. This playbook will read from the Kubernetes environment to determine the necessary configuration parameters for the NetApp ONTAP driver. As an operator, you will need to ensure that the necessary configuration parameters are set in the Kubernetes environment before running the playbook. +!!! warning + + **Before** deploying a new backend, ensure that your volume type has been set up correctly and that you have applied QoS policies, provisioning specifications (min and max volume size), and any extra specs. See [Cinder Volume QoS Policies](openstack-cinder-volume-qos-policies.md), [Cinder Volume Provisioning Specs](openstack-cinder-volume-provisioning-specs.md), and [Cinder Volume Type Specs](openstack-cinder-volume-type-specs.md). + ### Backends Configuration The NetApp ONTAP driver requires a backend configuration to be set in the Kubernetes environment. The backend configuration specifies the storage system that the NetApp Volume Worker will use to create and manage volumes. The backend configuration is a Kubernetes secret that contains the necessary configuration parameters for the NetApp ONTAP driver. To define the backends, update the helm overrides file with the necessary configuration parameters. @@ -24,7 +28,6 @@ The NetApp ONTAP driver requires a backend configuration to be set in the Kubern netapp_storage_protocol: iscsi netapp_transport_type: http netapp_vserver: - netapp_qos_policy_group: netapp_dedup: True netapp_compression: True netapp_thick_provisioned: True @@ -40,7 +43,6 @@ The NetApp ONTAP driver requires a backend configuration to be set in the Kubern netapp_storage_protocol: iscsi netapp_transport_type: http netapp_vserver: - netapp_qos_policy_group: netapp_dedup: True netapp_compression: True netapp_thick_provisioned: True @@ -56,10 +58,6 @@ The NetApp ONTAP driver requires a backend configuration to be set in the Kubern - **`SERVER_NAME_OR_ADDRESS`**: The address of the NetApp storage system. This can be either an IP address or a fully qualified domain name (FQDN). - **`SERVER_PORT`**: The port number used for communication with the NetApp storage system. Common ports are `80` for HTTP and `443` for HTTPS. - **`VSERVER`**: Specifies the virtual storage server (Vserver) on the NetApp storage system that will serve the volumes. -- **`QOS_POLICY_GROUP`**: The Quality of Service (QoS) policy group name that will be applied to volumes for this backend. - -!!! note - The Cinder backends configuration file utilizes a colon (:) in the variable name `netapp:qos_policy_group`, but it has been replaced in the Helm overrides file with an underscore. Helm recognizes a colon (:) as a special character and renders it as an equal (=) sign. This is undesired behavior for the final rendering of the Cinder backends configuration file, and the Ansible playbook replaces the underscore with a colon (:). The extra spec requires the colon (:) in its name because it is used by the NetAppdriver to assign the QoS policy group to the OpenStack Block Storage volume after it has been provisioned. See [NetApp unified driver](https://docs.openstack.org/cinder/latest/configuration/block-storage/drivers/netapp-volume-driver.html) for more detailed information on the NetApp driver extra specs. ## Host Setup diff --git a/docs/openstack-cinder-volume-provisioning-specs.md b/docs/openstack-cinder-volume-provisioning-specs.md new file mode 100644 index 00000000..d19b148c --- /dev/null +++ b/docs/openstack-cinder-volume-provisioning-specs.md @@ -0,0 +1,12 @@ +# Set Volume Type Provisioning Specifications + +*Before* creating a volume within any volume type, the provisioning specifications must be set. + +## Minimum and Maximum volume size + +These specifications are set in the volume type. The following commands constrain the `lvmdriver-1` volume type to a size between 10 GB and 2 TB. + +``` shell +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type set --property provisioning:min_vol_size=10 6af6ade2-53ca-4260-8b79-1ba2f208c91d +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type set --property provisioning:max_vol_size=2048 6af6ade2-53ca-4260-8b79-1ba2f208c91d +``` diff --git a/docs/openstack-cinder-volume-qos-policies.md b/docs/openstack-cinder-volume-qos-policies.md new file mode 100644 index 00000000..e1176778 --- /dev/null +++ b/docs/openstack-cinder-volume-qos-policies.md @@ -0,0 +1,109 @@ +# Volume QoS Policies + +## LVM + + +### Example QoS policy for LVM driver volume type + +In order to apply a QoS policy to the `lvmdriver-1` volume type, you must first create the QoS policy. + +``` shell +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume qos create --consumer "both" --property "read_iops_sec_per_gb=1" --property "write_iops_sec_per_gb=1" lvmdriver-1-iops ++------------+-----------------------------------------------------+ +| Field | Value | ++------------+-----------------------------------------------------+ +| consumer | both | +| id | b35fdf9c-d5bd-40f9-ae3a-8605c246ef2e | +| name | lvmdriver-1-iops | +| properties | read_iops_sec_per_gb='1', write_iops_sec_per_gb='1' | ++------------+-----------------------------------------------------+ +``` + +Once you have created the QoS policy, apply it to the `lvmdriver-1` volume type. +The command will utilize the `QOS_ID` and `VOLUME_TYPE_ID`. + +``` shell +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume qos associate b35fdf9c-d5bd-40f9-ae3a-8605c246ef2e 6af6ade2-53ca-4260-8b79-1ba2f208c91d +``` + +## NetAPP ONTAP + +The most recent releases of the ONTAP driver (OpenStack Train and higher) allow QoS policies to be set per volume at the Cinder volume type rather than trying to utilize a QoS policy created on a target NetApp device. For a more detailed explanation, consult [NetApp Cinder QoS Concepts](https://netapp-openstack-dev.github.io/openstack-docs/train/cinder/key_concepts/section_cinder-key-concepts.html#qos-spec) + +### Example QoS policy for NetApp ONTAP volume type + +In order to apply a QoS policy to the `netapp-1` volume type, you must first create the QoS policy. + +``` shell +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume qos create --consumer "back-end" --property "peakIOPSperGiB=8" --property "expectedIOPSperGiB=7" netapp-qos ++------------+--------------------------------------------+ +| Field | Value | ++------------+--------------------------------------------+ +| consumer | back-end | +| id | 9435160f-0e4a-4486-88b0-d6beb022732a | +| name | netapp-qos | +| properties | expectedIOPSperGiB='7', peakIOPSperGiB='8' | ++------------+--------------------------------------------+ +``` + +`expectedIOPSperGiB=7` was chosen because the target IOPSperGiB or expectedIOPSperGiB will be observed to be `5 IOPSperGiB` or `5,000 IOPS` when running FIO tests. Likewise, the `peakIOPSperGiB=8` was chosen because it is a value of `1` over the `expectedIOPSperGiB=7` and will effectively cap a burst in IOPs to an actual observerd value of `6 IOPSperGiB` or `6,000 IOPS`. + +``` shell +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type show 1bdb5364-ed04-4bbe-8e41-9c5fae148c3d ++--------------------+----------------------------------------+ +| Field | Value | ++--------------------+----------------------------------------+ +| access_project_ids | | +| description | None | +| id | 1bdb5364-ed04-4bbe-8e41-9c5fae148c3d | +| is_public | False | +| name | netapp-1 | +| properties | volume_backend_name='netapp-1-backend' | +| qos_specs_id | None | ++--------------------+----------------------------------------+ +``` + +Once you have created the QoS policy, apply it to the `netapp-1` volume type. +The command will utilize the `QOS_ID` and `VOLUME_TYPE_ID`. + +``` shell +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume qos associate 9435160f-0e4a-4486-88b0-d6beb022732a 1bdb5364-ed04-4bbe-8e41-9c5fae148c3d +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type show 1bdb5364-ed04-4bbe-8e41-9c5fae148c3d ++--------------------+----------------------------------------+ +| Field | Value | ++--------------------+----------------------------------------+ +| access_project_ids | | +| description | None | +| id | 1bdb5364-ed04-4bbe-8e41-9c5fae148c3d | +| is_public | False | +| name | netapp-1 | +| properties | volume_backend_name='netapp-1-backend' | +| qos_specs_id | 9435160f-0e4a-4486-88b0-d6beb022732a | ++--------------------+----------------------------------------+ +``` + +## Disassociate a QoS policy from a volume type + +In order to delete a QoS policy, you must first disassociate it from any volume types that it has been associated with. + +``` shell +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume qos disassociate --volume-type 1bdb5364-ed04-4bbe-8e41-9c5fae148c3d 9435160f-0e4a-4486-88b0-d6beb022732a +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type show 1bdb5364-ed04-4bbe-8e41-9c5fae148c3d ++--------------------+----------------------------------------+ +| Field | Value | ++--------------------+----------------------------------------+ +| access_project_ids | | +| description | None | +| id | 1bdb5364-ed04-4bbe-8e41-9c5fae148c3d | +| is_public | False | +| name | netapp-1 | +| properties | volume_backend_name='netapp-1-backend' | +| qos_specs_id | None | ++--------------------+----------------------------------------+ +``` + +## Delete a QoS policy + +``` shell +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume qos delete 9435160f-0e4a-4486-88b0-d6beb022732a +``` diff --git a/docs/openstack-cinder-volume-type-specs.md b/docs/openstack-cinder-volume-type-specs.md new file mode 100644 index 00000000..836d3797 --- /dev/null +++ b/docs/openstack-cinder-volume-type-specs.md @@ -0,0 +1,16 @@ +# Set Volume Type Specifications + +## Example: NetApp ONTAP driver volume specifications (deduplication and compression) + +To set and unset additional properties on a NetApp volume type, use the syntax examples. + +``` shell + +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type set --property netapp_dedup='true' + +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type set --property netapp_compression='true' + +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type unset --property netapp_dedup + +root@openstack-node-0:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack volume type unset --property netapp_compression +``` diff --git a/mkdocs.yml b/mkdocs.yml index abb1e031..0bcdca03 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -268,6 +268,10 @@ nav: - OpenStack: - CLI Access: - Generating Clouds YAML: openstack-clouds.md + - Block Storage: + - Cinder Volume Provisioning Specs: openstack-cinder-volume-provisioning-specs.md + - Cinder Volume QoS Policies: openstack-cinder-volume-qos-policies.md + - Cinder Volume Type Specs: openstack-cinder-volume-type-specs.md - Compute: - Nova Flavor Creation: openstack-flavors.md - Nova CPU Allocation Ratio: openstack-cpu-allocation-ratio.md