From 4a15bebfec80f562b2879160aafffeef43812d90 Mon Sep 17 00:00:00 2001 From: Archer Date: Fri, 22 Nov 2024 15:35:04 -0600 Subject: [PATCH 1/6] Add instructions for deploying migration assistant. Signed-off-by: Archer --- .../configuration-options.md | 153 ++++++------------ ...d-security-groups-for-existing-clusters.md | 26 +-- .../deploying-migration-assistant/index.md | 8 +- 3 files changed, 67 insertions(+), 120 deletions(-) diff --git a/_migrations/deploying-migration-assistant/configuration-options.md b/_migrations/deploying-migration-assistant/configuration-options.md index 5a77bd518a..3d9dba5796 100644 --- a/_migrations/deploying-migration-assistant/configuration-options.md +++ b/_migrations/deploying-migration-assistant/configuration-options.md @@ -7,66 +7,35 @@ parent: Deploying migration assistant # Configuration options -This page outlines the configuration options for three key migrations: -1. **Metadata Migration** -2. **Backfill Migration with Reindex-from-Snapshot (RFS)** -3. **Live Capture Migration with Capture and Replay (C&R)** +This page outlines the configuration options for three key migrations scenarios: -Each of these migrations may depend on either a snapshot or a capture proxy. The CDK context blocks below are shown as separate context blocks for each migration type for simplicity. If performing multiple migration types, combine these options, as the actual execution of each migration is controlled from the Migration Console. +1. **Metadata migration** +2. **Backfill migration with `Reindex-from-Snapshot` (RFS)** +3. **Live capture migration with Capture and Replay (C&R)** -It also has a section describing how to specify the auth details for the source and target cluster (no auth, basic auth with a username and password, or sigv4 auth). +Each of these migrations depend on either a snapshot or a capture proxy. They are shown as AWS Cloud Deployment Kits (CDKs). The CDKs on this page are shown as separate blocks for each migration type. If you are performing a migration that suits multiple scenarios, these options can be combined in the migration console. -> [!TIP] -For a complete list of configuration options, please refer to the [opensearch-migrations options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md) but please open an issue for consultation if changing an option that is not listed on this page. -Options for the source cluster endpoint, target cluster endpoint, and existing VPC should be configured for the Migration tools to function effectively. - - -## Metadata Migration Options +For a complete list of configuration options, please refer to the [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If there is a configuration option you would like that is not found on this page, open an issue in OpenSearch Migrations repository. +{: .tip } -## Sample Metadata Migration CDK Options +Options for the source cluster endpoint, target cluster endpoint, and existing VPC should be configured for the Migration tools to function effectively. -```json -{ - "metadata-migration": { - "stage": "dev", - "vpcId": , - "sourceCluster": { - "endpoint": , - "version": "ES 7.10", - "auth": {"type": "none"} - }, - "targetCluster": { - "endpoint": , - "auth": { - "type": "basic", - "username": , - "passwordFromSecretArn": - } - }, - "reindexFromSnapshotServiceEnabled": true, - "artifactBucketRemovalPolicy": "DESTROY" - } -} -``` +## Shared configuration options -There are currently no CDK options specific to Metadata migrations, which are performed from the Migration Console. This migration requires an existing snapshot, which can be created from the Migration Console. +Each migration configuration shares the following options. -
-Shared configuration options table - -| Name | Example | Description | -|-----------------------|-----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `sourceClusterEndpoint` | `"https://source-cluster.elb.us-east-1.endpoint.com"` | The endpoint for the source cluster. | -| `targetClusterEndpoint` | `"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"` | The endpoint for the target cluster. Required if using an existing target cluster for the migration instead of creating a new one. | -| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC where the migration resources will be placed. The VPC must have at least two private subnets that span two availability zones. | +| Name | Example | Description | +| :--- | :--- | :--- | +| `sourceClusterEndpoint` | `"https://source-cluster.elb.us-east-1.endpoint.com"` | The endpoint for the source cluster. | +| `targetClusterEndpoint` | `"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"` | The endpoint for the target cluster. Required if using an existing target cluster for the migration instead of creating a new one. | +| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC where the migration resources will be placed. The VPC must have at least two private subnets that span two availability zones. | -
-## Backfill Migration with Reindex-from-Snapshot (RFS) Options +## Backfill migration using RFS -### Sample Backfill Migration CDK Options +The following CDK performs a backfill migrations using RFS: ```json { @@ -93,22 +62,22 @@ There are currently no CDK options specific to Metadata migrations, which are pe } ``` -Performing a Reindex-from-Snapshot backfill migration requires an existing snapshot. The CDK options specific to backfill migrations are listed below. To view all available arguments for `reindexFromSnapshotExtraArgs`, see [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed. +Performing an RFS backfill migration requires an existing snapshot. -
-Backfill specific configuration options table - -| Name | Example | Description | -|---------------------------------|-----------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `reindexFromSnapshotServiceEnabled` | `true` | Enables deploying and configuring the RFS ECS service. | -| `reindexFromSnapshotExtraArgs` | `"--target-aws-region us-east-1 --target-aws-service-signing-name es"` | Extra arguments for the Document Migration command, with space separation. See the [RFS Extra Arguments](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments) for more details. You can pass `--no-insecure` to remove the `--insecure` flag. | +The RFS CDK uses the following options. To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed. -
-## Live Capture Migration with Capture and Replay (C&R) Options +| Name | Example | Description | +| :--- | :--- | :--- | +| `reindexFromSnapshotServiceEnabled` | `true` | Enables deploying and configuring the RFS ECS service. | +| `reindexFromSnapshotExtraArgs` | `"--target-aws-region us-east-1 --target-aws-service-signing-name es"` | Extra arguments for the Document Migration command, with space separation. See the [RFS Extra Arguments](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments) for more details. You can pass `--no-insecure` to remove the `--insecure` flag. | -### Sample Live Capture Migration CDK Options +To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed. + +## Live capture migration with C&R + +The following sample CDK performs a live capture migration with C&R ```json { @@ -137,28 +106,30 @@ Performing a Reindex-from-Snapshot backfill migration requires an existing snaps } ``` -Performing a live capture migration requires that a Capture Proxy be configured to capture incoming traffic and send it to the target cluster via the Traffic Replayer service. For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed. +Performing a live capture migration requires that a Capture Proxy be configured to capture incoming traffic and send it to the target cluster using the Traffic Replayer service. For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed.
Capture and Replay specific configuration options table -| Name | Example | Description | -|--------------------------------|----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment via a new CloudFormation stack. | -| `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). | -| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment via a new CloudFormation stack. | +| Name | Example | Description | +| :--- | :--- | :--- | +| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment via a new CloudFormation stack. | +| `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). | +| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment via a new CloudFormation stack. | | `trafficReplayerExtraArgs` | `"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"` | Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the [Traffic Replayer](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). | -
-## Cluster Authentication Options +For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed. + -Both the source and target cluster can use no authentication (e.g. limited to the VPC), basic authentication with a username and password, or SigV4 scoped to a user or role. -Examples of each of these are below. +## Cluster authentication pptions + +Both the source and target cluster can use no authentication, such as authentication limited to VPC, basic authentication with a username and password, or SigV4 scoped to a user or role. + +### No authentication -No auth: ``` "sourceCluster": { "endpoint": , @@ -167,7 +138,8 @@ No auth: } ``` -Basic auth: +### Basic authentication + ``` "sourceCluster": { "endpoint": , @@ -180,7 +152,8 @@ Basic auth: } ``` -SigV4 auth: +### SigV4 authentication + ``` "sourceCluster": { "endpoint": , @@ -195,40 +168,8 @@ SigV4 auth: The `serviceSigningName` can be `es` for an Elasticsearch or OpenSearch domain, or `aoss` for an OpenSearch Serverless collection. -All of these auth mechanisms apply to both source and target clusters. - -## Troubleshooting - -### Restricted Permissions -When deploying if part of an [AWS Organization](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html) ↗ some permissions / resources might not be allowed. The full list can be generated from the synthesized cdk output with the awsFeatureUsage script. - -``` -/opensearch-migrations/deployment/cdk/opensearch-service-migration/awsFeatureUsage.sh [contextId] -``` - -
-Capture and Replay specific configuration options table - - -```shell -$ /opensearch-migrations/deployment/cdk/opensearch-service-migration/awsFeatureUsage.sh default -Synthesizing all stacks... -Synthesizing stack: networkStack-default -Synthesizing stack: migrationInfraStack -Synthesizing stack: reindexFromSnapshotStack -Synthesizing stack: migration-console -Finding resource usage from synthesized stacks... ------------------------------------ -IAM Policy Actions: -cloudwatch:GetMetricData -... ------------------------------------ -Resources Types: -AWS::CDK::Metadata -... -``` -
+All of these authentication mechanisms apply to both source and target clusters. +## Network configuration -### Network Configuration The migration tooling expects the source cluster, target cluster, and migration resources to exist in the same VPC. If this is not the case, manual networking setup outside of this documentation is likely required. diff --git a/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md index 46f1d7e11e..0d9af94db2 100644 --- a/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md +++ b/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md @@ -7,25 +7,27 @@ parent: Deploying migration assistant # IAM and security groups for existing clusters -This page outlines scenarios for using the migration tools with existing clusters, including any necessary configuration changes to ensure proper communication between them. +This page outlines security scenarios for using the migration tools with existing clusters, including any necessary configuration changes to ensure proper communication between them. -## Importing an OpenSearch Service or OpenSearch Serverless Target Cluster +## Importing an OpenSearch Service or OpenSearch Serverless target cluster -### OpenSearch Service +Use the following scenarios for OpenSearch Service or Serverless target clusters: + +### OpenSearch service For an OpenSearch Domain, two main configurations are typically required to ensure proper functioning of the migration solution: -1. **Security Group Configuration**: - The Domain should have a security group that allows communication from the applicable Migration services (Traffic Replayer, Migration Console, Reindex-from-Snapshot). The CDK will automatically create an `osClusterAccessSG` security group, which is applied to the Migration services. The user should then add this security group to their existing Domain to allow access. +1. **Security Group Configuration** + + The Domain should have a security group that allows communication from the applicable Migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`). The CDK automatically creates an `osClusterAccessSG` security group, which is applied to the migration services. The user should then add this security group to their existing Domain to allow access. -2. **Access Policy Configuration**: - The Domain’s access policy should either: - - Be an open access policy that allows all access, or - - Be configured to allow at least the IAM task roles for the applicable Migration services (Traffic Replayer, Migration Console, Reindex-from-Snapshot) to access the Domain. +2. **Access Policy Configuration** should be one of the following: + - An open access policy that allows all access. + - Configured to allow at least the IAM task roles for the applicable Migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`) to access the Domain. ### OpenSearch Serverless -For an OpenSearch Serverless Collection, you will need to configure both Network and Data Access policies: +For an OpenSearch Serverless Collection, you will need to configure both network and data access policies: 1. **Network Policy Configuration**: The Collection should have a network policy that uses the `VPC` access type. This requires creating a VPC endpoint on the VPC used for the solution. The VPC endpoint should be configured for the private subnets of the VPC and should attach the `osClusterAccessSG` security group. @@ -35,7 +37,7 @@ For an OpenSearch Serverless Collection, you will need to configure both Network ## Capture Proxy on Coordinator Nodes of Source Cluster -Although the CDK does not automatically set up the Capture Proxy on source cluster nodes (except in the demo solution), the Capture Proxy instances must communicate with the resources deployed by the CDK (e.g., Kafka). This section outlines the necessary steps. +Although the CDK does not automatically set up the Capture Proxy on source cluster nodes (except in the demo solution), the Capture Proxy instances must communicate with the resources deployed by the CDK, such as Kafka. This section outlines the necessary steps to set up communication. Before [setting up Capture Proxy instances](https://github.com/opensearch-project/opensearch-migrations/tree/main/TrafficCapture/trafficCaptureProxyServer#installing-capture-proxy-on-coordinator-nodes) on the source cluster, ensure the following configurations are in place: @@ -69,4 +71,4 @@ Before [setting up Capture Proxy instances](https://github.com/opensearch-projec ## Related Links -- [OpenSearch Traffic Capture Setup](https://github.com/opensearch-project/opensearch-migrations/tree/main/TrafficCapture/trafficCaptureProxyServer#installing-capture-proxy-on-coordinator-nodes) ↗ \ No newline at end of file +- [OpenSearch traffic capture setup] \ No newline at end of file diff --git a/_migrations/deploying-migration-assistant/index.md b/_migrations/deploying-migration-assistant/index.md index cbe721dd12..a2f645a0d8 100644 --- a/_migrations/deploying-migration-assistant/index.md +++ b/_migrations/deploying-migration-assistant/index.md @@ -1,5 +1,9 @@ --- layout: default -title: Deploying migration assistant +title: Deploying Migration Assistant nav_order: 10 ---- \ No newline at end of file +--- + +# Deploying Migration Assistant + +This section details what options are available for deploying Migration Assistant. From 1a0cefa62130970996299991dd833932087ba06b Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:05:51 -0600 Subject: [PATCH 2/6] Apply suggestions from code review Co-authored-by: Peter Nied Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../configuration-options.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/_migrations/deploying-migration-assistant/configuration-options.md b/_migrations/deploying-migration-assistant/configuration-options.md index 3d9dba5796..560779ca9c 100644 --- a/_migrations/deploying-migration-assistant/configuration-options.md +++ b/_migrations/deploying-migration-assistant/configuration-options.md @@ -13,10 +13,10 @@ This page outlines the configuration options for three key migrations scenarios: 2. **Backfill migration with `Reindex-from-Snapshot` (RFS)** 3. **Live capture migration with Capture and Replay (C&R)** -Each of these migrations depend on either a snapshot or a capture proxy. They are shown as AWS Cloud Deployment Kits (CDKs). The CDKs on this page are shown as separate blocks for each migration type. If you are performing a migration that suits multiple scenarios, these options can be combined in the migration console. +Each of these migrations depend on either a snapshot or a capture proxy. The following example `cdk.context.json `configurations that are used by AWS Cloud Deployment Kits (CDKs) to deploy and configure the Migration Assistant, shown as separate blocks for each migration type. If you are performing a migration that suits multiple scenarios, these options can be combined. -For a complete list of configuration options, please refer to the [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If there is a configuration option you would like that is not found on this page, open an issue in OpenSearch Migrations repository. +For a complete list of configuration options, see [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If there is a configuration option you would like that is not found on this page, open an issue in OpenSearch Migrations repository. {: .tip } Options for the source cluster endpoint, target cluster endpoint, and existing VPC should be configured for the Migration tools to function effectively. @@ -65,7 +65,7 @@ The following CDK performs a backfill migrations using RFS: Performing an RFS backfill migration requires an existing snapshot. -The RFS CDK uses the following options. To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed. +RFS configuration uses the following options. To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed. | Name | Example | Description | @@ -114,9 +114,9 @@ Performing a live capture migration requires that a Capture Proxy be configured | Name | Example | Description | | :--- | :--- | :--- | -| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment via a new CloudFormation stack. | +| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment using a CloudFormation stack. | | `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). | -| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment via a new CloudFormation stack. | +| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment using a CloudFormation stack. | | `trafficReplayerExtraArgs` | `"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"` | Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the [Traffic Replayer](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). | @@ -126,7 +126,7 @@ For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fi ## Cluster authentication pptions -Both the source and target cluster can use no authentication, such as authentication limited to VPC, basic authentication with a username and password, or SigV4 scoped to a user or role. +Both the source and target cluster can use no authentication, such as authentication limited to VPC, basic authentication with a username and password, or AWS Signature Version 4 scoped to a user or role. ### No authentication From 4c30f5860ba5f196f4d5e618383da1f3b2540d5d Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 26 Nov 2024 07:20:59 -0600 Subject: [PATCH 3/6] Fix typos Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../configuration-options.md | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/_migrations/deploying-migration-assistant/configuration-options.md b/_migrations/deploying-migration-assistant/configuration-options.md index 560779ca9c..fede2eb8c0 100644 --- a/_migrations/deploying-migration-assistant/configuration-options.md +++ b/_migrations/deploying-migration-assistant/configuration-options.md @@ -28,9 +28,9 @@ Each migration configuration shares the following options. | Name | Example | Description | | :--- | :--- | :--- | -| `sourceClusterEndpoint` | `"https://source-cluster.elb.us-east-1.endpoint.com"` | The endpoint for the source cluster. | +| `sourceClusterEndpoint` | `"https://source-cluster.elb.us-east-1.endpoint.com"` | The endpoint for the source cluster. | | `targetClusterEndpoint` | `"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"` | The endpoint for the target cluster. Required if using an existing target cluster for the migration instead of creating a new one. | -| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC where the migration resources will be placed. The VPC must have at least two private subnets that span two availability zones. | +| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC where the migration resources will be placed. The VPC must have at least two private subnets that span two availability zones. | ## Backfill migration using RFS @@ -65,8 +65,7 @@ The following CDK performs a backfill migrations using RFS: Performing an RFS backfill migration requires an existing snapshot. -RFS configuration uses the following options. To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed. - +The RFS configuration uses the following options. All options are optional. | Name | Example | Description | | :--- | :--- | :--- | @@ -108,9 +107,6 @@ The following sample CDK performs a live capture migration with C&R Performing a live capture migration requires that a Capture Proxy be configured to capture incoming traffic and send it to the target cluster using the Traffic Replayer service. For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed. -
-Capture and Replay specific configuration options table - | Name | Example | Description | | :--- | :--- | :--- | @@ -120,13 +116,12 @@ Performing a live capture migration requires that a Capture Proxy be configured | `trafficReplayerExtraArgs` | `"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"` | Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the [Traffic Replayer](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). | -For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed. - +For arguments available in `captureProxyExtraArgs`, see the `@Parameter` fields [in the `CaptureProxy.java`](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, see the `@Parameter` fields [in TrafficReplayer.java](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). -## Cluster authentication pptions +## Cluster authentication options -Both the source and target cluster can use no authentication, such as authentication limited to VPC, basic authentication with a username and password, or AWS Signature Version 4 scoped to a user or role. +Both the source and target cluster can use no authentication, authentication limited to VPX, basic authentication with a username and password, or AWS Signature Version 4 scoped to a user or role. ### No authentication @@ -168,7 +163,7 @@ Both the source and target cluster can use no authentication, such as authentica The `serviceSigningName` can be `es` for an Elasticsearch or OpenSearch domain, or `aoss` for an OpenSearch Serverless collection. -All of these authentication mechanisms apply to both source and target clusters. +All of these authentication options apply to both source and target clusters. ## Network configuration From 2a66c395a4413ab6b9d5ee1278892811217b33e0 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 26 Nov 2024 14:39:48 -0600 Subject: [PATCH 4/6] Update _migrations/deploying-migration-assistant/configuration-options.md Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../deploying-migration-assistant/configuration-options.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_migrations/deploying-migration-assistant/configuration-options.md b/_migrations/deploying-migration-assistant/configuration-options.md index fede2eb8c0..5b7c03941e 100644 --- a/_migrations/deploying-migration-assistant/configuration-options.md +++ b/_migrations/deploying-migration-assistant/configuration-options.md @@ -16,7 +16,7 @@ This page outlines the configuration options for three key migrations scenarios: Each of these migrations depend on either a snapshot or a capture proxy. The following example `cdk.context.json `configurations that are used by AWS Cloud Deployment Kits (CDKs) to deploy and configure the Migration Assistant, shown as separate blocks for each migration type. If you are performing a migration that suits multiple scenarios, these options can be combined. -For a complete list of configuration options, see [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If there is a configuration option you would like that is not found on this page, open an issue in OpenSearch Migrations repository. +For a complete list of configuration options, see [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If you need a configuration option that is not found on this page, create an issue in the OpenSearch Migrations repository. {: .tip } Options for the source cluster endpoint, target cluster endpoint, and existing VPC should be configured for the Migration tools to function effectively. From 64ada079c7f326738fcb36cb3c0b5158aa27eae6 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 26 Nov 2024 14:41:52 -0600 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../configuration-options.md | 22 +++++++++---------- ...d-security-groups-for-existing-clusters.md | 8 +++---- .../deploying-migration-assistant/index.md | 2 +- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/_migrations/deploying-migration-assistant/configuration-options.md b/_migrations/deploying-migration-assistant/configuration-options.md index 5b7c03941e..7a682c1bb4 100644 --- a/_migrations/deploying-migration-assistant/configuration-options.md +++ b/_migrations/deploying-migration-assistant/configuration-options.md @@ -13,13 +13,13 @@ This page outlines the configuration options for three key migrations scenarios: 2. **Backfill migration with `Reindex-from-Snapshot` (RFS)** 3. **Live capture migration with Capture and Replay (C&R)** -Each of these migrations depend on either a snapshot or a capture proxy. The following example `cdk.context.json `configurations that are used by AWS Cloud Deployment Kits (CDKs) to deploy and configure the Migration Assistant, shown as separate blocks for each migration type. If you are performing a migration that suits multiple scenarios, these options can be combined. +Each of these migrations depends on either a snapshot or a capture proxy. The following example `cdk.context.json` configurations are used by AWS Cloud Development Kit (AWS CDK) to deploy and configure Migration Assistant for OpenSearch, shown as separate blocks for each migration type. If you are performing a migration applicable to multiple scenarios, these options can be combined. For a complete list of configuration options, see [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If you need a configuration option that is not found on this page, create an issue in the OpenSearch Migrations repository. {: .tip } -Options for the source cluster endpoint, target cluster endpoint, and existing VPC should be configured for the Migration tools to function effectively. +Options for the source cluster endpoint, target cluster endpoint, and existing virtual private cloud (VPC) should be configured in order for the migration tools to function effectively. ## Shared configuration options @@ -30,7 +30,7 @@ Each migration configuration shares the following options. | :--- | :--- | :--- | | `sourceClusterEndpoint` | `"https://source-cluster.elb.us-east-1.endpoint.com"` | The endpoint for the source cluster. | | `targetClusterEndpoint` | `"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"` | The endpoint for the target cluster. Required if using an existing target cluster for the migration instead of creating a new one. | -| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC where the migration resources will be placed. The VPC must have at least two private subnets that span two availability zones. | +| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC in which the migration resources will be stored. The VPC must have at least two private subnets that span two Availability Zones. | ## Backfill migration using RFS @@ -69,14 +69,14 @@ The RFS configuration uses the following options. All options are optional. | Name | Example | Description | | :--- | :--- | :--- | -| `reindexFromSnapshotServiceEnabled` | `true` | Enables deploying and configuring the RFS ECS service. | -| `reindexFromSnapshotExtraArgs` | `"--target-aws-region us-east-1 --target-aws-service-signing-name es"` | Extra arguments for the Document Migration command, with space separation. See the [RFS Extra Arguments](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments) for more details. You can pass `--no-insecure` to remove the `--insecure` flag. | +| `reindexFromSnapshotServiceEnabled` | `true` | Enables deployment and configuration of the RFS ECS service. | +| `reindexFromSnapshotExtraArgs` | `"--target-aws-region us-east-1 --target-aws-service-signing-name es"` | Extra arguments for the Document Migration command, with space separation. See [RFS Extra Arguments](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments) for more information. You can pass `--no-insecure` to remove the `--insecure` flag. | To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed. ## Live capture migration with C&R -The following sample CDK performs a live capture migration with C&R +The following sample CDK performs a live capture migration with C&R: ```json { @@ -110,18 +110,18 @@ Performing a live capture migration requires that a Capture Proxy be configured | Name | Example | Description | | :--- | :--- | :--- | -| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment using a CloudFormation stack. | +| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment using an AWS CloudFormation stack. | | `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). | -| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment using a CloudFormation stack. | +| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment using a CloudFormation stack. | | `trafficReplayerExtraArgs` | `"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"` | Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the [Traffic Replayer](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). | -For arguments available in `captureProxyExtraArgs`, see the `@Parameter` fields [in the `CaptureProxy.java`](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, see the `@Parameter` fields [in TrafficReplayer.java](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). +For arguments available in `captureProxyExtraArgs`, see the `@Parameter` fields in [`CaptureProxy.java`](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, see the `@Parameter` fields in [TrafficReplayer.java](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). ## Cluster authentication options -Both the source and target cluster can use no authentication, authentication limited to VPX, basic authentication with a username and password, or AWS Signature Version 4 scoped to a user or role. +Both the source and target cluster can use no authentication, authentication limited to VPC, basic authentication with a username and password, or AWS Signature Version 4 scoped to a user or role. ### No authentication @@ -147,7 +147,7 @@ Both the source and target cluster can use no authentication, authentication lim } ``` -### SigV4 authentication +### Signature Version 4 authentication ``` "sourceCluster": { diff --git a/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md index 0d9af94db2..a5cebac1cc 100644 --- a/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md +++ b/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md @@ -9,21 +9,21 @@ parent: Deploying migration assistant This page outlines security scenarios for using the migration tools with existing clusters, including any necessary configuration changes to ensure proper communication between them. -## Importing an OpenSearch Service or OpenSearch Serverless target cluster +## Importing an Amazon OpenSearch Service or Amazon OpenSearch Serverless target cluster Use the following scenarios for OpenSearch Service or Serverless target clusters: -### OpenSearch service +### OpenSearch Service For an OpenSearch Domain, two main configurations are typically required to ensure proper functioning of the migration solution: 1. **Security Group Configuration** - The Domain should have a security group that allows communication from the applicable Migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`). The CDK automatically creates an `osClusterAccessSG` security group, which is applied to the migration services. The user should then add this security group to their existing Domain to allow access. + The domain should have a security group that allows communication from the applicable migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`). The CDK automatically creates an `osClusterAccessSG` security group, which is applied to the migration services. The user should then add this security group to their existing domain to allow access. 2. **Access Policy Configuration** should be one of the following: - An open access policy that allows all access. - - Configured to allow at least the IAM task roles for the applicable Migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`) to access the Domain. + - Configured to allow at least the AWS Identity and Access Management (IAM) task roles for the applicable migration services (Traffic Replayer, Migration Console, `Reindex-from-Snapshot`) to access the domain. ### OpenSearch Serverless diff --git a/_migrations/deploying-migration-assistant/index.md b/_migrations/deploying-migration-assistant/index.md index a2f645a0d8..6e245aa5da 100644 --- a/_migrations/deploying-migration-assistant/index.md +++ b/_migrations/deploying-migration-assistant/index.md @@ -6,4 +6,4 @@ nav_order: 10 # Deploying Migration Assistant -This section details what options are available for deploying Migration Assistant. +This section provides information about the available options for deploying Migration Assistant. From 6047c62a7af54ef0e884348dcbbdeccb959d5a72 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 26 Nov 2024 14:44:23 -0600 Subject: [PATCH 6/6] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../deploying-migration-assistant/configuration-options.md | 2 +- .../iam-and-security-groups-for-existing-clusters.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/_migrations/deploying-migration-assistant/configuration-options.md b/_migrations/deploying-migration-assistant/configuration-options.md index 7a682c1bb4..2e7f43e1b5 100644 --- a/_migrations/deploying-migration-assistant/configuration-options.md +++ b/_migrations/deploying-migration-assistant/configuration-options.md @@ -16,7 +16,7 @@ This page outlines the configuration options for three key migrations scenarios: Each of these migrations depends on either a snapshot or a capture proxy. The following example `cdk.context.json` configurations are used by AWS Cloud Development Kit (AWS CDK) to deploy and configure Migration Assistant for OpenSearch, shown as separate blocks for each migration type. If you are performing a migration applicable to multiple scenarios, these options can be combined. -For a complete list of configuration options, see [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If you need a configuration option that is not found on this page, create an issue in the OpenSearch Migrations repository. +For a complete list of configuration options, see [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If you need a configuration option that is not found on this page, create an issue in the [OpenSearch Migrations repository](https://github.com/opensearch-project/opensearch-migrations/issues). {: .tip } Options for the source cluster endpoint, target cluster endpoint, and existing virtual private cloud (VPC) should be configured in order for the migration tools to function effectively. diff --git a/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md b/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md index a5cebac1cc..808de79689 100644 --- a/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md +++ b/_migrations/deploying-migration-assistant/iam-and-security-groups-for-existing-clusters.md @@ -11,7 +11,7 @@ This page outlines security scenarios for using the migration tools with existin ## Importing an Amazon OpenSearch Service or Amazon OpenSearch Serverless target cluster -Use the following scenarios for OpenSearch Service or Serverless target clusters: +Use the following scenarios for Amazon OpenSearch Service or Amazon OpenSearch Serverless target clusters. ### OpenSearch Service