Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add migration phases pages #8828

Merged
merged 31 commits into from
Dec 3, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
eaeb2a6
Add first three migration phases pages
Naarcha-AWS Nov 29, 2024
4df4a83
Apply suggestions from code review
Naarcha-AWS Dec 2, 2024
bab54e7
Add backfill page.
Naarcha-AWS Dec 2, 2024
2b843e0
Merge branch 'main' into migration-phases
Naarcha-AWS Dec 2, 2024
85f0a7e
Update backfill.md
Naarcha-AWS Dec 2, 2024
4d36814
Add replayer page.
Naarcha-AWS Dec 2, 2024
e881150
Fix grammar.
Naarcha-AWS Dec 2, 2024
c9dd5f2
Add final migration phases page.
Naarcha-AWS Dec 2, 2024
7ddd500
Apply suggestions from code review
Naarcha-AWS Dec 2, 2024
2cc8061
Apply suggestions from code review
Naarcha-AWS Dec 2, 2024
5ee6b78
Update _migrations/migration-phases/verifying-tools-for-migration.md
Naarcha-AWS Dec 2, 2024
5a5d16b
Update _migrations/migration-phases/backfill.md
Naarcha-AWS Dec 2, 2024
dfa7f5a
Add migration phase links
Naarcha-AWS Dec 2, 2024
9e8fef9
Edit migration console section
Naarcha-AWS Dec 2, 2024
d4e810b
Apply suggestions from code review
Naarcha-AWS Dec 2, 2024
ae5b540
Merge branch 'main' into migration-phases
Naarcha-AWS Dec 2, 2024
79ad135
Update migrating-metadata.md
Naarcha-AWS Dec 2, 2024
16aa4d8
Apply suggestions from code review
Naarcha-AWS Dec 2, 2024
424cc7c
Rename traffic replacer.
Naarcha-AWS Dec 3, 2024
6453650
Update _migrations/migration-phases/verifying-tools-for-migration.md
Naarcha-AWS Dec 3, 2024
624e41b
Add sentence about live traffic capture
Naarcha-AWS Dec 3, 2024
6078146
Apply suggestions from code review
Naarcha-AWS Dec 3, 2024
6faded5
Update _migrations/migration-phases/backfill.md
Naarcha-AWS Dec 3, 2024
d74ea60
Update _migrations/migration-phases/verifying-tools-for-migration.md
Naarcha-AWS Dec 3, 2024
30a609a
Update _migrations/migration-phases/verifying-tools-for-migration.md
Naarcha-AWS Dec 3, 2024
3825804
Add editorial review.
Naarcha-AWS Dec 3, 2024
5c49297
Additional editorial comments.
Naarcha-AWS Dec 3, 2024
e7a6244
Editorial for infra and traffic
Naarcha-AWS Dec 3, 2024
9463187
Editorial comments for using traffic replayer.
Naarcha-AWS Dec 3, 2024
ca4ba38
Add final editorial comments.
Naarcha-AWS Dec 3, 2024
ecaafe4
Apply suggestions from code review
Naarcha-AWS Dec 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions _migrations/getting-started-data-migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ To deploy Migration Assistant, use the following steps:
These commands deploy the following stacks:

* Migration Assistant network stack
* Reindex From Snapshot stack
* `Reindex-from-snapshot` stack
* Migration console stack

---
Expand Down Expand Up @@ -253,7 +253,7 @@ Run the following command to migrate metadata:
console metadata migrate [...]
```

For more information, see [Metadata migration].
For more information, see [Migrating metadata]({{site.url}}{{site.baseurl}}/migrations/migration-phases/migrating-metadata/).

---

Expand Down Expand Up @@ -285,7 +285,7 @@ You can now use RFS to migrate documents from your original cluster:
console backfill stop
```

For more information, see [Backfill execution].
For more information, see [Backfill]({{site.url}}{{site.baseurl}}/migrations/migration-phases/backfill/).

---

Expand Down Expand Up @@ -328,4 +328,3 @@ fields @message

If any failed documents are identified, you can index the failed documents directly as opposed to using RFS.

For more information, see [Backfill migration].
29 changes: 11 additions & 18 deletions _migrations/migration-console/accessing-the-migration-console.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
---
layout: default
title: Accessing the migration console
nav_order: 35
parent: Migration console
---

# Accessing the migration console

The bootstrap box deployed through Migration Assistant contains a script that simplifies access to the migration console through that instance.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

The Migrations Assistant deployment includes an ECS task that hosts tools to run different phases of the migration and check the progress or results of the migration.

## SSH into the Migration Console
Following the AWS Solutions deployment, the bootstrap box contains a script that simplifies access to the migration console through that instance.

To access the Migration Console, use the following commands:
To access the migration console, use the following commands:

```shell
export STAGE=dev
Expand All @@ -16,13 +19,7 @@ export AWS_REGION=us-west-2

When opening the console a message will appear above the command prompt, `Welcome to the Migration Assistant Console`.

<details>

<summary>
<b>SSH from any machine into Migration Console</b>
</summary>

On a machine with the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) ↗ and the [AWS Session Manager Plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) ↗, you can directly connect to the migration console. Ensure you've run `aws configure` with credentials that have access to the environment.
On a machine with the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and the [AWS Session Manager Plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html), you can directly connect to the migration console. Ensure you've run `aws configure` with credentials that have access to the environment.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

Use the following commands:

Expand All @@ -32,10 +29,6 @@ export SERVICE_NAME=migration-console
export TASK_ARN=$(aws ecs list-tasks --cluster migration-${STAGE}-ecs-cluster --family "migration-${STAGE}-${SERVICE_NAME}" | jq --raw-output '.taskArns[0]')
aws ecs execute-command --cluster "migration-${STAGE}-ecs-cluster" --task "${TASK_ARN}" --container "${SERVICE_NAME}" --interactive --command "/bin/bash"
```
</details>

## Troubleshooting

### Deployment Stage

Typically, `STAGE` is `dev`, but this may vary based on what the user specified during deployment.
Typically, `STAGE` is equivalent to a standard `dev` environment, but this may vary based on what the user specified during deployment.
7 changes: 6 additions & 1 deletion _migrations/migration-console/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,9 @@ layout: default
title: Migration console
nav_order: 30
has_children: true
---
---

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
The Migrations Assistant deployment includes an ECS task that hosts tools to run different phases of the migration and check the progress or results of the migration. This ECS task is called the **Migration Console**. The migration console is a command line interface to interact with the deployed components of the solution.

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
This section details how to access the migration console and what commands are supported.

Original file line number Diff line number Diff line change
@@ -1,83 +1,106 @@
---
layout: default
title: Command reference
nav_order: 35
parent: Migration console
---


# Migration console command reference

The Migration Assistant Console is a command line interface to interact with the deployed components of the solution.
Migration console commands follow this syntax: `console [component] [action]`. The components include `clusters`, `backfill`, `snapshot`, `metadata`, and `replay`. The console is configured with a registry of the deployed services and the source and target cluster, generated from the `cdk.context.json` values.

The commands are in the form of `console [component] [action]`. The components include `clusters`, `backfill` (e.g the Reindex from Snapshot service), `snapshot`, `metadata`, `replay`, etc. The console is configured with a registry of the deployed services and the source and target cluster, generated from the `cdk.context.json` values.

## Commonly Used Commands
## Commonly used commands

The exact commands used will depend heavily on use-case and goals, but the following are a series of common commands with a quick description of what they do.

### Check connection

Reports whether both the source and target clusters can be reached and their versions.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
```sh
console clusters connection-check
```
Reports whether both the source and target clusters can be reached and their versions.

### Run `cat-indices`

Runs the `cat-indices` API on the cluster.

```sh
console clusters cat-indices
```
Runs the `_cat/indices` command on each cluster and prints the results.

***
### Create a snapshot

Creates a snapshot on the source cluster, into a preconfigured Amazon S3 bucket.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
```sh
console snapshot create
```
Initiates creating a snapshot on the source cluster, into a pre-configured S3 bucket.

## Check snapshot status

Runs a detailed check on the status of the snapshot creation, including estimated completion time.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```sh
console snapshot status --deep-check
```
Runs a detailed check on the status of the snapshot creation, including estimated completion time.

***
## Evaluate metadata

Performs a dry run of metadata migration, showing which indexes, templates, and other objects will be migrated to the target cluster.

```sh
console metadata evaluate
```
Perform a dry run of metadata migration, showing which indices, templates, and other objects will be migrated to the target cluster.

## Migrate metadata

Migrates the metadata from the source cluster to the target cluster.

```sh
console metadata migrate
```
Perform an actual metadata migration.

***
## Start a backfill

```sh
console backfill start
```
If the Reindex From Snapshot service is enabled, start an instance of the service to begin moving documents to the target cluster.
If the `Reindex-From-Snapshot` (RFS) is enabled, starts an instance of the service to begin moving documents to the target cluster.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
There are similar `scale UNITS` and `stop` commands to change the number of active instances for RFS.


```sh
console backfill status --deep-check
console backfill start
```
See the current status of the backfill migration, with the number of instances operating and the progress of the shards.

***
## Check backfill status

Gets the current status of the backfill migration, with the number of instances operating and the progress of the shards.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved


## Start Traffic Replayer

Check failure on line 81 in _migrations/migration-console/migration-console-commands-references.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-console/migration-console-commands-references.md#L81

[OpenSearch.HeadingCapitalization] 'Start Traffic Replayer' is a heading and should be in sentence case.
Raw output
{"message": "[OpenSearch.HeadingCapitalization] 'Start Traffic Replayer' is a heading and should be in sentence case.", "location": {"path": "_migrations/migration-console/migration-console-commands-references.md", "range": {"start": {"line": 81, "column": 4}}}, "severity": "ERROR"}

Check failure on line 81 in _migrations/migration-console/migration-console-commands-references.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-console/migration-console-commands-references.md#L81

[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migrations/migration-console/migration-console-commands-references.md", "range": {"start": {"line": 81, "column": 18}}}, "severity": "ERROR"}

If the Traffic Replayer is enabled, starts an instance of the service to begin replaying traffic against the target cluster.

Check failure on line 83 in _migrations/migration-console/migration-console-commands-references.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-console/migration-console-commands-references.md#L83

[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migrations/migration-console/migration-console-commands-references.md", "range": {"start": {"line": 83, "column": 16}}}, "severity": "ERROR"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
The `stop` command stops all active instances.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```sh
console replay start
```
If the Traffic Replayer service is enabled, start an instance of the service to begin replaying traffic against the target cluster.
The `stop` command stops all active instances.

***
## Read logs

Reads any logs that exist when running Traffic Replayer. Use tab completion on the path to fill in the available `NODE_IDs` and, if applicable, log file names. The tuples logs roll over at a certain size threshold, so there may be many files named with timestamps. The `jq` command pretty-prints each line of the tuple output before writing it to file.

Check failure on line 92 in _migrations/migration-console/migration-console-commands-references.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/migration-console/migration-console-commands-references.md#L92

[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migrations/migration-console/migration-console-commands-references.md", "range": {"start": {"line": 92, "column": 48}}}, "severity": "ERROR"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```sh
console tuples show --in /shared-logs-output/traffic-replayer-default/[NODE_ID]/tuples/console.log | jq > readable_tuples.json
```
Use tab completion on the path to fill in the available node ids and, if applicable, log file names. The tuples logs roll over at a certain size threshold, so there may be many files named with timestamps. The `jq` command pretty-prints each line of the tuple output before writing it to file.


## Command Reference
All commands and options can be explored within the tool itself by using the `--help` option, either for the entire `console` application or for individual components (e.g. `console backfill --help`). The console also has command autocomplete set up to assist with usage.
## Help command

```
All commands and options can be explored within the tool itself by using the `--help` option, either for the entire `console` application or for individual components (e.g. `console backfill --help`). The console also has command autocomplete set up to assist with usage. For example:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
```bash
$ console --help
Usage: console [OPTIONS] COMMAND [ARGS]...

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
layout: default
title: Assessing your cluster for migration
nav_order: 60
has_children: true
parent: Migration phases
---

# Assessing your cluster for migration


Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
The goal of the Migration Assistant is to streamline the process of migrating from one location or version of Elasticsearch/OpenSearch to another. However, completing a migration sometimes requires resolving client compatibility issues before they can communicate directly with the target cluster.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

## Understanding breaking changes

Before performing any upgrade or migration, you should review any documentation of breaking changes. Even if the cluster is migrated there might be changes required for clients to connect to the new cluster.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

## Upgrade and breaking changes guides

For migrations paths between Elasticsearch 6.8 and OpenSearch 2.x users should be familiar with following documentation based on their specific use case:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

* [Upgrading Amazon Service Domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) ↗
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

* [Changes from Elasticsearch to OpenSearch fork](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html) ↗
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

* [OpenSearch Breaking Changes](https://opensearch.org/docs/latest/breaking-changes/) ↗
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

The next step is to set up a proper test bed to verify that your applications will work as expected on the target version.

## Impact of data transformations

Any time you apply a transformation to your data, such as:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

- Changing index names
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- Modifying field names or field mappings
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- Splitting indexes with type mappings
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

These changes might need to be reflected in your client configurations. For example, if your clients are reliant on specific index or field names, you must ensure that their queries are updated accordingly.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

We recommend running production-like queries against the target cluster before switching over actual production traffic. This helps verify that the client can:
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

- Communicate with the target cluster
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- Locate the necessary indexes and fields
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- Retrieve the expected results
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

For complex migrations involving multiple transformations or breaking changes, we highly recommend performing a trial migration with representative, non-production data (e.g., in a staging environment) to fully test client compatibility with the target cluster.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved



7 changes: 0 additions & 7 deletions _migrations/migration-phases/assessment/index.md

This file was deleted.

This file was deleted.

This file was deleted.

Loading
Loading