Skip to content

Commit

Permalink
Add migrations section.
Browse files Browse the repository at this point in the history
Signed-off-by: Archer <[email protected]>
  • Loading branch information
Naarcha-AWS committed Dec 4, 2024
1 parent 8017292 commit 8e45ac5
Show file tree
Hide file tree
Showing 28 changed files with 93 additions and 711 deletions.
23 changes: 15 additions & 8 deletions _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,6 @@ collections:
install-and-configure:
permalink: /:collection/:path/
output: true
upgrade-to:
permalink: /:collection/:path/
output: true
im-plugin:
permalink: /:collection/:path/
output: true
Expand Down Expand Up @@ -94,6 +91,9 @@ collections:
data-prepper:
permalink: /:collection/:path/
output: true
migration-assistant:
permalink: /:collection/:path/
output: true
tools:
permalink: /:collection/:path/
output: true
Expand Down Expand Up @@ -137,11 +137,6 @@ opensearch_collection:
install-and-configure:
name: Install and upgrade
nav_fold: true
upgrade-to:
name: Migrate to OpenSearch
# nav_exclude: true
nav_fold: true
# search_exclude: true
im-plugin:
name: Managing Indexes
nav_fold: true
Expand Down Expand Up @@ -213,6 +208,12 @@ clients_collection:
name: Clients
nav_fold: true

migration_assistant_collection:
collections:
migration-assistant:
name: Migration Assistant
nav_fold: true

benchmark_collection:
collections:
benchmark:
Expand Down Expand Up @@ -252,6 +253,12 @@ defaults:
values:
section: "benchmark"
section-name: "Benchmark"
-
scope:
path: "_migration-assistant"
values:
section: "migration-assistant"
section-name: "Migration Assistant"

# Enable or disable the site search
# By default, just-the-docs enables its JSON file-based search. We also have an OpenSearch-driven search functionality.
Expand Down
2 changes: 2 additions & 0 deletions _data/top_nav.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ items:
url: /docs/latest/clients/
- label: Benchmark
url: /docs/latest/benchmark/
- label: Migration Assistant
url: /docs/latest/migration-assistant/
- label: Platform
url: /platform/index.html
children:
Expand Down
8 changes: 7 additions & 1 deletion _includes/cards.html
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,14 @@
<p class="description">Measure performance metrics for your OpenSearch cluster</p>
<p class="last-link">Documentation &#x2192;</p>
</div>

<div class="card">
<a href="{{site.url}}/docs/latest/migration-assistant/" class='card-link'></a>
<p class="heading">Migration Assistant</p>
<p class="description">Migrate to OpenSearch from other platforms</p>
<p class="last-link">Documentation &#x2192;</p>
</div>
</div>

</div>


2 changes: 2 additions & 0 deletions _layouts/default.html
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@
{% assign section = site.clients_collection.collections %}
{% elsif page.section == "benchmark" %}
{% assign section = site.benchmark_collection.collections %}
{% elsif page.section == "migration-assistant" %}
{% assign section = site.migration_assistant_collection.collections %}
{% endif %}

{% if section %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,11 @@ nav_order: 10

This quickstart outlines how to deploy Migration Assistant for OpenSearch and execute an existing data migration using `Reindex-from-Snapshot` (RFS). It uses AWS for illustrative purposes. However, the steps can be modified for use with other cloud providers.


## Prerequisites and assumptions

Before using this quickstart, make sure you fulfill the following prerequisites:

* Verify that your migration path [is supported](https://opensearch.org/docs/latest/migrations/is-migration-assistant-right-for-you/#supported-migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported.
* Verify that your migration path [is supported]({{site.url}}{{site.baseurl}}/migration-assistant/is-migration-assistant-right-for-you/#migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported.
* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin.
* The target cluster must be deployed.

Expand Down Expand Up @@ -253,7 +252,7 @@ Run the following command to migrate metadata:
console metadata migrate [...]
```

For more information, see [Migrating metadata]({{site.url}}{{site.baseurl}}/migrations/migration-phases/migrating-metadata/).
For more information, see [Migrating metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/).

---

Expand Down Expand Up @@ -285,7 +284,7 @@ You can now use RFS to migrate documents from your original cluster:
console backfill stop
```

For more information, see [Backfill]({{site.url}}{{site.baseurl}}/migrations/migration-phases/backfill/).
For more information, see [Backfill]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/).

---

Expand Down
5 changes: 4 additions & 1 deletion _migrations/index.md → _migration-assistant/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ nav_order: 1
has_children: false
nav_exclude: true
has_toc: false
permalink: /migration-assistant/
redirect_from:
- /migration-assistant/index/
---

# Migration Assistant for OpenSearch
Expand Down Expand Up @@ -53,7 +56,7 @@ The Metadata migration tool integrated into the Migration CLI can be used indepe

The destination cluster for migration or comparison in an A/B test.

### Architecture overview
## Architecture overview

The Migration Assistant architecture is based on the use of an AWS Cloud infrastructure, but most tools are designed to be cloud independent. A local containerized version of this solution is also available.

Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
layout: default
title: Command reference
nav_order: 35
nav_order: 40
parent: Migration console
---

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
layout: default
title: Assessing your cluster for migration
nav_order: 60
parent: Migration phases
---

# Assessing your cluster for migration


The goal of the Migration Assistant is to streamline the process of migrating from one location or version of Elasticsearch/OpenSearch to another. However, completing a migration sometimes requires resolving client compatibility issues before they can communicate directly with the target cluster.

## Understanding breaking changes

Before performing any upgrade or migration, you should review any documentation of breaking changes. Even if the cluster is migrated there might be changes required for clients to connect to the new cluster

Check failure on line 15 in _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md#L15

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'changes. Even'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'changes. Even'.", "location": {"path": "_migration-assistant/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 15, "column": 93}}}, "severity": "ERROR"}

## Upgrade and breaking changes guides

For migrations paths between Elasticsearch 6.8 and OpenSearch 2.x users should be familiar with documentation in the links below that apply to their specific case:

Check warning on line 19 in _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md#L19

[OpenSearch.DirectionAboveBelow] Use 'following or later' instead of 'below' for versions or orientation within a document. Use 'above' and 'below' only for physical space or screen descriptions.
Raw output
{"message": "[OpenSearch.DirectionAboveBelow] Use 'following or later' instead of 'below' for versions or orientation within a document. Use 'above' and 'below' only for physical space or screen descriptions.", "location": {"path": "_migration-assistant/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 19, "column": 124}}}, "severity": "WARNING"}

* [Upgrading Amazon Service Domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html)

* [Changes from Elasticsearch to OpenSearch fork](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html)

* [OpenSearch Breaking Changes](https://opensearch.org/docs/latest/breaking-changes/)

The next step is to set up a proper test bed to verify that your applications will work as expected on the target version.

## Data Transformations and Client Impact

Check failure on line 29 in _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md#L29

[OpenSearch.HeadingCapitalization] 'Data Transformations and Client Impact' is a heading and should be in sentence case.
Raw output
{"message": "[OpenSearch.HeadingCapitalization] 'Data Transformations and Client Impact' is a heading and should be in sentence case.", "location": {"path": "_migration-assistant/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 29, "column": 4}}}, "severity": "ERROR"}

Any time you apply a transformation to your data, such as:

- Changing index names
- Modifying field names or field mappings
- Splitting indices with type mappings

Check failure on line 35 in _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md#L35

[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.
Raw output
{"message": "[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.", "location": {"path": "_migration-assistant/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 35, "column": 13}}}, "severity": "ERROR"}

These changes might need to be reflected in your client configurations. For example, if your clients are reliant on specific index or field names, you must ensure that their queries are updated accordingly.

We recommend running production-like queries against the target cluster before switching over actual production traffic. This helps verify that the client can:

- Communicate with the target cluster
- Locate the necessary indices and fields

Check failure on line 42 in _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migration-assistant/migration-phases/assessing-your-cluster-for-migration.md#L42

[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.
Raw output
{"message": "[OpenSearch.SubstitutionsError] Use 'indexes' instead of 'indices'.", "location": {"path": "_migration-assistant/migration-phases/assessing-your-cluster-for-migration.md", "range": {"start": {"line": 42, "column": 24}}}, "severity": "ERROR"}
- Retrieve the expected results

For complex migrations involving multiple transformations or breaking changes, we highly recommend performing a trial migration with representative, non-production data (e.g., in a staging environment) to fully test client compatibility with the target cluster.



Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ parent: Migration phases

# Backfill

After the [metadata]({{site.url}}{{site.baseurl}}/migrations/migration-phases/migrating-metadata/) for your cluster has been migrated, you can use capture proxy data replication and snapshots to backfill your data into the next cluster.
After the [metadata]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/) for your cluster has been migrated, you can use capture proxy data replication and snapshots to backfill your data into the next cluster.

## Capture proxy data replication

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ has_children: true

This page details how to conduct a migration with Migration Assistant. It encompasses a variety of scenarios including:

- [**Metadata migration**]({{site.url}}{{site.baseurl}}/migrations/migration-phases/migrating-metadata/): Migrating cluster metadata, such as index settings, aliases, and templates.
- [**Backfill migration**]({{site.url}}{{site.baseurl}}/migrations/migration-phases/backfill/): Migrating existing or historical data from a source to a target cluster.
- [**Metadata migration**]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrating-metadata/): Migrating cluster metadata, such as index settings, aliases, and templates.
- [**Backfill migration**]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/): Migrating existing or historical data from a source to a target cluster.
- **Live traffic migration**: Replicating live ongoing traffic from a source to a target cluster.

Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ After the source and target clusters are synchronized, traffic needs to be switc

This page assumes that the following has occurred before making the switch:

- All client traffic is being routed through a switchover listener in the [MigrationAssistant Application Load Balancer]({{site.url}}{{site.baseurl}}/migrations/migration-phases/backfill/).
- All client traffic is being routed through a switchover listener in the [MigrationAssistant Application Load Balancer]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/backfill/).
- Client traffic has been verified as compatible with the target cluster.
- The target cluster is in a good state to accept client traffic.
- The target proxy service is deployed.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ For example, if a document was deleted after a snapshot was taken, starting Traf

## Configuration options

[Traffic Replayer settings]({{site.url}}{{site.baseurl}}/migrations/deploying-migration-assisstant/configuation-options/) are configured during the deployment of Migration Assistant. Make sure to set the authentication mode for Traffic Replayer so that it can properly communicate with the target cluster. For more information about different types of traffic that are handled by Traffic Replayer, see [limitations](#limitations).
[Traffic Replayer settings]({{site.url}}{{site.baseurl}}/migration-assistant/deploying-migration-assistant/configuration-options/) are configured during the deployment of Migration Assistant. Make sure to set the authentication mode for Traffic Replayer so that it can properly communicate with the target cluster.

Check failure on line 20 in _migration-assistant/migration-phases/using-traffic-replayer.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migration-assistant/migration-phases/using-traffic-replayer.md#L20

[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migration-assistant/migration-phases/using-traffic-replayer.md", "range": {"start": {"line": 20, "column": 10}}}, "severity": "ERROR"}

Check failure on line 20 in _migration-assistant/migration-phases/using-traffic-replayer.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migration-assistant/migration-phases/using-traffic-replayer.md#L20

[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migration-assistant/migration-phases/using-traffic-replayer.md", "range": {"start": {"line": 20, "column": 246}}}, "severity": "ERROR"}

## Using Traffic Replayer

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ Look for failing tasks by navigating to **Traffic Capture Proxy ECS**. Change **

After all verifications are complete, reset all resources before using Migration Assistant for an actual migration.

The following steps outline how to reset resources with Migration Assistant before executing the actual migration. At this point all verifications are expected to have been completed. These steps can be performed after [Accessing the Migration Console]({{site.url}}{{site.baseurl}}/migrations/migration-console/accessing-the-migration-console/).
The following steps outline how to reset resources with Migration Assistant before executing the actual migration. At this point all verifications are expected to have been completed. These steps can be performed after [Accessing the Migration Console]({{site.url}}{{site.baseurl}}/migration-assistant/migration-console/accessing-the-migration-console/).

### Traffic Replayer

Expand Down

This file was deleted.

Loading

0 comments on commit 8e45ac5

Please sign in to comment.