Introduce interface changes to support read/write blob with object metadata #13023

skumawat2025 · 2024-04-02T13:56:02Z

Description

This PR is a sub task of feature request #13022. And is being tracked here #13091.

This PR consists related changes for blob container interface to read/write blob object with object metadata. This will allow us to upload/download translog files to remote store with it's object metadata.

Related Issues

Resolves #13092

Check List

~~New functionality includes testing.~~
- ~~All tests pass~~
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
~~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~~
~~Public documentation issue/PR created~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-04-02T14:05:48Z

❌ Gradle check result for 04aa874: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-04-02T14:18:30Z

Compatibility status:

Checks if related components are compatible with change 75fa239

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2024-04-02T14:53:09Z

❌ Gradle check result for 08c9432: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-04-02T21:10:19Z

❌ Gradle check result for 950e55b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

skumawat2025 · 2024-04-03T04:44:55Z

❌ Gradle check result for 950e55b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Failing Tests-

org.opensearch.backwards.IndexingIT.testIndexingWithReplicaOnBwcNodes
org.opensearch.backwards.IndexingIT.testIndexingWithPrimaryOnBwcNodes
org.opensearch.backwards.IndexingIT.testIndexVersionPropagation
org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.indices/20_hidden/Test cat indices output for hidden index}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fielddata fields - pattern}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fields - multi metric}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=nodes.stats/20_response_filtering/Nodes Stats filtered using both includes and excludes filters}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/14_groups/Groups - one}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fields - star}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Completion fields - star}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Completion - pattern}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=nodes.stats/11_indices_metrics/Metric - _all include_segment_file_sizes}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fielddata fields - star}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fields - blank}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.indices/20_hidden/Test cat indices output for dot-hidden index and dot-prefixed pattern}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=nodes.stats/20_response_filtering/Nodes Stats with response filtering}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.indices/20_hidden/Test cat indices output with a hidden index with a hidden alias}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.indices/20_hidden/Test cat indices output with a hidden index with a visible alias}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.shards/10_basic/Test cat shards sort}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fields - pattern}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.nodes/10_basic/Test cat nodes output}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Completion fields - one}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/40_updates_on_refresh/Completion stats}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Completion - all metric}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.nodes/10_basic/Additional disk information}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fielddata fields - all metric}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=nodes.stats/30_discovery/Discovery stats}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fields - _all metric}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fields - multi}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.fielddata/10_basic/Test cat fielddata output}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/14_groups/Groups - blank}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/14_groups/Groups - pattern}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Completion - multi metric}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.indices/20_hidden/Test cat indices output with a hidden index, dot-hidden alias and dot pattern}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fielddata fields - multi metric}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fielddata fields - multi}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Completion fields - multi}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/12_level/Level - indices}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fielddata fields - one}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.indices/10_basic/Test cat indices output}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.shards/10_basic/Help from 2.12.0 to 2.99.99}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=nodes.stats/11_indices_metrics/Metric - indices _all}
org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fields - one}

org.opensearch.backwards.IndexingIT is flaky. See #9407
qa:mixed-cluster:v2.14.0#mixedClusterTest is flaky. See #12980

… flow Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-04-03T10:38:41Z

❌ Gradle check result for 85711ae: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-04-05T19:47:30Z

✅ Gradle check result for e39ef16: SUCCESS

github-actions · 2024-04-08T05:06:06Z

❌ Gradle check result for a3a5de7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-04-08T06:00:10Z

✅ Gradle check result for 75fa239: SUCCESS

Bukhtawar · 2024-04-08T06:16:42Z

server/src/main/java/org/opensearch/common/blobstore/BlobDownloadResponse.java

+ *
+ * @opensearch.experimental
+ */
+public class BlobDownloadResponse {


The name is too generic and doesn't bring out the purpose of encapsulating metadata

…tadata (opensearch-project#13023) * Introduce interface changes to read/write blob with object metadata --------- Signed-off-by: Sandeep Kumawat <[email protected]> Co-authored-by: Sandeep Kumawat <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

opensearch-trigger-bot · 2024-05-20T11:34:08Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-13023-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9b0f578903f2cd56878b1c69ab6f4f05167238f8
# Push it to GitHub
git push --set-upstream origin backport/backport-13023-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-13023-to-2.x.

…tadata (opensearch-project#13023) * Introduce interface changes to read/write blob with object metadata --------- Signed-off-by: Sandeep Kumawat <[email protected]> Co-authored-by: Sandeep Kumawat <[email protected]> (cherry picked from commit 9b0f578) Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 · 2024-05-21T10:49:34Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-13023-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9b0f578903f2cd56878b1c69ab6f4f05167238f8
# Push it to GitHub
git push --set-upstream origin backport/backport-13023-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-13023-to-2.x.

Manual Backport PR: #13757

…tadata (#13023) (#13757) * Introduce interface changes to read/write blob with object metadata --------- Signed-off-by: Sandeep Kumawat <[email protected]>

Introduce interface changes to read/write blob with object metadata

08c9432

Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 force-pushed the interface-changes branch from 04aa874 to 08c9432 Compare April 2, 2024 14:17

empty commit

950e55b

Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 changed the title ~~Introduce related interface changes to read/write blob with object metadata~~ Introduce interface changes to support read/write blob with object metadata Apr 3, 2024

provide metadata in remoteTransferContainer for async translog upload…

85711ae

… flow Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 added the skip-changelog label Apr 3, 2024

skumawat2025 marked this pull request as ready for review April 3, 2024 11:03

skumawat2025 requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta and Rishikesh1159 as code owners April 3, 2024 11:03

addres comments

e39ef16

Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 force-pushed the interface-changes branch from d2ed0a3 to e39ef16 Compare April 5, 2024 18:52

skumawat2025 requested a review from sachinpkale April 5, 2024 19:53

resolve comments

75fa239

Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 force-pushed the interface-changes branch from a3a5de7 to 75fa239 Compare April 8, 2024 05:10

sachinpkale approved these changes Apr 8, 2024

View reviewed changes

sachinpkale merged commit 9b0f578 into opensearch-project:main Apr 8, 2024
31 checks passed

Bukhtawar reviewed Apr 8, 2024

View reviewed changes

skumawat2025 mentioned this pull request Apr 8, 2024

Implement interface changes for s3 plugin to read/write blob with obj… #13113

Merged

8 tasks

sachinpkale added the backport 2.x Backport to 2.x branch label May 20, 2024

opensearch-trigger-bot bot added the backport-failed label May 20, 2024

skumawat2025 mentioned this pull request May 20, 2024

Introduce interface changes to support read/write blob with object me… #13757

Merged

8 tasks

skumawat2025 added v2.15.0 Issues and PRs related to version 2.15.0 and removed v2.14.0 labels Jun 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce interface changes to support read/write blob with object metadata #13023

Introduce interface changes to support read/write blob with object metadata #13023

skumawat2025 commented Apr 2, 2024 •

edited

Loading

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 2, 2024 •

edited

Loading

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

skumawat2025 commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

Bukhtawar Apr 8, 2024

opensearch-trigger-bot bot commented May 20, 2024

skumawat2025 commented May 21, 2024

Introduce interface changes to support read/write blob with object metadata #13023

Introduce interface changes to support read/write blob with object metadata #13023

Conversation

skumawat2025 commented Apr 2, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 2, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 2, 2024

skumawat2025 commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

github-actions bot commented Apr 5, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

Bukhtawar Apr 8, 2024

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented May 20, 2024

skumawat2025 commented May 21, 2024

skumawat2025 commented Apr 2, 2024 •

edited

Loading

github-actions bot commented Apr 2, 2024 •

edited

Loading