Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support intermediate storage option for avro-schema-migration #52

Open
laurencedaluz opened this issue May 2, 2022 · 2 comments
Open

Comments

@laurencedaluz
Copy link

The --avro-schema-migration flag requires a "linked" cluster so that it can copy the avro schema file to the target cluster.
It would be great if this scenario also worked with the intermediate storage flag (instead of a "linked" cluster), as this would simplify the migration of avro tables to public cloud.

@dstreev
Copy link
Owner

dstreev commented May 3, 2022

The current problem is that the movement of the AVRO schema file is done through the HCFS client and relies on the host to be configured with the core/hdfs-site.xml files for the cluster. But in this case, you only have access to the side that you're running it on, but not the other side.

@dstreev
Copy link
Owner

dstreev commented May 3, 2022

Now, if we used 'common-storage' we could make it work because that's where the data will live too. And the other cluster could just reference the location where it was placed by the source cluster. In this scenario, which is usually the case because of visibility, hms-mirror is run from the LEFT cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants