Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3A and S3N path are not correctly handled #2195

Open
benedeki opened this issue Oct 18, 2023 · 1 comment
Open

S3A and S3N path are not correctly handled #2195

benedeki opened this issue Oct 18, 2023 · 1 comment
Assignees
@benedeki
Labels
bug Something isn't working priority: high Critical to the health of the project

Comments

@benedeki
Copy link
Collaborator

Describe the bug

While the s3a:// and s3n:// protocols are correctly recognized, they are replaced with hard-coded s3://

To Reproduce

See FileSystemUtils class

Expected behavior

Keep the protocol.
@benedeki benedeki added bug Something isn't working priority: undecided Undecided priority to be assigned after discussion labels Oct 18, 2023
@benedeki benedeki self-assigned this Oct 18, 2023
@benedeki benedeki added priority: high Critical to the health of the project and removed priority: undecided Undecided priority to be assigned after discussion labels Oct 18, 2023
benedeki added a commit that referenced this issue Oct 18, 2023
@benedeki
#2195: S3A and S3N path are not correctly handled
2aaabb4 
* changed to preserve the protocol
* added UT
lsulak added a commit that referenced this issue Oct 18, 2023
@lsulak
#2195: fixing unit tests
ca2ca37
lsulak added a commit that referenced this issue Oct 18, 2023
@lsulak
#2195: making jackson compatible with spark (was problematic on my M1…
6fafe05 
… Mac) and commenting out currently failing tests (failing on my machine at least - probably due to dependency problems)
@benedeki benedeki mentioned this issue Dec 8, 2023
Merged
@benedeki
Copy link
Collaborator Author

benedeki commented Dec 8, 2023 •
edited by Zejnilovic
Loading

Release notes:

  • when using s3 path, the protocol in the path description is used, instead of hard-coded s3:// prefix, thus supporting other s3 protocols too.
  • added --jceks-path option to run scripts that will add --conf spark.hadoop.fs.s3a.bucket.SOME_NAME=jceks:/path/to/file.jceks. Use it as --jceks-path spark.hadoop.fs.s3a.bucket.SOME_NAME=jceks:/path/to/file.jceks
  • added s3a_wrapper.sh script to auto-detect the jceks file and auto-clean-up temporary versions from ECS

benedeki added a commit that referenced this issue Dec 8, 2023
@benedeki @lsulak @Zejnilovic
#2195: S3A and S3N path are not correctly handled (#2196)
ced78c5 
* changed to preserve the protocol
* added UT

---------

Co-authored-by: Ladislav Sulak <[email protected]>
Co-authored-by: Saša Zejnilović <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benedeki benedeki

Labels
bug Something isn't working priority: high Critical to the health of the project
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@benedeki