-
Notifications
You must be signed in to change notification settings - Fork 362
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove ADLS hint from import #7581
Conversation
♻️ PR Preview d9337ee has been successfully destroyed since this PR has been closed. 🤖 By surge-preview |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Worried about backwards compatibility: if someone has an old workflow or even script:
- We prefer not to break it
- We must give a useful error message if they try.
As is, we must document this change, and it seems like a major version bump.
As an alternative, we might translate the modified URLs back to their original form, and ideally issue a warning that we did so. (Other alternatives are possible, of course!)
@@ -568,11 +568,11 @@ func (a *Adapter) CompleteMultiPartUpload(ctx context.Context, obj block.ObjectP | |||
|
|||
func (a *Adapter) GetStorageNamespaceInfo() block.StorageNamespaceInfo { | |||
info := block.DefaultStorageNamespaceInfo(block.BlockstoreTypeAzure) | |||
info.ImportValidityRegex = `^https?://[a-z0-9_-]+\.(blob|adls)\.core\.windows\.net` // added adls for import hint validation in UI | |||
info.ImportValidityRegex = `^https?://[a-z0-9_-]+\.blob\.core\.windows\.net` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This breaks any existing import scripts. Does this mean we need a major release 🤢 ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As stated in the PR description - this is a breaking change. But I believe we can avoid a major release since this affects specific users we can identify
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a WA for lakectl so it doesn't break. Effectively this is only going to impact WebUI users (I'm fine with that)
### Azure Data Lake Gen2 | ||
{:.no_toc} | ||
|
||
lakeFS requires a hint in the import source URL to understand that the provided storage account is ADLS Gen2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should document the change. Otherwise someone who's already imported has no way of understanding what went wrong!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added deprecation message
We need to get rid of this - the sooner the better. |
@arielshaqed Added also backwards compatibility for the delta exporter |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
cmd/lakectl/cmd/import.go
Outdated
Warning("'adls' hint will be deprecated soon, please use the original source url for import") | ||
source = strings.Replace(source, "adls.core.windows.net", "blob.core.windows.net", 1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer to say what is happening, not only what isn't:
Warning("'adls' hint will be deprecated soon, please use the original source url for import") | |
source = strings.Replace(source, "adls.core.windows.net", "blob.core.windows.net", 1) | |
Warning("'adls' hint will be deprecated soon, please use the original source url for import") | |
translatedSource = strings.Replace(source, "adls.core.windows.net", "blob.core.windows.net", 1) | |
if source != translatedSource { | |
Fmt(" Using %s\n", translatedSource) | |
} |
Closes #7580
Change Description
Remove ADLS hint from import
Background
We are saving the physical address of imported objects in ADLS Gen2 accounts wrong which creates issues when using the physical address (to export tables for example)
Bug Fix
blob
toadls
Testing Details
Verified by current unit and system tests
Breaking Change?
Yes - will require users who use the adls hint to stop using it