Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] [Copy Activity with Avro file sink] Is it possible to get incomplete state in sink location in case of source failure? #688

Open
underthehill79 opened this issue Aug 30, 2024 · 0 comments

Comments

@underthehill79
Copy link

In our pipeline we want to use Copy Activity to move data from SAP HANA source into Avro file sink pointing to a unique location in ADLSv2.
Activity is statically configured with MaxRowsPerFile = 500000 and may be dynamically configured for parallel copy.
We anticipate that source might sometimes fail due to transient connectivity issues and want to configure the activity to perform a number of retries.
At this point we want to understand: is it possible to get incomplete state in sink location in case of source failure? And if its possible, what are our options to keep sink location empty between retries, or before next pipeline run, if all retry attempts ended with failure?

Thanks for the help!
-s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant