Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ContentUploaddService: Retry uploads #339

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

mnaamani
Copy link
Member

Some storage nodes that have a slow storage-squid that might be lagging a few blocks behind, will fail to accept upload of a dataobject if the object was created just before the upload attempt for example:

{"@timestamp":"2024-10-17T04:42:43.230Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920679 doesn't exist in storage bag {\"dynamic\":{\"channel\":60210}}"}
{"@timestamp":"2024-10-17T13:54:48.899Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920740 doesn't exist in storage bag {\"dynamic\":{\"channel\":65925}}"}
{"@timestamp":"2024-10-17T13:55:18.518Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920748 doesn't exist in storage bag {\"dynamic\":{\"channel\":65925}}"}
{"@timestamp":"2024-10-17T14:00:48.856Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920788 doesn't exist in storage bag {\"dynamic\":{\"channel\":65925}}"}
{"@timestamp":"2024-10-17T22:10:55.624Z","ecs.version":"8.10.0","label":"StorageNodeApi","log.level":"error","message":"https://storage.mrbovo.xyz/storage/api/v1/files - errorCode: 400, msg: Error: Data object 2920855 doesn't exist in storage bag {\"dynamic\":{\"channel\":59045}}"}

This PR simply adds a retry mechanism.

@mnaamani mnaamani changed the title Retry uploads ContentUploaddService: Retry uploads Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant