Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can the JDP accept file requests > 10TB and throttle as needed on the backend? #3

Open
jeff-cohere opened this issue Mar 8, 2023 · 1 comment
Open

Can the JDP accept file requests > 10TB and throttle as needed on the backend? #3

jeff-cohere opened this issue Mar 8, 2023 · 1 comment
Labels
JDP Involves a feature request or bugfix to the JGI Data Portal

Comments

@jeff-cohere
Copy link
Collaborator

Alicia Clum pointed out that NMDC often needs large payloads (~50GB) transferred from the JGI data portal, but the data portal's daily per-user limit is 10TB, which means that several separate transfers under this limit must be chained together over several days. Obviously, it would be better if a single large request could be made and broken up into chunks by the data portal itself.

This isn't a DTS request per se, but I can see this helping the DTS as well.
@jeff-cohere jeff-cohere added the JDP Involves a feature request or bugfix to the JGI Data Portal label Mar 8, 2023
@jeff-cohere
Copy link
Collaborator Author

Now that we track payload sizes (in GB) and have subtasks for transfer tasks, it's possible in principle for us to implement a throttling/daisy-chaining mechanism for huge file transfers within the DTS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
JDP Involves a feature request or bugfix to the JGI Data Portal
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jeff-cohere