Files: Download/upload from external storage system

[lnielsen]

(extracted from #40)

Goals

Enable file uploads/downloads directly from an external storage system for performance and scalability reasons.

Use cases

• Support a large number of concurrent downloads - important when clients download large files over slow connections.
• Support fast download speeds as well as potentially range requests etc by directly leveraging the storage systems capabilities (the storage system is better at handling file upload/download than Invenio).

Tasks

• Deposit form:
  • Ensure the file uploader app is able to upload via Invenio or directly to the storage system.
• Backend:
  • Support redirect to storage
  • Evaluate if PyFilesystem should still be used or we need direct access to e.g. S3/XRootD etc.
• S3 test case - support for uploading directly to min.io in the container.
• File integrity checking

Things to take into account:

• File download statistics.
• Restricted files
• File processing (e.g. image/width extraction)