Hyku Documentation Wiki Space
Improved Support and Usability for Large File Uploads (and Large Quantity Imports)
Summary and Current Challenges
Currently, there is fear and uncertainty around large file (or quantity) uploads. The current functionality regarding chunked uploads often works. However, when it doesn’t, it is often difficult for users to understand what restrictions on uploads are from their admins vs which are technical errors or defects.
Needs
Libraries must be able to:
import large files, one at a time and via Bulkrax. Currently, the system cannot import files over ??GB. Videos in particular can be this size and larger, as well as high-resolution images and maps.
Import a large number of files via Bulkrax. Currently, the system has a difficult time importing over ?? files at one time. Libraries have had to chunk these files to get them into repositories. Technically, there should not be a limit to the number, assuming the infrastructure can support it.
If there is a limit, it should be communicated on the upload page
Solution / Design discussion
Current state and questions
What are the causes for errors or difficulty right now?
Where did the work done on Globus go? (Rutgers research data repository)
Is globus a good solution?
Do we need / want to replace chunked uploads?
Is chunked uploads just a workaround for CDN / ingress restrictions?
What is the best approach to handle large file uploads in a rails app? << Old Stack overflow thread (2013)
Is it possible to show progress for individual files? Do we now?
Can users pause / resume?