feat/fix: FORMS-1483 queued file handling

[jasonchung1871]

Description

Adds the functionality so that files are not uploaded/deleted until the submitter clicks submit or save draft. Users that would delete or upload a file, then navigate away without saving the draft or submitting would have their files either exist in our storage or deleted with no recovery. This mitigates that issue.

Resolves https://bcdevex.atlassian.net/browse/FORMS-1483

When adding a file to the file upload, it will give you a notification that it won't be uploaded until you hit save or save as draft.

Type of Change

feat (a new feature)
fix (a bug fix)

test (add missing tests or correct existing tests)

Checklist

• I have read the CONTRIBUTING doc
• I have checked that unit tests pass locally with my changes
• I have run the npm script lint on the frontend and backend
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• I have approval from the product owner for the contribution in this pull request

Further comments

[jasonchung1871]

I think we may not be able to queue the uploaded files. How does this behave when uploading 100 files that are multi-megabytes each?

What happens if one file upload fails in the middle? Is the current behaviour that the user can delete it and re-upload? What will the behaviour be with this?

I think the file deletes can be queued, as there is not a lot of data being transmitted. But the uploads can be very long on slow connnections.

Yes, that would be the behaviour. The formio file component has a property called dataValue which determines whether a file was successfully uploaded or not. When a file is dragged or selected into the file component, that information is forwarded to the front end where we queue it for upload. When the user submits the form or saves the draft, we synchronously begin uploading the files. If it fails to upload a single file, it will show an error to the user, call the error handler for the file component, and continue uploading the rest of the files synchronously. If a file succeeds, it gets removed from the queue and we call the event handler for a successful file upload for the file component. If all files upload successfully, then it will continue to submit or save the draft.

Some potential changes:

• Display a message to the user that files are being uploaded, the file upload component might be hidden in a container like tabs.
• Decide if we want to asynchronously upload files, it may be a concern for users with slower internet connections that are uploading multiple large files. It may end up timing out for all the files if they take too long.
• Decide if we want to attempt a re-upload. It would depend on what the issue is though. It could be client sided errors where the user is uploading a document, selects it in the file component, moves the file on their local machine, then submits the form and then it fails to upload because they moved it or deleted it, etc. Attempts to re-upload would just keep failing because it's no longer in the location that the file upload thinks it is.

[WalterMoar]

There's a variety of behaviour with files around 25MB. Dev/test/prod have strange behaviour, but sticking to this PR:

If I upload a 25MB + 1B (26,214,401B) file, I get the expected error but I can still submit (it's not required) and the file isn't attached to the submission - maybe this isn't the desired behaviour?

---

If I upload a 25MB (26,214,400B) file, I do not get an error:

but when I click "Submit" the button continues to spin for 30s after the upload completes. Then it produces some error alerts:

---

If I upload a 25MB - 1B (26,214,399B) file, everything works as expected.

---

So perhaps the backend is < 25MB, and the frontend is <= 25MB? This is a super-unlikely case for real files, but maybe we should fix it now to make sure the error handling in the frontend is correct (or that the problem just goes away)?

[WalterMoar]

Just curious: is there a reason to upload them in reverse order? If I add file 0, 1, and 2, my submission shows file 2, 1, and 0. Might the users find this to be unexpected behaviour?

[jasonchung1871]

The reasoning for this was because it was uploading files from the array, and when successful, would remove those files from the array. That way when running the upload file function again, it would just run through the array until it is empty.

[jasonchung1871]

There's a variety of behaviour with files around 25MB. Dev/test/prod have strange behaviour, but sticking to this PR:

If I upload a 25MB + 1B (26,214,401B) file, I get the expected error but I can still submit (it's not required) and the file isn't attached to the submission - maybe this isn't the desired behaviour?

If I upload a 25MB (26,214,400B) file, I do not get an error: but when I click "Submit" the button continues to spin for 30s after the upload completes. Then it produces some error alerts:

If I upload a 25MB - 1B (26,214,399B) file, everything works as expected.

So perhaps the backend is < 25MB, and the frontend is <= 25MB? This is a super-unlikely case for real files, but maybe we should fix it now to make sure the error handling in the frontend is correct (or that the problem just goes away)?

I suspect that this may end up being desired behaviour, unless we intend to enforce that all selected files are valid even for forms that don't have the required set for the file components.

This may end up being a back end change for the validation. The validation for the file upload component is a built-in method for the component. Their method is much cleaner than handling it ourselves, because it translates the string 1GB into a scalar that can be used for validating the file size.

[github-actions]

Release cc6a871 deployed at https://chefs-dev.apps.silver.devops.gov.bc.ca/pr-1515

[jasonchung1871]

There's a variety of behaviour with files around 25MB. Dev/test/prod have strange behaviour, but sticking to this PR:
If I upload a 25MB + 1B (26,214,401B) file, I get the expected error but I can still submit (it's not required) and the file isn't attached to the submission - maybe this isn't the desired behaviour?
If I upload a 25MB (26,214,400B) file, I do not get an error: but when I click "Submit" the button continues to spin for 30s after the upload completes. Then it produces some error alerts:
If I upload a 25MB - 1B (26,214,399B) file, everything works as expected.
So perhaps the backend is < 25MB, and the frontend is <= 25MB? This is a super-unlikely case for real files, but maybe we should fix it now to make sure the error handling in the frontend is correct (or that the problem just goes away)?

I suspect that this may end up being desired behaviour, unless we intend to enforce that all selected files are valid even for forms that don't have the required set for the file components.

This may end up being a back end change for the validation. The validation for the file upload component is a built-in method for the component. Their method is much cleaner than handling it ourselves, because it translates the string 1GB into a scalar that can be used for validating the file size.

So, I looked into this again. Both the front end and the back end use the config to set their max file size. The back end uses multer for handling files, and the front end uses the built in formio methods. I don't think we can as easily customize the functionality for multer, so we might need to copy the formio functionality and rewrite it just so we can convert the configs file size into a scalar and then subtract it by 1.