Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large filename can crash celery / redis inflights #1411

Open
sastels opened this issue Nov 27, 2023 · 12 comments
Open

Large filename can crash celery / redis inflights #1411

sastels opened this issue Nov 27, 2023 · 12 comments
Assignees
Labels
Bug | Bogue Content l Contenu Tasks to write, edit or delete content from any Notify pages Dev Task for implementation of a technical solution Incident Incident Ticket Low Priority | Faible priorité

Comments

@sastels
Copy link
Contributor

sastels commented Nov 27, 2023

Describe the bug

If a user sends a file attachment with an extremely long filename, celery cannot process the inflight and crashes. The inflight returns the the inbox and is pulled out again, and crashes. This will repeat without end.

Bug Severity

See examples in the documentation

SEV-3 Minor
Should not occur in normal use, however this impacts notifications before they are added to the database, so it will lead to lost notifications.

To Reproduce

Steps to reproduce the behavior:

  1. use the api to upload a file with a filename that is several MB in size.
  2. See the 201 response.
  3. See the non-stop celery errors

Expected behavior

We should catch this and reject the POST.

Impact

Could lead to lost notifications (both the one with the long filename and potentially others in the inflight)

@jzbahrai jzbahrai added Dev Task for implementation of a technical solution Incident Incident Ticket Low Priority | Faible priorité labels Dec 11, 2023
@whabanks whabanks self-assigned this Jan 18, 2024
@whabanks
Copy link

Draft PR is ready for eyes. Need to write tests next.

@jzbahrai jzbahrai self-assigned this Jan 24, 2024
@jzbahrai
Copy link
Collaborator

@whabanks

here is a screenshot of how I tested:

  1. I used this as the text to input https://github.com/aglover/moo/blob/master/etc/256kb-text.txt

  2. Screenshot 2024-01-24 at 1.43.36 PM.png
  3. The above step never completes ( it does if I put a small text)

  4. I also don't see any logs in the API for that request Screenshot 2024-01-24 at 1.44.20 PM.png

  5. Somewhere the validation is failing, but the frontend isn't getting a 400?

@whabanks
Copy link

TODO: Add front end error messages when the notification exceeds the allowed payload size limit.

@jzbahrai
Copy link
Collaborator

We can then merge this PR: cds-snc/notification-api#2085

@yaelberger-commits
Copy link
Collaborator

Hey team! Please add your planning poker estimate with Zenhub @andrewleith @jzbahrai @whabanks

@yaelberger-commits yaelberger-commits added the Content l Contenu Tasks to write, edit or delete content from any Notify pages label Dec 9, 2024
@whabanks
Copy link

whabanks commented Dec 11, 2024

Just adding this info for the future...

AWS offers an extended SQS client that would allow us to send SQS messages up to 2GB in size. Instead of returning an error to the user when their notification exceeds the 256kb limit, we would instead store the data in an S3 bucket and provide the extended client with a reference to this resource.

While this constitutes extra work, it aligns with our mission to ensure that GCNotify is simple and easy to use for our clients. It mitigates the burden of surfacing technical limitations and removes responsibility from our clients to account for these limitations.

@YedidaZalik
Copy link

YedidaZalik commented Dec 17, 2024

@YedidaZalik
Copy link

YedidaZalik commented Dec 18, 2024

Will answered my questions in the morning and I continued on content here: https://www.figma.com/design/Di5yjYVTyeoUDJ4YmWYCPs/Form-validation-and-errors---Validation-et-erreurs?node-id=2005-155&t=lgYGTwQx6n5FdRJ0-0

Now I have a few more questions:
Does the column/row info assist API users who are sending without a spreadsheet?
Or is that only for UI users and we need to refer to it differently for API users?
If so, how would API users refer to the content that should be removed?

Will continue this work on Monday

@jzbahrai jzbahrai removed their assignment Dec 19, 2024
@YedidaZalik
Copy link

After discussion with Will, we do not need error messages for bulk sends, only need error messages for sending to one recipient. Will says in API also tell user how to fix error, as we would do in the API.
Draft messages in green box for the different scenarios for Will to check:
https://www.figma.com/design/Di5yjYVTyeoUDJ4YmWYCPs/Form-validation-and-errors---Validation-et-erreurs?node-id=2005-154&p=f&t=ZqUkaoTO04HIS20I-0

@YedidaZalik
Copy link

Implemented feedback from Will
He will implement and we can send UI error message to localization after holiday

@yaelberger-commits
Copy link
Collaborator

Will collaborated with Yedida on En content, needs localisation. Wrapping up backend work today

@YedidaZalik
Copy link

Checked with Will, only the UI error message needs localization.
Prepped for Marie-Sophie here: https://www.figma.com/design/Di5yjYVTyeoUDJ4YmWYCPs/Form-validation-and-errors---Validation-et-erreurs?node-id=2067-34&p=f&t=Z8N89Vv2rW0VbDRd-0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug | Bogue Content l Contenu Tasks to write, edit or delete content from any Notify pages Dev Task for implementation of a technical solution Incident Incident Ticket Low Priority | Faible priorité
Projects
None yet
Development

No branches or pull requests

5 participants