Issue w/ process terminating - CWW testing the pipeline

[trouille]

Mason of the Chicago Wildlife Watch team testing out https://subject-assistant.zooniverse.org/#/intro with their data.

His notes on this issue:

Start on hamlet at 8:17, upload the Spring 2019 subject set

Refresh on 8:24, not ready

Refresh on 8:33, not ready

Refresh on 9:07, complete, follow link to go back to subject assistant and try to fetch getting error saying that “ML task is still running, check again later”

Try to fetch on 9:26, same response (ML still running)

Try to fetch on 9:45, same response (ML still running)

Try to fetch on 10:30, terminated with this response.

[trouille]

Note - Mason has tried this a few times. His internet has remained on the entire time.

[shaunanoordin]

Thank you for reporting this - I think the issue lies with with the Subject Assistant Proxy service.

Investigation

Issue: attempting to retrieve Mason's task at https://subject-assistant.zooniverse.org/#/tasks/ad68546e-9ec4-4bed-a023-b779cb7fc40f results in a "Request has been terminated" error.

• The "Request has been terminated" error is unexpected.
  • If the Task has expired - or the Task ID was mistyped - we expect a "no suck Task can be found" error
  • If the Task encountered errors on the Microsoft processing side, we expect a "Task had problems" error
  • If the user's internet connection was wonky, we would expect a complete inability to load the Subject Assistant itself.
• Following the network requests, we see that we're getting a 502 error from the Subject Assistant Proxy service - https://subject-assistant-proxy.zooniverse.org/?url=https%3A%2F%2Fcameratrap.blob.core.windows.net%2Fasync-api-zooniverse%2Fad68546e-9ec4-4bed-a023-b779cb7fc40f%2Fad68546e_detections_zooniverse-subject-assistant_20200902135513.json%3Fsp%3Dr%26sr%3Db%26sv%3D2019-02-02%26se%3D2020-12-01T14%253A57%253A21Z%26sig%3DGeNOKU10GQDw8Jm1XhqJdb8%2FnFx5TP0CiXdKVqSV8Rg%253D
  • Note: the proxy service sits between the Subject Assistant app and the Microsoft servers.
• The actual JSON result file from Microsoft (which the proxy was trying to fetch) actually looks fine! https://cameratrap.blob.core.windows.net/async-api-zooniverse/ad68546e-9ec4-4bed-a023-b779cb7fc40f/ad68546e_detections_zooniverse-subject-assistant_20200902135513.json?sp=r&sr=b&sv=2019-02-02&se=2020-12-01T14%3A57%3A21Z&sig=GeNOKU10GQDw8Jm1XhqJdb8/nFx5TP0CiXdKVqSV8Rg%3D (15.6MB JSON) (Note: user is free to download this file directly, but it's raw data that doesn't benefit from the 'data massaging' that comes from the Subject Assistant.)

At this point, we can determine that the misbehaviour is at the proxy service. Question is: why?

⚠️ SIDE NOTE this investigation is time-sensitive, as results on the Microsoft server have a limited lifespan before they're deleted (to save space, presumably.)

Further Observations and Debug Hypothesis

• My own sample task actually works fine - https://subject-assistant.zooniverse.org/#/tasks/066f454f-4004-43a0-9d88-d95bc1090b9c - neither the Subject Assistant nor proxy service has issues fetching data from their respective partners.
• The only notable difference I can determine is that my sample task is very small (about 100+ results) vs Mason of CWW's task (the file from Microsoft is 15.6MB)

❗ (❓ ) My current hypothesis is that the proxy service isn't able to handle large requests very well.

This makes a certain amount of sense - the Subject Assistant has, up to this point, been mostly tested with limited curations, with fairly small Subject Set sizes in the range of hundreds of images. A "real" job using a full CWW Subject Set could number in the tens of thousands of images; meaning the theoretical server load wasn't calibrated for practical real world values. 🤦

FAQ

Q: Wait, why use a Proxy service? If you can already pull the actual URL from Microsoft - https://cameratrap.blob.core.windows.net/async-api-zooniverse/ad68546e-9ec4-4bed-a023-b779cb7fc40f/ad68546e_detections_zooniverse-subject-assistant_20200902135513.json?sp=r&sr=b&sv=2019-02-02&se=2020-12-01T14%3A57%3A21Z&sig=GeNOKU10GQDw8Jm1XhqJdb8/nFx5TP0CiXdKVqSV8Rg%3D - why bother with a middle man?

A: JavaScript cross-domain request restrictions, unfortunately. Web browsers don't let code from *.zooniverse.org go willy-nilly downloading files from *.windows.net for safety/security reasons.

Status

We have an idea why things are going wrong - now to confirm it and find a solution.

[camallen]

I'm happy to help here @shaunanoordin

When you are free let's schedule a screen share, to reproduce the failure and debug the subject assistant proxy server. Ive seen a few crashes on this service so it might be a simple resource limit (i.e. our of ram).

[shaunanoordin]

@camallen You already figured out the problem and the solution - it WAS an issue of memory! PR #57 bumped up the available memory for the Proxy from 50Mi to 100Mi.

This means the example problem (link to Proxy Service) now correctly fetches the results (a 15.6MB JSON file) instead of dying with a 502 error.

@trouille please tell Mason to try https://subject-assistant.zooniverse.org/#/tasks/ad68546e-9ec4-4bed-a023-b779cb7fc40f once again, and to tell them thank you for highlighting this issue.

I'm going to open up a follow up issue in this repo in case anyone else with a larger Subject Set encounters the same problem, so we know the issue lies in the available RAM.