BUG: Premature closure of Dask Client

[davedavemckay]

BUG 🐛:

• Dask Client closes when process_files() completes

INVESTIGATION 🕵️:

• process_files() completes when all futures reach 'Completed' or 'Exception' status ('Exception' is a failsafe that hasn't ever been observed)
• occurs with collated uploads (unknown if occurs with file upload)
• futures reach 'Completed' when zip_and_upload returns
• zip_and_upload returns when upload_and_callback returns or no files are found to zip (i.e., an empty folder is passed to zip_and_upload)
• upload_and_callback returns when upload_to_bucket_collated returns result and result is written to log
• upload_to_bucket_collated returns when swiftclient.Connection.put_object returns
• swiftclient.Connection.put_object is a wrapper of swiftclient.client.post_object and returns a swiftclient.Connection._retry object
• swiftclient.client.post_object seems to run in the background and update a response_dict on completion or fail after a given number of retries
  • introduction a while True that checks this response_dict may prevent premature closing of the Dask Client. 😅

[davedavemckay]

Temporary fix with loop script.

[davedavemckay]

PR, but keep branch