toil clean should be able to destroy non-resumable AWS job stores

[adamnovak]

This might go away with #964 and the SimpleDB removal, but right now AWS job stores can be in a partially-existing state, where the S3 bucket (and maybe the per-jobstore SimpleDB domain?) exist, but the entry in the "toil-registry" SimpleDB domain isn't there or says the job store isn't registered.

Then, toil clean can't resume the job store, so it can't be used to destroy the remaining pieces. The S3 bucket is simple enough to manually clean up, but the SimpleDB domain doesn't have a web UI to delete it with.

We should fix toil clean (and the job store implementation) to allow deleting job stores that don't quite exist, and aren't in a good enough state to be resumable. This might involve something like a best_effort=True flag we can pass to the job store resume() method, or else allowing destroy() on a job store without resume() being called first.

┆Issue is synchronized with this Jira Task
┆friendlyId: TOIL-1093

[adamnovak]

I now actually need this. I have a bunch of buckets like toil-cwl-6a8536e5-1cfb-433d-abef-9f03cb51d8bb--files from some TES(?) testing, and they all lack entries in the Toil registry SDB domain, and I can't delete them with the normal AWS command line tools because they have versioning enabled and are non-empty.

So either I need to write/find some scripts to enumerate and destroy all the versions in the buckets and then destroy the buckets, or I need to teach Toil to do it.

[DailyDreaming]

@adamnovak https://github.com/DataBiosphere/toil/blob/master/contrib/admin/cleanup_aws_resources.py should do this, though changing toil clean to handle this should still be the long term goal I think.