Possible issue with retries on a put to S3 when using the iput -X flag.

[JustinKyleJames]

I saw that there were S3 errors when someone did an iput -r -X to an S3 resource.

I did a little testing where I created a directory of ten 100MiB files then recursively put that directory to an S3 resource. If I left it alone it passed.

However, if I did a control-c on the first iput, then restarted the iput (with the -f flag), I would sometimes get an S3 error. If I kept trying it would eventually pass.

My working theory is that because S3 must store some state in shared memory, the shared memory didn't get cleaned up on the exit and a retry would have an inconsistent state.

The shared memory logic does have a timeout so once that timeout expires the memory will be flushed. That could be why it eventually passes.

Some notes:

1. This is just an educated guess at this point. I never actually reproduced the exact error (timeout reading circular buffer on part upload) that the user was seeing.
2. I also saw failures when retrying after a control-C with iput -X to a unixfilesystem resource. One such failure was due to a file being left in intermediate state.

[alanking]

One note 2, did the data object in question eventually come out of the intermediate state due to the agent timing out and tearing itself down?

[JustinKyleJames]

One note 2, did the data object in question eventually come out of the intermediate state due to the agent timing out and tearing itself down?

It didn't but maybe I didn't wait long enough?

[alanking]

Okay, if possible, please confirm that the object is not stuck. If it is stuck, this is a situation we need to examine for the core code because that should not happen no matter what.

[JustinKyleJames]

Okay, if possible, please confirm that the object is not stuck. If it is stuck, this is a situation we need to examine for the core code because that should not happen no matter what.

Well now I can't reproduce it. ;-)

[trel]

iput and ctrl-c should send the signal to the server, and it should 'finalize'... but if it didn't get a chance to send anything, the server will wait for the timeout before finalizing.

or we have a bug, like alan said.