Potentially corrupt blob

[com6056]

Haven't narrowed it down yet, but ran into this pull error which went away when I cleared our the S3 bucket:

  Warning  Failed                  3s                     kubelet                  Failed to pull image "docker.io/foo/bar@sha256:f05918ce7a92b84415e1abaa1c6ba0d7212b05adca41360673873bc85be88b91": rpc error: code = Unknown desc = writing blob: storing blob to file "/var/tmp/storage2866776745/3": happened during read: unexpected EOF

Same deal when trying to pull directly from the node too:

[root@node ~]# crictl pull docker.io/foo/bar@sha256:f05918ce7a92b84415e1abaa1c6ba0d7212b05adca41360673873bc85be88b91
E1216 00:47:48.098893    6802 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unknown desc = writing blob: storing blob to file \"/var/tmp/storage4078425335/7\": happened during read: unexpected EOF" image="docker.io/foo/bar@sha256:f05918ce7a92b84415e1abaa1c6ba0d7212b05adca41360673873bc85be88b91"
FATA[0005] pulling image: rpc error: code = Unknown desc = writing blob: storing blob to file "/var/tmp/storage4078425335/7": happened during read: unexpected EOF

I'm guessing the blob being pulled somehow got into a funky state in the bucket since the issue went away after I cleared out the S3 bucket, need to dig in more but wanted to put this into an issue in case anyone knew what might have happened. Thanks!

[mcronce]

This is, hopefully, fixed by the digest checking added in v0.4.4

[thejosephstevens]

We'd be happy to test this out! We see this issue fairly frequently across our systems, so we should be able to validate if it's gone. We're upgrading from 0.3.8, is there anything we should be aware of while upgrading? I don't see upgrade notes anywhere, not sure if I'm looking in the right place

[mcronce]

@thejosephstevens you shouldn't need to do anything special, the only reason I bumped from v0.3 to v0.4 is because of an additional feature added to URL parsing to support CRI-O. I'm uploading v0.4.5 now to fix an issue with enabling this digest checking feature in the helm chart.