Podman - Error Reading blob : no such file or directory when doing a push to ECR Registry

[questionson]

We are trying to push an image from AWS EC2 Instance. Though we can push multiple images without any issues, once a while we get the error which says no such file or directory.

Here is the Podman and EC2 OS version info

[ec2-user@ec2-host ~]$ podman --version
podman version 4.9.4-rhel

[2-user@ec2-host ~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.10 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.10 (Ootpa)"

Here are the images

[ec2-user@ec2-host ~]$ podman images
REPOSITORY                                                                TAG                  IMAGE ID      CREATED      SIZE
1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer  1.23.1-debian-12-r3  828bd4cee5c3  11 days ago  216 MB
docker.io/bitnami/pgbouncer   

Here is the output when doing a push

[ec2-user@ec2-host ~]$ podman push 1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3

Found image "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" as "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" in local containers storage
Error: reading blob sha256:d581f1e65f7a7426c948b631239d35ec006e704d1fd05d2d11af42fc3e09f3bd: open /home/ec2-user/.local/share/containers/storage/overlay/d581f1e65f7a7426c948b631239d35ec006e704d1fd05d2d11af42fc3e09f3bd/diff/usr/bin/bash: no such file or directory

The below is the output in debug level during a push

[ec2-user@ec2-host ~]$ podman push --log-level=debug 1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3

    INFO[0000] podman filtering at log level debug
DEBU[0000] Called push.PersistentPreRunE(podman push --log-level=debug 1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] Overriding run root "/run/user/1000/containers" with "/tmp/containers-user-1000/containers" from database
DEBU[0000] Overriding tmp dir "/run/user/1000/libpod/tmp" with "/tmp/podman-run-1000/libpod/tmp" from database
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/ec2-user/.local/share/containers/storage
DEBU[0000] Using run root /tmp/containers-user-1000/containers
DEBU[0000] Using static dir /home/ec2-user/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/podman-run-1000/libpod/tmp
DEBU[0000] Using volume path /home/ec2-user/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Setting parallel job count to 7
DEBU[0000] Looking up image "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" ...
DEBU[0000] parsed reference into "[overlay@/home/ec2-user/.local/share/containers/storage+/tmp/containers-user-1000/containers]@828bd4cee5c33795afb402f3a8c1ecc71c5aa5381c15dcd1f54aa64b81f1ea85"
DEBU[0000] Found image "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" as "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" in local containers storage
DEBU[0000] Found image "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" as "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" in local containers storage ([overlay@/home/ec2-user/.local/share/containers/storage+/tmp/containers-user-1000/containers]@828bd4cee5c33795afb402f3a8c1ecc71c5aa5381c15dcd1f54aa64b81f1ea85)
DEBU[0000] exporting opaque data as blob "sha256:828bd4cee5c33795afb402f3a8c1ecc71c5aa5381c15dcd1f54aa64b81f1ea85"
DEBU[0000] Pushing image 1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3 to 1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Copying source image [overlay@/home/ec2-user/.local/share/containers/storage+/tmp/containers-user-1000/containers]@828bd4cee5c33795afb402f3a8c1ecc71c5aa5381c15dcd1f54aa64b81f1ea85 to destination image //1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
DEBU[0000] Found credentials for 1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer in credential helper containers-auth.json in file /run/user/1000/containers/auth.json
DEBU[0000]  No signature storage configuration found for 1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3, using built-in default file:///home/ec2-user/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/1xxxxx.dkr.ecr.us-east-2.amazonaws.com
DEBU[0000] Using SQLite blob info cache at /home/ec2-user/.local/share/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/home/ec2-user/.local/share/containers/storage]@828bd4cee5c33795afb402f3a8c1ecc71c5aa5381c15dcd1f54aa64b81f1ea85
DEBU[0000]  Using transport "containers-storage" policy section ""
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Checking if we can reuse blob sha256:d581f1e65f7a7426c948b631239d35ec006e704d1fd05d2d11af42fc3e09f3bd: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar" = true
DEBU[0000] Checking /v2/docker.io/bitnami/pgbouncer/blobs/sha256:d581f1e65f7a7426c948b631239d35ec006e704d1fd05d2d11af42fc3e09f3bd
DEBU[0000] GET https://1xxxxx.dkr.ecr.us-east-2.amazonaws.com/v2/
DEBU[0000] Ping https://1xxxxx.dkr.ecr.us-east-2.amazonaws.com/v2/ status 401
DEBU[0000] HEAD https://1xxxxx.dkr.ecr.us-east-2.amazonaws.com/v2/docker.io/bitnami/pgbouncer/blobs/sha256:d581f1e65f7a7426c948b631239d35ec006e704d1fd05d2d11af42fc3e09f3bd
DEBU[0000] ... not present
DEBU[0000] Trying to reuse blob with cached digest sha256:ea261118fbcdf100baedf94adcffadbaea91b5aa93c3551191acaacc4b4e020f compressed with gzip with no location match, checking current repo
DEBU[0000] Checking /v2/docker.io/bitnami/pgbouncer/blobs/sha256:ea261118fbcdf100baedf94adcffadbaea91b5aa93c3551191acaacc4b4e020f
DEBU[0000] HEAD https://1xxxxx.dkr.ecr.us-east-2.amazonaws.com/v2/docker.io/bitnami/pgbouncer/blobs/sha256:ea261118fbcdf100baedf94adcffadbaea91b5aa93c3551191acaacc4b4e020f
DEBU[0000] ... not present
DEBU[0000] exporting filesystem layer "d581f1e65f7a7426c948b631239d35ec006e704d1fd05d2d11af42fc3e09f3bd" without compression for blob "sha256:d581f1e65f7a7426c948b631239d35ec006e704d1fd05d2d11af42fc3e09f3bd"
Copying blob d581f1e65f7a [--------------------------------------] 0.0b / 205.8MiB | 0.0 b/s
DEBU[0001] Looking up image "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" in local containers storage
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0001] Trying "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" ...
DEBU[0001] parsed reference into "[overlay@/home/ec2-user/.local/share/containers/storage+/tmp/containers-user-1000/containers]@828bd4cee5c33795afb402f3a8c1ecc71c5aa5381c15dcd1f54aa64b81f1ea85"
DEBU[0001] Found image "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" as "1xxxxx.dkr.ecr.us-east-2.amazonaws.com/docker.io/bitnami/pgbouncer:1.23.1-debian-12-r3" in local containers storage
Error: reading blob sha256:d581f1e65f7a7426c948b631239d35ec006e704d1fd05d2d11af42fc3e09f3bd: open /home/ec2-user/.local/share/containers/storage/overlay/d581f1e65f7a7426c948b631239d35ec006e704d1fd05d2d11af42fc3e09f3bd/diff/usr/bin/bash: no such file or directory
DEBU[0001] Shutting down engines
[ec2-user@ec2-host  ~]$