Photon 4 causing pull-ova-all to fail on all PRs

[AverageMarcus]

Environment

N/A - This is happening on the pull-ova-all Prow job on all PRs.

Example PRs:

• Update Ubuntu 22.04 ISO URLs to use latest stable URLs #1438
• Bump ansible-core to v2.15.10 #1439

What steps did you take and what happened?

Run /test pull-ova-all on a PR and the following error occurs in the photon-4 log output:

Error verifying signature of: /var/cache/tdnf/photon-updates/rpms/x86_64/linux-5.10.212-4.ph4.x86_64.rpm\nError processing package: x86_64/linux-5.10.212-4.ph4.x86_64.rpm\nError(1514) : RPM is signed but failed to match with known keys. Use --nogpgcheck to ignore.

(Full log below)

What did you expect to happen?

The test to pass.

Relevant log output

Log Output

hack/ensure-python.sh
Checking if python is available
Python 3.11.2
hack/ensure-ansible.sh
ansible [core 2.15.9]
Starting galaxy collection install process
Nothing to do. All requested collections are already installed. If you want to reinstall them, consider using `--force`.
hack/ensure-packer.sh
hack/ensure-ansible-windows.sh
hack/ensure-goss.sh
Right version of binary present
hack/ensure-ovftool.sh
/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/.local/bin/packer init packer/config.pkr.hcl
/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/.local/bin/packer build -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/kubernetes.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/cni.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/containerd.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/wasm-shims.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/ansible-args.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/goss-args.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/common.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/additional_components.json"  -color=true -var-file="packer/ova/packer-common.json" -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/ova/photon-4.json" -var-file="packer/ova/vsphere.json" -except=local -only=vsphere-clone -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/ci-photon-4.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/scripts/ci-disable-goss-inspect.json"  packer/ova/packer-node.json
Warning: Bundled plugins used

This template relies on the use of plugins bundled into the Packer binary.
The practice of bundling external plugins into Packer will be removed in an
upcoming version.

To remove this warning and ensure builds keep working you can install these
external plugins with the 'packer plugins install' command

* packer plugins install github.com/hashicorp/vmware
* packer plugins install github.com/hashicorp/vsphere

Alternatively, if you upgrade your templates to HCL2, you can use 'packer init'
with a 'required_plugins' block to automatically install external plugins.

You can try HCL2 by running 'packer hcl2_upgrade
/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/ova/packer-node.json'


�[1;32mvsphere-clone: output will be in this color.�[0m

�[1;32m==> vsphere-clone: Cloning VM...�[0m
�[1;32m==> vsphere-clone: Customizing hardware...�[0m
�[1;32m==> vsphere-clone: Mounting ISO images...�[0m
�[1;32m==> vsphere-clone: Adding configuration parameters...�[0m
�[1;32m==> vsphere-clone: Power on VM...�[0m
�[1;32m==> vsphere-clone: Waiting for IP...�[0m
�[1;32m==> vsphere-clone: IP address: 192.168.8.14�[0m
�[1;32m==> vsphere-clone: Using SSH communicator to connect: 192.168.8.14�[0m
�[1;32m==> vsphere-clone: Waiting for SSH to become available...�[0m
�[1;32m==> vsphere-clone: Connected to SSH!�[0m
�[1;32m==> vsphere-clone: Provisioning with shell script: ./packer/files/flatcar/scripts/bootstrap-flatcar.sh�[0m
�[1;32m==> vsphere-clone: Provisioning with Ansible...�[0m
�[0;32m    vsphere-clone: Setting up proxy adapter for Ansible....�[0m
�[1;32m==> vsphere-clone: Executing Ansible: ansible-playbook -e packer_build_name="vsphere-clone" -e packer_*****_type=vsphere-clone -e packer_http_addr=10.60.109.113:0 --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url=https://github.com/containerd/containerd/releases/download/v1.7.13/cri-containerd-cni-1.7.13-linux-amd64.tar.gz containerd_sha256=9be621c0206b5c20a1dea05fae12fc698e5083cc81f65c9d918c644090696d19 pause_image=registry.k8s.io/pause:3.9 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.7.13 containerd_wasm_shims_url=https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.11.1/containerd-wasm-shims-<RTVERSION>-<SHIM>-linux-x86_64.tar.gz containerd_wasm_shims_version=v0.11.1 containerd_wasm_shims_sha256={"lunatic":"7054bc882db755ce5f3ded46d114bfd4e0a318e437fa18a2601295d20b616b32","slight":"a6ea87d965037933a7d9edb5e20cfc175265c8e1ca92a16535f1f3c3f376f5b0","spin":"dcffedb8e4d2f585a851b3de489fa1e8a0054ec0ad72cf111c623623919245d0","wws":"e917f90692d798d80873aa0f37990c7d652f2846129d64fecbfd41ffa77799b8"} containerd_wasm_shims_runtimes="" containerd_wasm_shims_runtime_versions="{"lunatic":"v1","slight":"v1","spin":"v2","wws":"v1"}" crictl_url=https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.26.0/crictl-v1.26.0-linux-amd64.tar.gz crictl_sha256=https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.26.0/crictl-v1.26.0-linux-amd64.tar.gz.sha256 crictl_source_type=pkg custom_role_names="" firstboot_custom_roles_pre="" firstboot_custom_roles_post="" node_custom_roles_pre="" node_custom_roles_post="" disable_public_repos=false extra_debs="" extra_repos="" extra_rpms="" http_proxy= https_proxy= kubeadm_template=etc/kubeadm.yml kubernetes_apiserver_port=6443 kubernetes_cni_http_source=https://github.com/containernetworking/plugins/releases/download kubernetes_cni_http_checksum=sha256:https://storage.googleapis.com/k8s-artifacts-cni/release/v1.2.0/cni-plugins-linux-amd64-v1.2.0.tgz.sha256 kubernetes_goarch=amd64 kubernetes_http_source=https://dl.k8s.io/release kubernetes_container_registry=registry.k8s.io kubernetes_rpm_repo=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/ kubernetes_rpm_gpg_key=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml.key kubernetes_rpm_gpg_check=True kubernetes_deb_repo=https://pkgs.k8s.io/core:/stable:/v1.26/deb/ kubernetes_deb_gpg_key=https://pkgs.k8s.io/core:/stable:/v1.26/deb/Release.key kubernetes_cni_deb_version= kubernetes_cni_rpm_version= kubernetes_cni_semver=v1.2.0 kubernetes_cni_source_type=pkg kubernetes_semver=v1.26.7 kubernetes_source_type=pkg kubernetes_load_additional_imgs=false kubernetes_deb_version=1.26.7-1.1 kubernetes_rpm_version=1.26.7 no_proxy= pip_conf_file= python_path= redhat_epel_rpm=https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm epel_rpm_gpg_key= reenable_public_repos=true remove_extra_repos=false systemd_prefix=/usr/lib/systemd sysusr_prefix=/usr sysusrlocal_prefix=/usr/local load_additional_components=false additional_registry_images=false additional_registry_images_list= additional_url_images=false additional_url_images_list= additional_executables=false additional_executables_list= additional_executables_destination_path= additional_s3=false build_target=virt amazon_ssm_agent_rpm= enable_containerd_audit= kubernetes_enable_automatic_resource_sizing= --extra-vars guestinfo_datasource_slug=https://raw.githubusercontent.com/vmware/cloud-init-vmware-guestinfo guestinfo_datasource_ref=v1.4.1 guestinfo_datasource_script=https://raw.githubusercontent.com/vmware/cloud-init-vmware-guestinfo/v1.4.1/install.sh --extra-vars  --scp-extra-args "-O" -e ansible_ssh_private_key_file=/tmp/ansible-key2121582285 -i /tmp/packer-provisioner-ansible4033390761 /home/prow/go/src/sigs.k8s.io/image-*****/images/capi/ansible/firstboot.yml�[0m
�[0;32m    vsphere-clone:�[0m
�[0;32m    vsphere-clone: PLAY [all] *********************************************************************�[0m
�[0;32m    vsphere-clone: [WARNING]: Platform linux on host default is using the discovered Python�[0m
�[0;32m    vsphere-clone: interpreter at /usr/bin/python3.10, but future installation of another Python�[0m
�[0;32m    vsphere-clone: interpreter could change the meaning of that path. See�[0m
�[0;32m    vsphere-clone: https://docs.ansible.com/ansible-�[0m
�[0;32m    vsphere-clone: core/2.15/reference_appendices/interpreter_discovery.html for more information.�[0m
�[0;32m    vsphere-clone:�[0m
�[0;32m    vsphere-clone: TASK [Gathering Facts] *********************************************************�[0m
�[0;32m    vsphere-clone: ok: [default]�[0m
�[0;32m    vsphere-clone:�[0m
�[0;32m    vsphere-clone: TASK [ansible.builtin.include_role : firstboot] ********************************�[0m
�[0;32m    vsphere-clone:�[0m
�[0;32m    vsphere-clone: TASK [setup : Add bash_profile] ************************************************�[0m
�[0;32m    vsphere-clone: changed: [default]�[0m
�[0;32m    vsphere-clone:�[0m
�[0;32m    vsphere-clone: TASK [setup : Perform a tdnf distro-sync] **************************************�[0m
�[0;32m    vsphere-clone: fatal: [default]: FAILED! => {"changed": true, "cmd": ["tdnf", "distro-sync", "-y", "--refresh"], "delta": "0:00:14.983168", "end": "2024-04-08 10:25:36.706671", "msg": "non-zero return code", "rc": 234, "start": "2024-04-08 10:25:21.723503", "stderr": "Error verifying signature of: /var/cache/tdnf/photon-updates/rpms/x86_64/linux-5.10.212-4.ph4.x86_64.rpm\nError processing package: x86_64/linux-5.10.212-4.ph4.x86_64.rpm\nError(1514) : RPM is signed but failed to match with known keys. Use --nogpgcheck to ignore.", "stderr_lines": ["Error verifying signature of: /var/cache/tdnf/photon-updates/rpms/x86_64/linux-5.10.212-4.ph4.x86_64.rpm", "Error processing package: x86_64/linux-5.10.212-4.ph4.x86_64.rpm", "Error(1514) : RPM is signed but failed to match with known keys. Use --nogpgcheck to ignore."], "stdout": "Refreshing metadata for: 'VMware Photon Extras 4.0 (x86_64)'\nRefreshing metadata for: 'VMware Photon Linux 4.0 (x86_64)'\nRefreshing metadata for: 'VMware Photon Linux 4.0 (x86_64) Updates'\n\nInstalling:\nlzo                      x86_64       2.10-1.ph4       photon-release 143.85k 147304\nslang                    x86_64       2.3.2-1.ph4      photon-release   3.12M 3273998\nnewt                     x86_64       0.52.21-1.ph4    photon-release 179.54k 183846\nbtrfs-progs              x86_64       5.7-3.ph4        photon-updates   4.88M 5122206\nlibatomic_ops            x86_64       7.6.10-2.ph4     photon-updates  49.93k 51124\nlibnftnl                 x86_64       1.2.1-1.ph4      photon-updates 225.46k 230874\nlibmnl                   x86_64       1.0.4-4.ph4      photon-updates  27.41k 28063\njansson                  x86_64       2.13.1-1.ph4     photon-release  84.06k 86078\nlibsepol                 x86_64       3.3-1.ph4        photon-updates 764.12k 782456\nchkconfig                x86_64       1.9-3.ph4        photon-updates 737.53k 755232\nkbd                      x86_64       2.3.0-2.ph4      photon-updates   3.07M 3218981\nshadow-libs              x86_64       4.13-2.ph4       photon-updates 158.34k 162145\niptables-libs            x86_64       1.8.9-4.ph4      photon-updates 154.69k 158405\ncoreutils-selinux        x86_64       9.1-2.ph4        photon-updates   6.97M 7311161\npython3-pyserial         noarch       3.5-1.ph4        photon-updates 853.34k 873822\n\nTotal installed size:  21.35M 22385695\n\nUpgrading:\nzstd-libs                x86_64       1.5.2-2.ph4      photon-updates 934.59k 957016\nzstd                     x86_64       1.5.2-2.ph4      photon-updates   1.13M 1183428\nzlib                     x86_64       1.2.11-6.ph4     photon-updates 112.18k 114870\nxz-libs                  x86_64       5.2.5-2.ph4      photon-updates 165.26k 169224\nxz                       x86_64       5.2.5-2.ph4      photon-updates 167.07k 171084\nvim                      x86_64       9.0.2142-1.ph4   photon-updates   3.68M 3857049\nutil-linux-libs          x86_64       2.37.4-1.ph4     photon-updates 765.03k 783386\nutil-linux               x86_64       2.37.4-1.ph4     photon-updates   6.58M 6895245\ntzdata                   noarch       2022g-1.ph4      photon-updates   1.37M 1441626\nsudo                     x86_64       1.9.15p5-1.ph4   photon-updates   8.06M 8449277\nsqlite-libs              x86_64       3.38.5-4.ph4     photon-updates   1.29M 1355499\nshadow-tools             x86_64       4.13-2.ph4       photon-updates 170.83k 174928\nshadow                   x86_64       4.13-2.ph4       photon-updates   2.07M 2171937\nsed                      x86_64       4.8-3.ph4        photon-updates 194.55k 199218\nprocps-ng                x86_64       3.3.17-2.ph4     photon-updates   1.40M 1469820\npkg-config               x86_64       0.29.2-5.ph4     photon-updates 618.32k 633156\nphoton-repos             noarch       4.0-3.ph4        photon-updates   4.10k 4199\npcre-libs                x86_64       8.44-4.ph4       photon-updates 480.89k 492434\npcre                     x86_64       8.44-4.ph4       photon-updates   1.10M 1149057\nnss-libs                 x86_64       3.72-5.ph4       photon-updates   2.47M 2593686\nnss                      x86_64       3.72-5.ph4       photon-updates   1.74M 1826853\nnet-tools                x86_64       1.60-13.ph4      photon-updates 573.62k 587388\nlinux                    x86_64       5.10.212-4.ph4   photon-updates  37.52M 39342720\nlibxslt                  x86_64       1.1.34-7.ph4     photon-updates 398.89k 408463\nlibxml2-devel            x86_64       2.9.12-13.ph4    photon-updates   6.24M 6545930\nlibxml2                  x86_64       2.9.12-13.ph4    photon-updates   1.55M 1629444\nlibunistring             x86_64       0.9.10-2.ph4     photon-updates   4.39M 4600735\nlibtirpc                 x86_64       1.2.6-4.ph4      photon-updates 197.58k 202322\nlibtasn1                 x86_64       4.14-2.ph4       photon-updates 127.08k 130135\nlibselinux               x86_64       3.3-1.ph4        photon-updates 170.36k 174450\nlibseccomp               x86_64       2.5.0-3.ph4      photon-updates 167.71k 171735\nlibpwquality             x86_64       1.4.2-3.ph4      photon-updates 370.77k 379668\nlibmspack                x86_64       0.10.1alpha-3.ph4 photon-updates  79.81k 81722\nlibmicrohttpd            x86_64       0.9.76-2.ph4     photon-updates 173.06k 177216\nlibgcrypt                x86_64       1.9.4-2.ph4      photon-updates   1.31M 1371731\nlibstdc++                x86_64       10.5.0-1.ph4     photon-updates   1.98M 2071594\nlibgcc                   x86_64       10.5.0-1.ph4     photon-updates 100.79k 103208\nlibcap                   x86_64       2.43-3.ph4       photon-updates 135.08k 138324\nlibacl                   x86_64       2.2.53-2.ph4     photon-updates  39.70k 40650\nkrb5                     x86_64       1.17-10.ph4      photon-updates   3.34M 3503602\niputils                  x86_64       20200821-3.ph4   photon-updates 335.93k 343993\niptables                 x86_64       1.8.9-4.ph4      photon-updates   2.28M 2394617\niproute2                 x86_64       5.10.0-4.ph4     photon-updates   2.71M 2838059\ninitramfs                x86_64       2.0-7.ph4        photon-updates  50.00b 50\ngzip                     x86_64       1.12-2.ph4       photon-updates 146.04k 149549\nguile                    x86_64       2.0.13-4.ph4     photon-updates  10.73M 11255203\ngrub2-efi-image          x86_64       2.06-12.ph4      photon-updates   1.04M 1086208\ngrep                     x86_64       3.4-2.ph4        photon-updates 234.56k 240186\ngnutls                   x86_64       3.7.10-3.ph4     photon-updates   4.09M 4288567\nglibc-iconv              x86_64       2.32-16.ph4      photon-updates   7.90M 8285233\nglibc                    x86_64       2.32-16.ph4      photon-updates   8.83M 9262762\ngc                       x86_64       8.0.4-2.ph4      photon-updates 525.82k 538435\nfile-libs                x86_64       5.39-3.ph4       photon-updates   6.52M 6837585\nfile                     x86_64       5.39-3.ph4       photon-updates  43.75k 44799\nexpat-libs               x86_64       2.4.9-2.ph4      photon-updates 193.40k 198041\nexpat                    x86_64       2.4.9-2.ph4      photon-updates  39.55k 40504\nelfutils-libelf          x86_64       0.181-7.ph4      photon-updates 105.10k 107623\nelfutils                 x86_64       0.181-7.ph4      photon-updates   3.56M 3728269\ne2fsprogs-libs           x86_64       1.46.5-2.ph4     photon-updates  58.42k 59822\ne2fsprogs                x86_64       1.46.5-2.ph4     photon-updates   2.12M 2221049\ndracut                   x86_64       050-13.ph4       photon-updates 805.89k 825234\ndhcp-libs                x86_64       4.4.2-6.ph4      photon-updates 959.87k 982902\ndhcp-client              x86_64       4.4.2-6.ph4      photon-updates   3.85M 4040741\ndbus                     x86_64       1.13.18-4.ph4    photon-updates 956.31k 979262\ncracklib-dicts           x86_64       2.9.7-4.ph4      photon-updates   8.87M 9299226\ncpio                     x86_64       2.13-9.ph4       photon-updates 452.10k 462948\nrpm                      x86_64       4.16.1.3-18.ph4  photon-updates 370.26k 379146\nlua                      x86_64       5.4.4-3.ph4      photon-updates 773.58k 792141\npython3-pyOpenSSL        noarch       23.2.0-2.ph4     photon-updates 408.68k 418489\npython3-xml              x86_64       3.10.11-8.ph4    photon-updates 360.79k 369451\npython3                  x86_64       3.10.11-8.ph4    photon-updates   3.46M 3627187\nsystemd-udev             x86_64       247.13-12.ph4    photon-updates   8.22M 8623629\nsystemd-rpm-macros       noarch       247.13-12.ph4    photon-updates   5.45k 5581\nsystemd-pam              x86_64       247.13-12.ph4    photon-updates 452.88k 463749\nsystemd-libs             x86_64       247.13-12.ph4    photon-updates   1.60M 1673908\nglib                     x86_64       2.68.4-1.ph4     photon-updates   3.53M 3705347\ntdnf-cli-libs            x86_64       3.3.9-3.ph4      photon-updates  66.64k 68235\nrpm-libs                 x86_64       4.16.1.3-18.ph4  photon-updates 828.42k 848297\nlibsolv                  x86_64       0.7.19-6.ph4     photon-updates   1.10M 1153107\ncurl                     x86_64       8.1.2-8.ph4      photon-updates 279.49k 286202\ncurl-libs                x86_64       8.1.2-8.ph4      photon-updates 680.21k 696536\nlibarchive               x86_64       3.4.3-10.ph4     photon-updates 882.09k 903260\nlibssh2                  x86_64       1.11.0-1.ph4     photon-updates 282.62k 289408\nopen-vm-tools            x86_64       12.3.5-1.ph4     photon-updates   3.16M 3313637\npython3-cryptography     x86_64       41.0.7-2.ph4     photon-updates   4.20M 4399288\npython3-libs             x86_64       3.10.11-8.ph4    photon-updates  26.87M 28174096\nsystemd                  x86_64       247.13-12.ph4    photon-updates  12.92M 13552710\ntdnf                     x86_64       3.3.9-3.ph4      photon-updates 267.71k 274140\nxmlsec1                  x86_64       1.2.30-6.ph4     photon-updates 1017.38k 1041800\nopenssh-clients          x86_64       8.9p1-7.ph4      photon-updates   4.84M 5073525\ncracklib                 x86_64       2.9.7-4.ph4      photon-updates  44.59k 45657\nopenssh-server           x86_64       8.9p1-7.ph4      photon-updates   1.14M 1196443\nopenssl                  x86_64       3.0.13-3.ph4     photon-updates   6.21M 6515163\npython3-urllib3          noarch       1.25.11-5.ph4    photon-updates 645.24k 660727\npython3-certifi          noarch       2023.11.17-1.ph4 photon-updates 295.26k 302351\npython3-markupsafe       x86_64       2.1.1-1.ph4      photon-updates  54.49k 55801\npython3-setuptools       noarch       65.5.1-3.ph4     photon-updates   7.96M 8348899\npython3-requests         noarch       2.26.0-2.ph4     photon-updates 469.64k 480911\npython3-jinja2           noarch       3.1.2-1.ph4      photon-updates   1.28M 1342483\npython3-PyYAML           x86_64       5.4.1-3.ph4      photon-updates 843.04k 863278\ncloud-init               noarch       23.4-4.ph4       photon-updates   5.78M 6065667\nca-certificates-pki      x86_64       20201001-4.ph4   photon-updates 766.32k 784712\nca-certificates          x86_64       20201001-4.ph4   photon-updates 720.60k 737899\nncurses-terminfo         x86_64       6.2-6.ph4        photon-updates   2.01M 2105268\nncurses                  x86_64       6.2-6.ph4        photon-updates 363.42k 372138\nncurses-libs             x86_64       6.2-6.ph4        photon-updates 625.83k 640848\nbash                     x86_64       5.0-4.ph4        photon-updates   1.85M 1935927\nautogen-libopts          x86_64       5.18.16-4.ph4    photon-updates 148.20k 151754\nattr                     x86_64       2.4.48-2.ph4     photon-updates  91.93k 94133\nLinux-PAM                x86_64       1.5.3-2.ph4      photon-updates   1.16M 1220417\n\nTotal installed size: 269.24M 282318241\n\nRemoving:\ncoreutils                x86_64       8.32-3.ph4       @System        6.47M 6785360\n\nTotal installed size:   6.47M 6785360\n\nObsoleting:\ncoreutils                x86_64       8.32-3.ph4       @System        6.47M 6785360\n\nTotal installed size:   6.47M 6785360\n\nDownloading:\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nimporting key from file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY", "stdout_lines": ["Refreshing metadata for: 'VMware Photon Extras 4.0 (x86_64)'", "Refreshing metadata for: 'VMware Photon Linux 4.0 (x86_64)'", "Refreshing metadata for: 'VMware Photon Linux 4.0 (x86_64) Updates'", "", "Installing:", "lzo                      x86_64       2.10-1.ph4       photon-release 143.85k 147304", "slang                    x86_64       2.3.2-1.ph4      photon-release   3.12M 3273998", "newt                     x86_64       0.52.21-1.ph4    photon-release 179.54k 183846", "btrfs-progs              x86_64       5.7-3.ph4        photon-updates   4.88M 5122206", "libatomic_ops            x86_64       7.6.10-2.ph4     photon-updates  49.93k 51124", "libnftnl                 x86_64       1.2.1-1.ph4      photon-updates 225.46k 230874", "libmnl                   x86_64       1.0.4-4.ph4      photon-updates  27.41k 28063", "jansson                  x86_64       2.13.1-1.ph4     photon-release  84.06k 86078", "libsepol                 x86_64       3.3-1.ph4        photon-updates 764.12k 782456", "chkconfig                x86_64       1.9-3.ph4        photon-updates 737.53k 755232", "kbd                      x86_64       2.3.0-2.ph4      photon-updates   3.07M 3218981", "shadow-libs              x86_64       4.13-2.ph4       photon-updates 158.34k 162145", "iptables-libs            x86_64       1.8.9-4.ph4      photon-updates 154.69k 158405", "coreutils-selinux        x86_64       9.1-2.ph4        photon-updates   6.97M 7311161", "python3-pyserial         noarch       3.5-1.ph4        photon-updates 853.34k 873822", "", "Total installed size:  21.35M 22385695", "", "Upgrading:", "zstd-libs                x86_64       1.5.2-2.ph4      photon-updates 934.59k 957016", "zstd                     x86_64       1.5.2-2.ph4      photon-updates   1.13M 1183428", "zlib                     x86_64       1.2.11-6.ph4     photon-updates 112.18k 114870", "xz-libs                  x86_64       5.2.5-2.ph4      photon-updates 165.26k 169224", "xz                       x86_64       5.2.5-2.ph4      photon-updates 167.07k 171084", "vim                      x86_64       9.0.2142-1.ph4   photon-updates   3.68M 3857049", "util-linux-libs          x86_64       2.37.4-1.ph4     photon-updates 765.03k 783386", "util-linux               x86_64       2.37.4-1.ph4     photon-updates   6.58M 6895245", "tzdata                   noarch       2022g-1.ph4      photon-updates   1.37M 1441626", "sudo                     x86_64       1.9.15p5-1.ph4   photon-updates   8.06M 8449277", "sqlite-libs              x86_64       3.38.5-4.ph4     photon-updates   1.29M 1355499", "shadow-tools             x86_64       4.13-2.ph4       photon-updates 170.83k 174928", "shadow                   x86_64       4.13-2.ph4       photon-updates   2.07M 2171937", "sed                      x86_64       4.8-3.ph4        photon-updates 194.55k 199218", "procps-ng                x86_64       3.3.17-2.ph4     photon-updates   1.40M 1469820", "pkg-config               x86_64       0.29.2-5.ph4     photon-updates 618.32k 633156", "photon-repos             noarch       4.0-3.ph4        photon-updates   4.10k 4199", "pcre-libs                x86_64       8.44-4.ph4       photon-updates 480.89k 492434", "pcre                     x86_64       8.44-4.ph4       photon-updates   1.10M 1149057", "nss-libs                 x86_64       3.72-5.ph4       photon-updates   2.47M 2593686", "nss                      x86_64       3.72-5.ph4       photon-updates   1.74M 1826853", "net-tools                x86_64       1.60-13.ph4      photon-updates 573.62k 587388", "linux                    x86_64       5.10.212-4.ph4   photon-updates  37.52M 39342720", "libxslt                  x86_64       1.1.34-7.ph4     photon-updates 398.89k 408463", "libxml2-devel            x86_64       2.9.12-13.ph4    photon-updates   6.24M 6545930", "libxml2                  x86_64       2.9.12-13.ph4    photon-updates   1.55M 1629444", "libunistring             x86_64       0.9.10-2.ph4     photon-updates   4.39M 4600735", "libtirpc                 x86_64       1.2.6-4.ph4      photon-updates 197.58k 202322", "libtasn1                 x86_64       4.14-2.ph4       photon-updates 127.08k 130135", "libselinux               x86_64       3.3-1.ph4        photon-updates 170.36k 174450", "libseccomp               x86_64       2.5.0-3.ph4      photon-updates 167.71k 171735", "libpwquality             x86_64       1.4.2-3.ph4      photon-updates 370.77k 379668", "libmspack                x86_64       0.10.1alpha-3.ph4 photon-updates  79.81k 81722", "libmicrohttpd            x86_64       0.9.76-2.ph4     photon-updates 173.06k 177216", "libgcrypt                x86_64       1.9.4-2.ph4      photon-updates   1.31M 1371731", "libstdc++                x86_64       10.5.0-1.ph4     photon-updates   1.98M 2071594", "libgcc                   x86_64       10.5.0-1.ph4     photon-updates 100.79k 103208", "libcap                   x86_64       2.43-3.ph4       photon-updates 135.08k 138324", "libacl                   x86_64       2.2.53-2.ph4     photon-updates  39.70k 40650", "krb5                     x86_64       1.17-10.ph4      photon-updates   3.34M 3503602", "iputils                  x86_64       20200821-3.ph4   photon-updates 335.93k 343993", "iptables                 x86_64       1.8.9-4.ph4      photon-updates   2.28M 2394617", "iproute2                 x86_64       5.10.0-4.ph4     photon-updates   2.71M 2838059", "initramfs                x86_64       2.0-7.ph4        photon-updates  50.00b 50", "gzip                     x86_64       1.12-2.ph4       photon-updates 146.04k 149549", "guile                    x86_64       2.0.13-4.ph4     photon-updates  10.73M 11255203", "grub2-efi-image          x86_64       2.06-12.ph4      photon-updates   1.04M 1086208", "grep                     x86_64       3.4-2.ph4        photon-updates 234.56k 240186", "gnutls                   x86_64       3.7.10-3.ph4     photon-updates   4.09M 4288567", "glibc-iconv              x86_64       2.32-16.ph4      photon-updates   7.90M 8285233", "glibc                    x86_64       2.32-16.ph4      photon-updates   8.83M 9262762", "gc                       x86_64       8.0.4-2.ph4      photon-updates 525.82k 538435", "file-libs                x86_64       5.39-3.ph4       photon-updates   6.52M 6837585", "file                     x86_64       5.39-3.ph4       photon-updates  43.75k 44799", "expat-libs               x86_64       2.4.9-2.ph4      photon-updates 193.40k 198041", "expat                    x86_64       2.4.9-2.ph4      photon-updates  39.55k 40504", "elfutils-libelf          x86_64       0.181-7.ph4      photon-updates 105.10k 107623", "elfutils                 x86_64       0.181-7.ph4      photon-updates   3.56M 3728269", "e2fsprogs-libs           x86_64       1.46.5-2.ph4     photon-updates  58.42k 59822", "e2fsprogs                x86_64       1.46.5-2.ph4     photon-updates   2.12M 2221049", "dracut                   x86_64       050-13.ph4       photon-updates 805.89k 825234", "dhcp-libs                x86_64       4.4.2-6.ph4      photon-updates 959.87k 982902", "dhcp-client              x86_64       4.4.2-6.ph4      photon-updates   3.85M 4040741", "dbus                     x86_64       1.13.18-4.ph4    photon-updates 956.31k 979262", "cracklib-dicts           x86_64       2.9.7-4.ph4      photon-updates   8.87M 9299226", "cpio                     x86_64       2.13-9.ph4       photon-updates 452.10k 462948", "rpm                      x86_64       4.16.1.3-18.ph4  photon-updates 370.26k 379146", "lua                      x86_64       5.4.4-3.ph4      photon-updates 773.58k 792141", "python3-pyOpenSSL        noarch       23.2.0-2.ph4     photon-updates 408.68k 418489", "python3-xml              x86_64       3.10.11-8.ph4    photon-updates 360.79k 369451", "python3                  x86_64       3.10.11-8.ph4    photon-updates   3.46M 3627187", "systemd-udev             x86_64       247.13-12.ph4    photon-updates   8.22M 8623629", "systemd-rpm-macros       noarch       247.13-12.ph4    photon-updates   5.45k 5581", "systemd-pam              x86_64       247.13-12.ph4    photon-updates 452.88k 463749", "systemd-libs             x86_64       247.13-12.ph4    photon-updates   1.60M 1673908", "glib                     x86_64       2.68.4-1.ph4     photon-updates   3.53M 3705347", "tdnf-cli-libs            x86_64       3.3.9-3.ph4      photon-updates  66.64k 68235", "rpm-libs                 x86_64       4.16.1.3-18.ph4  photon-updates 828.42k 848297", "libsolv                  x86_64       0.7.19-6.ph4     photon-updates   1.10M 1153107", "curl                     x86_64       8.1.2-8.ph4      photon-updates 279.49k 286202", "curl-libs                x86_64       8.1.2-8.ph4      photon-updates 680.21k 696536", "libarchive               x86_64       3.4.3-10.ph4     photon-updates 882.09k 903260", "libssh2                  x86_64       1.11.0-1.ph4     photon-updates 282.62k 289408", "open-vm-tools            x86_64       12.3.5-1.ph4     photon-updates   3.16M 3313637", "python3-cryptography     x86_64       41.0.7-2.ph4     photon-updates   4.20M 4399288", "python3-libs             x86_64       3.10.11-8.ph4    photon-updates  26.87M 28174096", "systemd                  x86_64       247.13-12.ph4    photon-updates  12.92M 13552710", "tdnf                     x86_64       3.3.9-3.ph4      photon-updates 267.71k 274140", "xmlsec1                  x86_64       1.2.30-6.ph4     photon-updates 1017.38k 1041800", "openssh-clients          x86_64       8.9p1-7.ph4      photon-updates   4.84M 5073525", "cracklib                 x86_64       2.9.7-4.ph4      photon-updates  44.59k 45657", "openssh-server           x86_64       8.9p1-7.ph4      photon-updates   1.14M 1196443", "openssl                  x86_64       3.0.13-3.ph4     photon-updates   6.21M 6515163", "python3-urllib3          noarch       1.25.11-5.ph4    photon-updates 645.24k 660727", "python3-certifi          noarch       2023.11.17-1.ph4 photon-updates 295.26k 302351", "python3-markupsafe       x86_64       2.1.1-1.ph4      photon-updates  54.49k 55801", "python3-setuptools       noarch       65.5.1-3.ph4     photon-updates   7.96M 8348899", "python3-requests         noarch       2.26.0-2.ph4     photon-updates 469.64k 480911", "python3-jinja2           noarch       3.1.2-1.ph4      photon-updates   1.28M 1342483", "python3-PyYAML           x86_64       5.4.1-3.ph4      photon-updates 843.04k 863278", "cloud-init               noarch       23.4-4.ph4       photon-updates   5.78M 6065667", "ca-certificates-pki      x86_64       20201001-4.ph4   photon-updates 766.32k 784712", "ca-certificates          x86_64       20201001-4.ph4   photon-updates 720.60k 737899", "ncurses-terminfo         x86_64       6.2-6.ph4        photon-updates   2.01M 2105268", "ncurses                  x86_64       6.2-6.ph4        photon-updates 363.42k 372138", "ncurses-libs             x86_64       6.2-6.ph4        photon-updates 625.83k 640848", "bash                     x86_64       5.0-4.ph4        photon-updates   1.85M 1935927", "autogen-libopts          x86_64       5.18.16-4.ph4    photon-updates 148.20k 151754", "attr                     x86_64       2.4.48-2.ph4     photon-updates  91.93k 94133", "Linux-PAM                x86_64       1.5.3-2.ph4      photon-updates   1.16M 1220417", "", "Total installed size: 269.24M 282318241", "", "Removing:", "coreutils                x86_64       8.32-3.ph4       @System        6.47M 6785360", "", "Total installed size:   6.47M 6785360", "", "Obsoleting:", "coreutils                x86_64       8.32-3.ph4       @System        6.47M 6785360", "", "Total installed size:   6.47M 6785360", "", "Downloading:", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "importing key from file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY"]}�[0m
�[0;32m    vsphere-clone:�[0m
�[0;32m    vsphere-clone: PLAY RECAP *********************************************************************�[0m
�[0;32m    vsphere-clone: default                    : ok=2    changed=1    unreachable=0    failed=1    skipped=37   rescued=0    ignored=0�[0m
�[0;32m    vsphere-clone:�[0m
�[1;32m==> vsphere-clone: Provisioning step had errors: Running the cleanup provisioner, if present...�[0m
�[1;32m==> vsphere-clone: Power off VM...�[0m
�[1;32m==> vsphere-clone: Destroying VM...�[0m
�[1;31mBuild 'vsphere-clone' errored after 2 minutes 4 seconds: Error executing Ansible: Non-zero exit status: exit status 2�[0m

==> Wait completed after 2 minutes 4 seconds

==> Some builds didn't complete successfully and had errors:
--> vsphere-clone: Error executing Ansible: Non-zero exit status: exit status 2

==> Builds finished but no artifacts were created.
make: *** [Makefile:465: build-node-ova-vsphere-clone-photon-4] Error 1

Anything else you would like to add?

This is currently blocking all PRs from being merged.

---

/kind bug

[mboersma]

Here is a recent example of this failure: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_image-builder/1439/pull-ova-all/1779968309270155264/artifacts/photon-4.log

AFAICT there's a signed Photon 4.0 package that doesn't pass the GPG check here. But I can't reproduce this by doing manually what the ansible task does:

% docker run -it photon:4.0
root [ / ]# tdnf distro-sync -y --refresh

image-builder/images/capi/ansible/roles/setup/tasks/photon.yml

Lines 25 to 28 in fc50721

	- name: Perform a tdnf distro-sync
	ansible.builtin.command: tdnf distro-sync -y --refresh
	register: distro
	changed_when: '"Nothing to do" not in distro.stderr'

@chrischdi @killianmuldoon @sbueringer any ideas?

[chrischdi]

I was able to reproduce it in a container using docker run -ti --rm photon:4.0-20220114 (which somewhat matches the age of the iso).

TLDR: the old gpg key seems to expired and they have new ones.