failed to scan all layer contents: rhel: unable to create a mappingFile object

[majewsky]

Description of Problem / Feature Request

Since upgrading to Clair 4.6.0, we're sometimes seeing the indexing error failed to scan all layer contents: rhel: unable to create a mappingFile object pop up at random. This is not reproducible. Upon deleting the index report and indexing again, the error does not show up again.

One of the images in question that this happened on is index.docker.io/curlimages/curl@sha256:17468885fb8a20cd6bc25316f8267492c4d758ba63a6838ce74b9a0ffe4d2e90 (the amd64 variant of the image index tagged as latest as of the time of this writing), so I recommend to use this image for testing. We only saw the unable to create a mappingFile object in one of our regional deployments (out of 15 regions), so that demonstrates the stochastic nature of the issue.

Expected Outcome

Indexing should not fail.

Actual Outcome

clair=# SELECT scan_result FROM indexreport WHERE manifest_id IN (SELECT id FROM manifest WHERE hash = 'sha256:17468885fb8a20cd6bc25316f8267492c4d758ba63a6838ce74b9a0ffe4d2e90');
-[ RECORD 1 ]----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
scan_result | {"err": "failed to scan all layer contents: rhel: unable to create a mappingFile object", "state": "IndexError", "success": false, "packages": {}, "repository": {}, "environments": {}, "distributions": {}, "manifest_hash": "sha256:17468885fb8a20cd6bc25316f8267492c4d758ba63a6838ce74b9a0ffe4d2e90"}

After deleting this index report and reindexing, we get the following index report:

{"err": "", "state": "IndexFinished", "success": true, "packages": {"1626": {"id": "1626", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "libc-utils", "source": {"id": "1625", "cpe": "", "kind": "source", "name": "libc-dev", "version": "0.7.2-r3", "normalized_version": ""}, "version": "0.7.2-r3", "normalized_version": ""}, "207007": {"id": "207007", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "alpine-keys", "source": {"id": "207006", "cpe": "", "kind": "source", "name": "alpine-keys", "version": "2.4-r1", "normalized_version": ""}, "version": "2.4-r1", "normalized_version": ""}, "463765": {"id": "463765", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "apk-tools", "source": {"id": "463764", "cpe": "", "kind": "source", "name": "apk-tools", "version": "2.12.9-r3", "normalized_version": ""}, "version": "2.12.9-r3", "normalized_version": ""}, "463767": {"id": "463767", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "scanelf", "source": {"id": "463766", "cpe": "", "kind": "source", "name": "pax-utils", "version": "1.3.4-r0", "normalized_version": ""}, "version": "1.3.4-r0", "normalized_version": ""}, "529795": {"id": "529795", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "busybox", "source": {"id": "529794", "cpe": "", "kind": "source", "name": "busybox", "version": "1.35.0-r17", "normalized_version": ""}, "version": "1.35.0-r17", "normalized_version": ""}, "529801": {"id": "529801", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "ca-certificates-bundle", "source": {"id": "529800", "cpe": "", "kind": "source", "name": "ca-certificates", "version": "20220614-r0", "normalized_version": ""}, "version": "20220614-r0", "normalized_version": ""}, "529807": {"id": "529807", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "ssl_client", "source": {"id": "529794", "cpe": "", "kind": "source", "name": "busybox", "version": "1.35.0-r17", "normalized_version": ""}, "version": "1.35.0-r17", "normalized_version": ""}, "529809": {"id": "529809", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "zlib", "source": {"id": "529808", "cpe": "", "kind": "source", "name": "zlib", "version": "1.2.12-r3", "normalized_version": ""}, "version": "1.2.12-r3", "normalized_version": ""}, "571145": {"id": "571145", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "alpine-baselayout-data", "source": {"id": "571144", "cpe": "", "kind": "source", "name": "alpine-baselayout", "version": "3.2.0-r23", "normalized_version": ""}, "version": "3.2.0-r23", "normalized_version": ""}, "571147": {"id": "571147", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "alpine-baselayout", "source": {"id": "571144", "cpe": "", "kind": "source", "name": "alpine-baselayout", "version": "3.2.0-r23", "normalized_version": ""}, "version": "3.2.0-r23", "normalized_version": ""}, "629525": {"id": "629525", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "musl", "source": {"id": "629524", "cpe": "", "kind": "source", "name": "musl", "version": "1.2.3-r2", "normalized_version": ""}, "version": "1.2.3-r2", "normalized_version": ""}, "629527": {"id": "629527", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "musl-utils", "source": {"id": "629524", "cpe": "", "kind": "source", "name": "musl", "version": "1.2.3-r2", "normalized_version": ""}, "version": "1.2.3-r2", "normalized_version": ""}, "696381": {"id": "696381", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "libcrypto1.1", "source": {"id": "696380", "cpe": "", "kind": "source", "name": "openssl", "version": "1.1.1t-r0", "normalized_version": ""}, "version": "1.1.1t-r0", "normalized_version": ""}, "696383": {"id": "696383", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "libssl1.1", "source": {"id": "696380", "cpe": "", "kind": "source", "name": "openssl", "version": "1.1.1t-r0", "normalized_version": ""}, "version": "1.1.1t-r0", "normalized_version": ""}}, "repository": {}, "environments": {"1626": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "207007": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "463765": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "463767": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "529795": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "529801": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "529807": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "529809": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "571145": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "571147": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "629525": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "629527": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "696381": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "696383": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}]}, "distributions": {"283": {"id": "283", "cpe": "", "did": "alpine", "arch": "", "name": "Alpine Linux", "version": "3.16", "version_id": "", "pretty_name": "Alpine Linux v3.16", "version_code_name": ""}}, "manifest_hash": "sha256:17468885fb8a20cd6bc25316f8267492c4d758ba63a6838ce74b9a0ffe4d2e90"}

What is funny to me is that this is apparently an Alpine image, but the error indicates that it's related to rhel-specific code.

Environment

• Clair version/image: 4.6.0
• Clair client name/version: Keppel
• Host OS: RHEL 8.7 container on Flatcar 3374.2.4
• Kernel (e.g. uname -a): Linux clair-indexer-64cd54fcb7-9dbzz 5.15.89-flatcar #1 SMP Wed Feb 15 18:00:42 -00 2023 x86_64 x86_64 x86_64 GNU/Linux
• Kubernetes version (use kubectl version): 1.25.6
• Network/Firewall setup: should not be relevant

[yoyz]

Might be in a way link to the following commit :

commit e9f553e0dbe815203d012bcf3c23c4c2505d2cec
Author: crozzy <[email protected]>
Date:   Thu Dec 15 15:04:58 2022 -0800

    rhel: Check that after casting to mappingFile we have a usable mapper

    Currently it is possible that if the repo2cpe_mapping_url or the
    repo2cpe_mapping_file (or indeed if the endpoint is down) that we will
    panic as the mappingFile will cast to a nil. This will check for a nil
    mapper before it gets accessed and error out. This is also an issue
    for name2repos_mapping_url and name2repos_mapping_file used by RHCC
    scanner.

    Signed-off-by: crozzy <[email protected]>

[crozzy]

What is funny to me is that this is apparently an Alpine image, but the error indicates that it's related to rhel-specific code.

Because we don't know anything about the image when we index it, all the (configured) scanners are run on every layer, hence why the rhel specific scanning is running.

The PR that holds the commit mentioned was to avert a panic in the above situation and instead surface. Since then we've changed the instantiation of a number of components and this should be non-issue going forward (quay/claircore#867) as the ingesting of the files should be a lot more infrequent.

[hdonnay]

This should be fixed in 4.7

[IsaacVaughn]

I think this may still be an issue when running in airgapped mode. I am seeing this error nearly constantly in a new instance running behind a firewall. I have followed https://quay.github.io/clair/concepts/updatersandairgap.html to turn on the relevant config settings.

EDIT: I think it's essentially quay/claircore#525, but the current error message is unclear. After implementing the fix from redhat's docs with a local file, I now get "rhcc: unable to create a mappingFile object" instead of "rhel: unable to create a mappingFile object", but at a lower frequency.

EDIT2: It's because I also needed indexer.scanner.package.rhel_containerscanner.name2repos_mapping_file set.

[crozzy]

Hey @IsaacVaughn sounds like the issue is straightened out, is there something specific that could be added to the docs that would have made it less painful?