Revert #284 + fix namespace resolution for quality gate testing

[wagoodman]

We recently saw a passing quality gate for the debian provider:

2023-09-22T15:55:38.9170593Z poetry run ./gate.py
2023-09-22T15:55:40.2988841Z Loading label entries... done! 215 entries loaded
2023-09-22T15:55:40.2990509Z �[95m�[1mValidating with 'pr_vs_latest_via_sbom' �[0m 
2023-09-22T15:55:40.2991659Z 
2023-09-22T15:55:40.2992340Z �[95m�[1mRestricting results to the following DB namespaces: �[0m
2023-09-22T15:55:40.2994084Z  - debian:distro:debian:8
2023-09-22T15:55:40.2994812Z  - debian:distro:debian:9
2023-09-22T15:55:40.2995773Z  - debian:distro:debian:7
2023-09-22T15:55:40.2996224Z 
2023-09-22T15:55:40.2996947Z �[95m�[1mConfiguration: �[0m
2023-09-22T15:55:40.2997627Z    max year limit: 2021
2023-09-22T15:55:40.2998330Z 
2023-09-22T15:55:40.2998965Z Testing image: docker.io/debian@sha256:81e88820a7759038ffa61cff59dfcc12d3772c3a2e75b7cfe963c952da2ad264
2023-09-22T15:55:40.3000064Z     with syft@latest
2023-09-22T15:55:40.3000959Z     with grype@latest+import-db=build/grype-db.tar.gz
2023-09-22T15:55:40.3001960Z     with grype@latest
2023-09-22T15:55:40.3002437Z 
2023-09-22T15:55:40.3003454Z �[95mRunning relative comparison... �[0m
2023-09-22T15:55:40.3004128Z    Results used:
2023-09-22T15:55:40.3005960Z     ├── e2cb1d9e-802b-45a2-87cc-f06191b17afb : grype[custom-db]@v0.69.0 against docker.io/debian@sha256:81e88820a7759038ffa61cff59dfcc12d3772c3a2e75b7cfe963c952da2ad264
2023-09-22T15:55:40.3007749Z     └── 764cf1c6-e946-40bf-bf10-03b66c34f6c0 : [email protected] against docker.io/debian@sha256:81e88820a7759038ffa61cff59dfcc12d3772c3a2e75b7cfe963c952da2ad264
2023-09-22T15:55:40.3008745Z 
2023-09-22T15:55:40.3009199Z no differences found between tool results
2023-09-22T15:55:40.3010029Z 
2023-09-22T15:55:40.3010937Z ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
2023-09-22T15:55:40.3012476Z ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
2023-09-22T15:55:40.3013933Z ▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
2023-09-22T15:55:40.3014803Z 
2023-09-22T15:55:40.3015290Z �[92m�[1mQuality gate passed!�[0m

Even though there were several failures:

...
2023-09-22T15:45:41.7506747Z �[31m[ERROR] ignoring error parsing vuln: CVE-2018-6540, pkg: zziplib, rel: sid�[0m
2023-09-22T15:45:41.7506928Z Traceback (most recent call last):
2023-09-22T15:45:41.7507364Z   File "/home/runner/work/vunnel/vunnel/src/vunnel/providers/debian/parser.py", line 340, in get_vuln_records
2023-09-22T15:45:41.7507603Z     sev = self.get_severity(distro_record)
2023-09-22T15:45:41.7508018Z TypeError: Parser.get_severity() missing 1 required positional argument: 'distro_record'
2023-09-22T15:45:41.7508409Z �[31m[ERROR] ignoring error parsing vuln: CVE-2018-6540, pkg: zziplib, rel: trixie�[0m
2023-09-22T15:45:41.7508600Z Traceback (most recent call last):
2023-09-22T15:45:41.7509036Z   File "/home/runner/work/vunnel/vunnel/src/vunnel/providers/debian/parser.py", line 340, in get_vuln_records
2023-09-22T15:45:41.7509277Z     sev = self.get_severity(distro_record)
2023-09-22T15:45:41.7509702Z TypeError: Parser.get_severity() missing 1 required positional argument: 'distro_record'
2023-09-22T15:45:41.7510109Z �[31m[ERROR] ignoring error parsing vuln: CVE-2018-6541, pkg: zziplib, rel: bookworm�[0m
2023-09-22T15:45:41.7510288Z Traceback (most recent call last):
2023-09-22T15:45:41.7510742Z   File "/home/runner/work/vunnel/vunnel/src/vunnel/providers/debian/parser.py", line 340, in get_vuln_records
2023-09-22T15:45:41.7510978Z     sev = self.get_severity(distro_record)
2023-09-22T15:45:41.7511395Z TypeError: Parser.get_severity() missing 1 required positional argument: 'distro_record'
2023-09-22T15:45:41.7511798Z �[31m[ERROR] ignoring error parsing vuln: CVE-2018-6541, pkg: zziplib, rel: bullseye�[0m
2023-09-22T15:45:41.7511973Z Traceback (most recent call last):
...

This is because of a few reasons:

1. we restrict the namespaces to only those present in the db for the given results
2. the results were only being tested against debian 7 and not future versions

For problem 1, on a good day this is what the logs should show:

Restricting results to the following DB namespaces: 
 - debian:distro:debian:8
 - debian:distro:debian:9
 - debian:distro:debian:10
 - debian:distro:debian:11
 - debian:distro:debian:12
 - debian:distro:debian:13
 - debian:distro:debian:unstable
 - debian:distro:debian:7

However, on the day of the issue we saw:

2023-09-22T15:55:40.2992340Z �[95m�[1mRestricting results to the following DB namespaces: �[0m
2023-09-22T15:55:40.2994084Z  - debian:distro:debian:8
2023-09-22T15:55:40.2994812Z  - debian:distro:debian:9
2023-09-22T15:55:40.2995773Z  - debian:distro:debian:7

This is because the namespaces are gotten from the database under test not the database being used to compare against it:

def get_namespaces_from_db() -> list[str]:
    # open sqlite db at build/vulnerability.db and get a list of unique values in the namespace column
    import sqlite3

    conn = sqlite3.connect("build/vulnerability.db")
    c = conn.cursor()
    c.execute("SELECT DISTINCT namespace FROM vulnerability")
    return [row[0] for row in c.fetchall()]

This behavior has been changed to get the list of expected DBs from a static config file that we use for testing:

❯ make validate
poetry run ./gate.py
Loading label entries... done! 2614 entries loaded
Validating with 'pr_vs_latest_via_sbom'

Traceback (most recent call last):
...
  File "/Users/wagoodman/code/vunnel/tests/quality/./gate.py", line 148, in get_namespaces_from_db
    raise RuntimeError(f"mismatched namespaces:\nextra:   {extra}\nmissing: {missing}")
RuntimeError: mismatched namespaces:
extra:   {'github:language:swift', 'github:language:dart'}
missing: set()
make: *** [validate] Error 1

For problem 2 as a work around I've added an existing debian 11 image to the test section for the debian provider.

[wagoodman]

We saw the failure we were looking for:

023-09-29T15:12:32.2892481Z Traceback (most recent call last):
2023-09-29T15:12:32.2893133Z   File "/home/runner/work/vunnel/vunnel/tests/quality/./gate.py", line 462, in <module>
2023-09-29T15:12:32.2893630Z Loading label entries... done! 755 entries loaded
2023-09-29T15:12:32.2894286Z [95m[1mValidating with 'pr_vs_latest_via_sbom' [0m 
2023-09-29T15:12:32.2894504Z 
2023-09-29T15:12:32.2896960Z     main()
2023-09-29T15:12:32.2897767Z   File "/home/runner/.virtualenvs/vunnel-A95CY2kd-py3.10/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
2023-09-29T15:12:32.2908611Z     return self.main(*args, **kwargs)
2023-09-29T15:12:32.2909665Z   File "/home/runner/.virtualenvs/vunnel-A95CY2kd-py3.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
2023-09-29T15:12:32.2913302Z     rv = self.invoke(ctx)
2023-09-29T15:12:32.2914102Z   File "/home/runner/.virtualenvs/vunnel-A95CY2kd-py3.10/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
2023-09-29T15:12:32.2917951Z     return ctx.invoke(self.callback, **ctx.params)
2023-09-29T15:12:32.2918744Z   File "/home/runner/.virtualenvs/vunnel-A95CY2kd-py3.10/lib/python3.10/site-packages/click/core.py", line 783, in invoke
2023-09-29T15:12:32.2921746Z     return __callback(*args, **kwargs)
2023-09-29T15:12:32.2922246Z   File "/home/runner/work/vunnel/vunnel/tests/quality/./gate.py", line 382, in main
2023-09-29T15:12:32.2925168Z     validate(
2023-09-29T15:12:32.2925647Z   File "/home/runner/work/vunnel/vunnel/tests/quality/./gate.py", line 164, in validate
2023-09-29T15:12:32.2928008Z     namespaces = get_namespaces_from_db()
2023-09-29T15:12:32.2928865Z   File "/home/runner/work/vunnel/vunnel/tests/quality/./gate.py", line 148, in get_namespaces_from_db
2023-09-29T15:12:32.2931452Z     raise RuntimeError(f"mismatched namespaces:\nextra:   {extra}\nmissing: {missing}")
2023-09-29T15:12:32.2931916Z RuntimeError: mismatched namespaces:
2023-09-29T15:12:32.2932220Z extra:   set()
2023-09-29T15:12:32.2933279Z missing: {'debian:distro:debian:11', 'debian:distro:debian:10', 'debian:distro:debian:12', 'debian:distro:debian:unstable', 'debian:distro:debian:13'}
2023-09-29T15:12:32.3483405Z make: *** [Makefile:29: validate] Error 1
2023-09-29T15:12:32.3496363Z ##[error]Process completed with exit code 2.

Now I've cherry-picked the fix that @willmurphyscode put together #306 . We should see this PR pass now 🤞