Change DB publish workflow to account for V6

[wagoodman]

Today the v1-5 schemas have a fan-out-fan-in approach, where we fan out to build and upload the DB and fan in to update the listing file (since each DB schema is combined into a single listing).

With the changes proposed in anchore/grype#2125 , there will be a different listing file (latest.json), a new history.json file, and there is one schema per fileset. This means we no loner need to fan in for v6+ 🎉

However, grype-db/grype-db-manager is still responsible for assembling db v1-5 listing files -- we need the new and old paradigms to live side-by-side.

Grype DB changes

• still output metadata.json and provider-metadata.json for v1-5 use cases (affects build step)
• when schema v6 is indicated:
  • on build: output only metadata.json and DB
  • on package:
    • output archive vulnerability-db_VERSION_OLDESTDATADATE_BUILTEPOCH.tar.gz
    • Where:
      • VERSION: schema version in the form of v#.#.#
      • OLDESTDATADATE: RFC3338 formatted value of the oldest .compiled date found for all contained providers
      • BUILTEPOCH: linux epoch formatted value of the .database.built field
    • output metadata filevulnerability-db_VERSION_OLDESTDATADATE_BUILTEPOCH.json, which is the basis of the latest.json file downstream in the manager (same variable meanings as a the archive)

Grype DB Manager changes

• when schema v6 is indicated:
  • use the relative URLs as provided directly by grype-db, no need to recreate the latest.json file
  • still smoke test validate the given latest.json with new assumptions (schema is in hosted path, use given latest file, etc)
  • when uploading the DB tar archive, also upload: grype-db_VERSION_OLDESTDATADATE_BUILTEPOCH.json latest.json
  • when uploading the listing
    • the user indicates a remote grype-db_VERSION_OLDESTDATADATE_BUILTEPOCH.json name...
    • we check for the existence of the JSON file, the archive tar, and verify the JSON file checksum matches that of the archive
    • copy the grype-db_VERSION_OLDESTDATADATE_BUILTEPOCH.json as the new latest.json and upload
  • operationally we should start segmenting schema DBs into separate paths in S3 (...databases/v6/vulnerability-....tar.gz vs databases/vulnerability-....tar.gz). V1-5 should still be placed in "databases/vulnerability-....tar.gz"
  • the history.json file should be built based on : the last history.json file and the current s3 state, pruning beyond 90 days. Instead of using the tar files as state indications, use the newly uploaded json files.

edits:

• history.json functionality has been descoped for this effort
• latest.json can be smoke tested in place and uploaded with the DB (no need for a separate json file being renamed and moved)

[wagoodman]

I've decided to defer any work around the history.json functionality

[wagoodman]

Currently mostly implemented in #446 but cannot continue until there is prototype matching implemented