rewrite hardware validation engine: move validations into JSON Schemas

[karenetheridge]

details TBD. This involves a total rewrite of the validation engine and reworking related database tables.

[karenetheridge]

Action plan (checked boxes indicate code is complete, but not merged or released):

• add new json_schema database table (each row containing a JSON Schema document and related metadata such as type, name, version, created_user)
• create new API endpoints for creating, deleting, fetching json schemas
  • POST /json_schema/:json_schema_type/:json_schema_name
  • GET /json_schema/:json_schema_type/:json_schema_name/:json_schema_version
  • GET /json_schema/:json_schema_type/:json_schema_name/latest
  • GET /json_schema/:json_schema_type/:json_schema_id
  • DELETE /json_schema/:id and /json_schema/:json_schema_type/:json_schema_name/:json_schema_version (not /latest)
• add hardware_product_json_schema table
• create new API endpoints for manipulating associations between hardware products and json schemas
  • POST /hardware/:hardware_product_id_or_other/json_schema/:json_schema_id
  • POST /hardware/:hardware_product_id_or_other/json_schema/:json_schema_type/:json_schema_name/:json_schema_version
  • POST /hardware/:hardware_product_id_or_other/json_schema/:json_schema_type/:json_schema_name/latest
  • GET /hardware/:hardware_product_id_or_other/json_schema
  • DELETE /hardware/:hardware_product_id_or_other/json_schema/:json_schema_id
  • DELETE /hardware/:hardware_product_id_or_other/json_schema/:json_schema_type/:json_schema_name/:json_schema_version (not .../latest)
  • DELETE /hardware/:hardware_product_id_or_other/json_schema
• write a design document: https://docs.google.com/document/d/1OONmQBGyWRGyPIsi2y9KWKxJGmMKLx27N2GyO7gQTZk/edit#heading=h.bep7odob8qxh
• extend json_schema handling to accomodate references between documents
  • new json_schema_references database table
  • look for unresolvable references during schema creation
  • record references in a new tracking table
  • adjust ".../latest" references forward as new schemas in a type/name series are added
  • look for newly-unresolvable references when deleting a schema
  • adjust ".../latest" $refs backwards to point to previous schemas as schemas are deleted
  • extend GET /json_schema/* and GET /hardware_product/*/json_schema endpoints to include referenced schemas in output
• patch implement new open source JSON Schema library to support adding multiple JSON Schema documents that reference each other
• update device report submission endpoints to use json schemas rather than legacy validation modules
  • remove POST /device/:device_id_or_serial_number/validation_plan/:validation_plan_id
  • remove POST /device/:device_id_or_serial_number/validation/:validation_id
  • add POST /device_report/json_schema/:json_schema_id
  • add POST /device_report/json_schema/:json_schema_type/:json_schema_name/:json_schema_version
  • add POST /device_report/json_schema/:json_schema_type/:json_schema_name/latest
  • update POST /device_report
  • update POST /device_report?no_save_db=1
• preserve historical data and a mechanism to access it
  • rename old database tables:
    • validation -> legacy_validation
    • validation_result -> legacy_validation_result
    • validation_state_member -> legacy_validation_state_member
  • GET /validation_state/:validation_state_id to return data from both legacy and new validation results
  • GET /device/:id_or_serial/validation_state "" ""
• delete all now-unused pieces of the legacy validation infrastructure
• write a script, to be run in current-production, to extract all BOM-related hardware_product parameters, to be used in writing new validation schema documents
• document the new API endpoints
• write a new utility that removes a certain legacy validation by name and replaces it with a certain /json_schema/$type/$name/latest. that is, mark the validation deactivated, find the validation in all validation_plans and remove it, and then adds hardware_product_json_schema endpoints for all hardware products pointing to that validation plan(s). (can use this to start converting validations to new style.)
• write a new utility that goes through the entire json_schema table and checks that each is still conformant to the expected metaschema (which is a more strict version of the specification metaschema, and can change over time or even change drafts)

[karenetheridge]

implement new open source JSON Schema library to support adding multiple JSON Schema documents that reference each other

This depends on:

• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/1
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/2
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/3
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/4
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/5
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/7
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/8
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/9
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/14
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/19
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/20
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/32
• https://github.com/karenetheridge/JSON-Schema-Draft201909/issues/33

[karenetheridge]

Documentation for new endpoints is included below (screenshots, because it is not up on github.io yet, but only locally served via jekyll in the git branch).

specifically, they are:

• POST /json_schema/:json_schema_type/:json_schema_name
• GET /json_schema/:json_schema_id
• GET /json_schema/:json_schema_type/:json_schema_name/:json_schema_version
• GET /json_schema/:json_schema_type/:json_schema_name/latest
• DELETE /json_schema/:json_schema_id
• GET /json_schema/:json_schema_type
• GET /json_schema/:json_schema_type/:json_schema_name
• POST /hardware/:hardware_product_id_or_other/json_schema/:json_schema_id
• POST /hardware/:hardware_product_id_or_other/json_schema/:json_schema_type/:json_schema_name/:json_schema_version
• POST /hardware/:hardware_product_id_or_other/json_schema/:json_schema_type/:json_schema_name/latest
• GET /hardware/:hardware_product_id_or_other/json_schema
• DELETE /hardware/:hardware_product_id_or_other/json_schema/:json_schema_id
• DELETE /hardware/:hardware_product_id_or_other/json_schema/:json_schema_type/:json_schema_name/:json_schema_version
• DELETE /hardware/:hardware_product_id_or_other/json_schema