Vamc 15446 spike vamc police data page

[eselkin]

Description

relates to department-of-veterans-affairs/va.gov-cms#15446

Creates a new process for queryType: "curl" in static-data-files

This allows an arbitrarily large number of files to be downloaded via curly/node-libcurl GET and a postProcess step that receives the data in an array and can process the resulting data in any manner needed.

It uses libcurl because node-fetch has problems using file:// URLs which we need for the police instance.

The police postProcess step writes the files to temporary files and reads them in with csvtojson to later process (this ticket only starts the structure, since it is unknown what the format of the CSVs will be at the moment).

Changes were made to the static file creation, in that if any of the files are not present, all are generated. Previously only a test was done to determine if the directory exists, but now that there are multiple processes generating files, it seemed appropriate to convert to a per-file test.

Testing done & Screenshots

Added cypress testing for the existence of all JSON static data files.

QA steps

What needs to be checked to prove it didn't break any related things?
What variations of circumstances (users, actions, values) need to be checked?

1. Check if https://web-w1a5evfuonymjntciz3pduj1ofhepwky.demo.cms.va.gov/data/cms/vamc-facility-supplemental-status.json exists
2. Check if https://web-w1a5evfuonymjntciz3pduj1ofhepwky.demo.cms.va.gov/data/cms/vamc-ehr.json exists
3. Check if https://web-w1a5evfuonymjntciz3pduj1ofhepwky.demo.cms.va.gov/data/cms/vamc-police.json exists
   1. it is an empty object because I didn't want to put fake data into production.

Acceptance criteria

• A skeleton is started for the general file structure pending any missing information. (e.g., if the output has to be JSON, the Ruby requirements are documented)

Definition of done

• Events are logged appropriately
• Documentation has been updated, if applicable (added testing with comments, looking for other locations for other notes about static data files)
• A link has been provided to the originating GitHub issue (or connected to it via ZenHub)
• No sensitive information (i.e. PII/credentials/internal URLs/etc.) is captured in logging, hardcoded, or specs

[va-vfs-bot]

ESLint is disabled

vets-website uses ESLint to help enforce code quality. In most situations we would like ESLint to remain enabled.

What you can do

See if the code can be refactored to avoid disabling ESLint, or wait for a VSP review.

[eselkin]

Following a pattern from drupal graphql static files here. Not sure if at some point we'll need any of these variables.

[swirtSJW]

Not really an official review. This is shaping up pretty nicely. Mainly just a couple questions to help me understand it better.

[eselkin]

ESLint is disabled

vets-website uses ESLint to help enforce code quality. In most situations we would like ESLint to remain enabled.

What you can do

See if the code can be refactored to avoid disabling ESLint, or wait for a VSP review.

I think this is a funny message, considering this is not a vets-website PR.

[va-vfs-bot]

ESLint is disabled

vets-website uses ESLint to help enforce code quality. In most situations we would like ESLint to remain enabled.

What you can do

See if the code can be refactored to avoid disabling ESLint, or wait for a VSP review.

[maxx1128]

The pull request looks good to me, I just have a few suggestions and questions before I'm ready to approve. Overall, it's looking good!

[maxx1128]

This could maybe be simplified so we have an array of the two strings in each pathToFileURL function, and we map them out to get this.

    query: queryVAPoliceData(['police-contact.csv', 'police-events.csv']
              .map(filename => url.pathToFileURL(join(__dirname, 'vaPoliceData', filename)))
    ),

[va-vfs-bot]

ESLint is disabled

vets-website uses ESLint to help enforce code quality. In most situations we would like ESLint to remain enabled.

What you can do

See if the code can be refactored to avoid disabling ESLint, or wait for a VSP review.

[va-vfs-bot]

ESLint is disabled

vets-website uses ESLint to help enforce code quality. In most situations we would like ESLint to remain enabled.

What you can do

See if the code can be refactored to avoid disabling ESLint, or wait for a VSP review.

[maxx1128]

👏

[rmessina1010]

LGTM

[randimays]

ESLint is disabled

vets-website uses ESLint to help enforce code quality. In most situations we would like ESLint to remain enabled.

What you can do

See if the code can be refactored to avoid disabling ESLint, or wait for a VSP review.

I think this is a funny message, considering this is not a vets-website PR.

Sharp eye 👀 Seems it's just copy/pasta

[chriskim2311]

LGTM