validate: Validate Raw Data, store in DB #23
Conversation
ghost
requested a review
from 
ghost
self-assigned this
|
#14 for notes - raw data, filter only some results, and it's a % we show on the results page Technically we don't need validation_passed - you could do validation_results==[] to get the same but setting up this should make for some very easy SQL queries to get the data we need for generating the stat, and some space efficient indexes later. |
| @@ -0,0 +1,3 @@ | |||
| ALTER TABLE raw_data ADD validation_done BOOLEAN NOT NULL DEFAULT 'f'; | |||
| ALTER TABLE raw_data ADD validation_passed BOOLEAN NOT NULL DEFAULT 'f'; | |||
There was a problem hiding this comment.
Not a blocker, would prefer the SQL compliant FALSE or TRUE whenever possible. I had to look up what postgres accepts for booleans. Then you won't have to spend energy escaping quotes later on too.
| try { | ||
| await client.query( | ||
| 'UPDATE raw_data SET validation_done= \'t\', validation_results=$1, validation_passed=$2 WHERE id=$3', | ||
| [JSON.stringify(result_filtered), (result_filtered.length == 0), raw_data.id] |
There was a problem hiding this comment.
Your === above reminded me we probably want an === here too. It is good practice in JS to keep the equivalency(?) consistent .
| To run this: | ||
|
|
||
| `$ node ./src/bin/validate-raw-data.js` | ||
|
|
There was a problem hiding this comment.
We could really use npm run scripts, the advantage of those is that you avoid encoding the filename/path in lots of places. Though perhaps this is the only place anyway.
There was a problem hiding this comment.
When we set up Heroku some more hard coded paths will appear so this is a good point
|
|
||
| const client = await database_pool.connect(); | ||
| try { | ||
| while(true) { |
There was a problem hiding this comment.
Just wondering if you're expecting this to be OK to fail?
There was a problem hiding this comment.
Not confident, but this applies to many other places and we need a general review of handling error conditions #26
No description provided.