Create content for repository data-bag

[sbocconi]

With npm -v 3.3.12 and node -v 5.3.0 npm install fails with

npm ERR! Darwin 15.2.0
npm ERR! argv "/usr/local/Cellar/node/5.3.0/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v5.3.0
npm ERR! npm  v3.3.12
npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1

Upgrading to the latest release fixes the problem for me, but might break with other versions of node/npm?

Stefanos-MacBook-Pro:data SB$ git diff
diff --git a/package.json b/package.json
index afb2766..62b1d46 100644
--- a/package.json
+++ b/package.json
@@ -21,7 +21,7 @@
     "geojsonhint": "^1.0.0",
     "highland": "^2.5.1",
     "is-my-json-valid": "^2.10.1",
-    "level": "^0.19.1",
+    "level": "^1.4.0",
     "minimist": "^1.1.1",
     "ndjson": "^1.3.0",
     "pg": "^4.3.0",

[bertspaan]

I've been working on a new version of the data package: https://github.com/histograph/data/tree/data-modules

I've moved all dataset-specific logic out of data module, and into separate modules:

• https://github.com/histograph/data-tgn
• https://github.com/histograph/data-geonames

I'm using this version here in NYC, with newest Node + NPM. However, I've not changed NWB + BAG data modules. I can explain how to make one! It's easy!
And then I can merge data-modules branch of this repo into master.

[bertspaan]

By the way: hello @sbocconi!

[sbocconi]

hello @bertspaan!!

[bertspaan]

This is how data module works in new data-modules branch:

• It looks for subdirectories named data-xxx in a specified directory (i.e. data-tgn)
• From such a directory, it loads xxx.js (i.e. require('data-tgn/tgn.js'))
• (You have to make sure that you've ran npm install for each data module)

Each data module should have the following API:

module.exports.steps = [
  step1, // step1 and step2 are functions inside the data module
  step2
];

[sbocconi]

I changed the title of this issue since the original one is fixed but the discussion here is useful to solve the issue described in the title.

[sbocconi]

https://github.com/reinvantveer is working on this:

Yes I'm doing BAG at the moment. Got as far as neatly recursively unpacking
those dreadful zips (at least the Amstelveen test set) , so I can get set
to streaming thise dreadful XMLs to ndjson :)
Greetings,
Rein

[reinvantveer]

Doing the BAG... 👜

[reinvantveer]

Dank @jobspierings, je mag me ook assignen als je wil, geloof niet dat ik dat kan...
Of doet @sbocconi dit?

[reinvantveer]

Downscaling issue to BAG