Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port to python3, stop storing DBs in git, and support downloading DBs from USDA #8

Open
wants to merge 37 commits into
base: master
Choose a base branch
from
Open

Port to python3, stop storing DBs in git, and support downloading DBs from USDA #8

wants to merge 37 commits into from

Conversation

lee-b
Copy link

@lee-b lee-b commented Jul 31, 2016

NOTE: this breaks existing git history and compatibility. It:

  • rewrites git history, to remove the DB data. Not to be merged lightly. I think it's necessary though; DBs should not be in git.
  • Moves code to python3, dropping support for python2. Not strictly necessary, probably could support both, but I feel python2 should have been dropped years ago, so I've no interest in spending time on backwards compatibility.

Sigfrido Chirinos and others added 30 commits May 8, 2013 02:00
Initial import of parsing code.
a778e27
Added all table to parser and raw data files along with sqlite db.
b16110b
Better messaging and only refreshes when data doesn't exist.
31a5f00
Moved flat file data to sub folder.
3c52ddb
Adding conversion to document format for mongo.
cda581b
Adding transforms to document for foods.
8d3c769
Finished initial conversion of nutrient data to document structure.
c6d7050
Added mongo exporting.
b46cb0f
Adding script documentation.
ab2cb82
More documentation tweaks.
db83e94
Add a bit more clarity to readme.
5ee3eab
Readme update.
af16215
Readme updates.
b645ed6
readme updates again.
9fc1a99
Readme updates.
a52f573
Added more command line options and further info to readme. Fixed som…
eba0138 
…e descriptions in program command line options.
Added code to langual info.
64a69a8
Adjusted document dump schema.
85aadcd
Modified schema again.
c7af387
@gesinger
Convert port to int for pymongo plugin
db9c320
@gesinger
Updated README to use nutrientdb.py instead of nutrient.py
e76606f
@schirinos
Merge pull request schirinos#2 from gesinger/bugfix/port_to_int
a1e112d 
Convert port to int for pymongo plugin
@schirinos
Merge pull request schirinos#3 from gesinger/bugfix/update_readme
206bca8 
Updated README to use nutrientdb.py instead of nutrient.py
@philtom
added numeric types for any fields with an "N" type in the sr28 file …
8976061 
…format docs
@philtom
adding sr28 ascii data files and sqlite db
6a6dfad
@asazernik
Set exec permissions bit on nutrientdb.py
49b4445 
So that the #! line at the beginning is actually useful.
@asazernik
Mark data files as binary
b9bf03d 
They're kind-of-text-ish, but the files (and their diffs) are not really
intended for human consumption.
@asazernik
Order weights by ndb sequence number
d3fd91a 
So far as I can tell this ordering falls out by default because of
insertion order into the db, but it should be very cheap to ask sqlite
to check this at dump time.
@asazernik
Don't convert ndb_no to int
8b2b920 
The original datatype is actually 'A 5', not 'N'. The code for dumping
to sqlite seems to agree?
@asazernik
Set default DB release to sr28
b7ae369
@jackmcdade
Copy link

When exporting to JSON, it does not create a valid JSON object (it should be an array of objects, not just a text file with JSON objects inside it), and also includes some logging in the beginning of the file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@lee-b @jackmcdade @gesinger @schirinos @philtom @asazernik