Consolidate the DB structure and the API (harmonise db.user/db.users, db.project/db.projects, etc.)

[xgui3783]

sub task of #2

per discussion with @r03ert0 on mattermost:

a big chunk of the DB handling from microdraw (MD) and brainbox (BB) is now done in the neuroWebLab (NWL) module, but there's still several things to finish. That would be the aim of 4. Consolidate the DB structure and the API (harmonise db.user/db.users, db.project/db.projects, etc.)

Tasks:

• Plurals only

One very confusing aspect of MD and BB is that on one side the Mongo collection for users is db.user and in the other side it's db.users. In one side you have db.project, and in the other db.projects.

At the moment, the NWL module takes as inputs the names of the collections, to allow the same code to work in both cases.

We need to change that, and keep the plural version only (which entails changing BBs database... and updating aaaaall our old code)

• Data-type agnostic

The second issue with the DB, is that MD deals with slices and vectorial annotations, and BB deals with brain atlases, etc. Both deal with arbitrary metadata, such as subject name, or comments.

We need to encapsulate modality-specific data into a generic data object, with a variable indicating the data type (deep zoom images, MRI volumes, etc.)

All the aspects related to text-metadata should be generic, and the data-specific container should be only concerned with MD or BB type of data

[xgui3783]

plurals only

per conversation on mattermost:

i'm not sure in which extent it would be easy to make something exhaustive but maybe you could play a bit with your local version of MD and BB and make a script to make the db migration on those, then we could test to run the scripts on a duplicated version and see if a part was left out or if everything is fine...

here is what I propose I will do in the near future:

• create local instance of mongodb
• run latest microdraw & bb , targeting local mongodb, extract user & project schema
• create script that migrates existing db
• patch nwl, microdraw, bb to use updated collection and/or schema

[xgui3783]

related to #1

[xgui3783]

per

create script that migrates existing db

I would like to validate the collections in prod database. I have created a quick and dirty schema checker at my fork here:

https://github.com/xgui3783/microdraw/blob/refactor_migration/_migration/validateMicrodrawMongo.js
https://github.com/xgui3783/microdraw/blob/refactor_migration/_migration/validateBrainboxMongo.js

(or use git hash)

https://github.com/xgui3783/microdraw/blob/972fd6f/_migration/validateMicrodrawMongo.js
https://github.com/xgui3783/microdraw/blob/972fd6f/_migration/validateBrainboxMongo.js

the script assumes monk is installed as dependency.

can someone run the script against prod db, and let me know if any error was thrown?

(validating users should throw for Anyone (ie anonymouse account))

[xgui3783]

cc @r03ert0 @katjaq

no rush, but would be good to know when you get a chance.

[xgui3783]

more questions (unfortunately) on how we would like the migration to go:

1/ should the databases remain separate? (e.g. we will still have localhost:27017/microdraw and localhost:27017/brainbox) or combined (ie localhost:27017/neuroweblab)
2/ should annotations be their own collection (per microdraw) or be inside projects (per brainbox), or maintain status quo (microdraw have annotations as separate collections, brainbox have annoations as sub objects of projects)

I would like some clarification on 1/ , for 2/ my understanding is having annotation as a separate collection is a clearer solution.