feat: add support for multiple database connections (#146)

* feat: new migrations folder structure for model-defined database connections * fix: do not generate static adapters index * fix: utils require path in migrations template * feat: explicit naming for default databases * fix: access internal sequelize model reference * tests: remove and ignore instance folders * fix: case-insensitive storageType check * feat: updated integration tests toolchain to use yarn workspaces * docs: added new flags to man page * fix: use distro node-jq binary * fix: yarn workspace not cleaning acl_rules * fix: branch checkout should delete generated code * feat: remove yarn and jq dependencies from CLI * feat: add helper for standard logs * fix: remove npm install from restartContainers() * chore: handle branch checkout in workspace * refactor: remove redundant function * fix: increase test 21. Limit check timeout to 10s * chore: remove deprecated sequelize config * fix: remove unused paths from .gitignore * docs: updates and fixes to the project README * fix: change docs from yarn to npm run * fix: -g should not crash if servers are not created * fix: remove ReconstructDb from comment * chore: use logTask for output messages * feat: async check and wait for gql server start * refactor: renamed T1 -> SERVER_CHECK_WAIT_TIME * refactor: do one thing, do it well; better log messages * fix: docker-compose creates empty node_modules * refactor: extract waitForGql() from doTests() * fix: out of place log * refactor: remove redundant checkCode() function * fix: do not attempt to list servers if the workspace has not been set * fix: using -b should checkout, fetch, and reset <branch> * chore: bump 0.2.0 -> 0.3.0
Zendro-dev · Aug 28, 2020 · 0f155a0 · 0f155a0
1 parent c4353c3
commit 0f155a0
Show file tree

Hide file tree

Showing 21 changed files with 711 additions and 527 deletions.
diff --git a/README.md b/README.md
@@ -1,16 +1,15 @@
-# Code-generator
+<h1 align=center>Code Generator</h1>
 
-Command line utility to auto-generate the structure files that [this server](https://github.com/ScienceDb/graphql-server)
-will use to perform CRUD operations for each model created.
+Command line utility to generate the structure files that Zendro [graphql-server](https://github.com/Zendro-dev/graphql-server) will use to perform CRUD operations for each model created.
 
-## Set up:
+## Set up
 Clone the repository and run:
 ```
 $ npm install -g
 ```
 If you only want to install it locally run `npm install` instead
 
-## Usage:
+## Usage
 
 To run the unit-test case:
 ```
@@ -22,82 +21,103 @@ To run the integration-test case
 $ npm run test-integration [-- OPTIONS]
 ```
 Note:
-Intergation-test case creates a docker-compose ambient with three servers:
+Integration-test case creates a docker-compose environment with three servers:
 
-    gql_postgres
-    gql_science_db_graphql_server
-    gql_ncbi_sim_srv
-
-By default, after the test run, all corresponding Docker images will be completely removed from the docker, this cleanup step can be skiped with __-k__ option.
+```
+gql_postgres
+gql_science_db_graphql_server
+gql_ncbi_sim_srv
+```
 
-Please run this utility with __-h__ option to see the full documentation in manpage style.
+### Examples of use - Test integration
 
-## Examples of use - Test integration:
 To see full test-integration info:
-```
+```bash
 $ npm run test-integration -- -h
 ```
-To restart containers:
-```
-$ npm run test-integration -- -r
+
+To execute a default test run:
+```bash
+$ npm run test-integration
 ```
-To generate code and start containers:
+
+To test a specific `graphql-server` branch:
+```bash
+$ npm run test-integration -- b <branch_name>
 ```
+
+To generate code:
+```bash
 $ npm run test-integration -- -g
 ```
-To do the tests only and keep the containers running at end:
-```
-$ npm run test-integration -- -t -k
-```
+
 To generate code and do the tests, removing all Docker images at end:
-```
-$ npm run test-integration -- -T
-```
-To do a full clean up (removes containers, images and code):
-```
+```bash
 $ npm run test-integration -- -T
 ```
 
-
-And to generate the structure files:
+To generate code, do the tests, and keep the containers running at end:
+```bash
+$ npm run test-integration -- -T -k
 ```
-$ code-generator -f <input-json-files> -o <output-directory>
+
+To do the tests only and keep the containers running at end:
+```bash
+$ npm run test-integration -- -t -k
 ```
+
+To restart containers:
+```bash
+$ npm run test-integration -- -r
 ```
-INPUT:
-<input-json-files> - directory where json models are stored
-<output-directory> - directory where the generated code will be written
+
+To clean generated code, remove containers and volumes:
+```bash
+$ npm run test-integration -- -C
 ```
-This command will create(if doesn't exist) four folders containing the generated files for each model in the ```input-json-files```:
 
-* models ----> sequelize model
-* schemas ----> graphQL schema
-* resolvers ----> basic CRUD resolvers
-* migrations ----> create and delete table migration file
+To do a full clean up (removes containers, images and code):
+```bash
+$ npm run test-integration -- -c
+```
 
+### Examples of use - Code Generator
 
-## Examples of use - Code generator:
 In the same directory of this repository run:
 
-```
+```bash
+# -f <input-json-files>   directory where json models are stored
+# -o <output-directory>   directory where the generated code will be written
 $ code-generator -f ./example_json_files -o /your_path_directory
 ```
-If you want to complete the example with the [server](https://github.com/ScienceDb/graphql-server)
-make ```/your_path_directory``` the same directory where the server repository is stored.
 
-NOTE: For displaying the explanation about usage we can run the help flag:
-```
+For help using this command:
+
+```bash
 $ code-generator -h
+
+# Code generator for the GraphQL server
+#
+# Options:
+#
+#   -f, --jsonFiles <filesFolder>      Folder containing one json file for each model
+#   -o, --outputDirectory <directory>  Directory where generated code will be written
+#   -h, --help                         output usage information
 ```
-```
-Code generator for GraphQL server
 
- Options:
+This command will create four sub-folders within the `output-directory` folder, containing the generated files for each model:
+
 
-   -f, --jsonFiles <filesFolder>      Folder containing one json file for each model
-   -o, --outputDirectory <directory>  Directory where generated code will be written
-   -h, --help                         output usage information
 ```
+models/      -> sequelize model
+schemas/     -> graphQL schema
+resolvers/   -> basic CRUD resolvers
+migrations/  -> create and delete table migration file
+```
+
+To use the code generator with the [graphql-server](https://github.com/Zendro-dev/graphql-server),
+use its path in the `output-directory`.
+
 
 ## JSON files Spec
 
@@ -107,15 +127,15 @@ For each model we need to specify the following fields in the json file:
 
 Name | Type | Description
 ------- | ------- | --------------
-*model* | String | Name of the model (it is recommended uppercase for the initial character).
-*storageType* | String | Type of storage where the model is stored. So far can be one of __sql__ or __Webservice__ or __zendro\_server__
-*url* | String | This field is only mandatory for __zendro\_server__ stored models. Indicates the url where the zendro server storing the model is runnning. 
-*attributes* | Object | The key of each entry is the name of the attribute and theres two options for the value . Either can be a string indicating the type of the attribute or an object where the user can indicates the type of the attribute(in the _type_ field) but also can indicates an attribute's description (in the _description_ field). See the [table](#types-spec) below for allowed types. Example of option one: ```{ "attribute1" : "String", "attribute2: "Int" }``` Example of option two: ``` { "attribute1" : {"type" :"String", "description": "Some description"}, "attribute2: "Int ```
+*model*       | String | Name of the model (it is recommended to Capitalize the name).
+*storageType* | String | Type of storage where the model is stored. Currently supported types are __sql__, __Webservice__, and __zendro\_server__
+*url*         | String | This field is only mandatory for __zendro\_server__ stored models. Indicates the URL of the Zendro server storing the model.
+*attributes*  | Object | The key of each entry is the name of the attribute. There are two options for the value: a string indicating the type of the attribute, or an object with two properties: _type_ (the type of the attribute) and _description_ (attribute description). See [types-spec](#types-spec) table below for allowed types. Example of option one: ```{ "attribute1" : "String", "attribute2: "Int" }``` Example of option two: ``` { "attribute1" : {"type" :"String", "description": "Some description"}, "attribute2: "Int ```
 *associations* | Object | The key of each entry is the name of the association and the value should be an object describing the associations. See [Associations Spec](associations-spec) section below for the specifications of the associations.
 
 EXAMPLES OF VALID JSON FILES
 
-```
+```jsonc
 //Dog.json
 {
   "model" : "Dog",
@@ -139,7 +159,7 @@ EXAMPLES OF VALID JSON FILES
 
 ```
 
-```
+```jsonc
 //Publisher.json
 {
   "model" : "Publisher",
@@ -175,7 +195,7 @@ Date |
 Time |
 DateTime |
 
-For more info about `Date`, `Time`, and `DateTime` types, please see this [info](https://github.com/excitement-engineer/graphql-iso-date/blob/HEAD/rfc3339.txt).
+For more info about `Date`, `Time`, and `DateTime` types, please see the [graphql-iso-date/rfc3339.txt](https://github.com/excitement-engineer/graphql-iso-date/blob/HEAD/rfc3339.txt).
 
 Example:
 * Date: A date string, such as `2007-12-03`.
@@ -211,7 +231,7 @@ name | Type | Description
 ## NOTE:
 Be aware that in the case of this type of association the user is required to describe the cross table used in the field _keysIn_ as a model in its own. For example, if we have a model `User` and a model `Role` and they are associated in a _manytomany_ way, then we also need to describe the `role_to_user` model:
 
-```
+```jsonc
 //User model
 {
   "model" : "User",
@@ -235,7 +255,7 @@ Be aware that in the case of this type of association the user is required to de
 }
 ```
 
-```
+```jsonc
 //Role model
 {
   "model" : "Role",
@@ -258,7 +278,7 @@ Be aware that in the case of this type of association the user is required to de
 }
 ```
 
-```
+```jsonc
 //role_to_user model
 {
   "model" : "role_to_user",
@@ -275,7 +295,7 @@ Be aware that in the case of this type of association the user is required to de
  It's important to notice that when a model involves a foreign key for the association, this key should be explicitly written into the attributes field of the given local model.
 
 Example:
-```
+```jsonc
 {
   "model" : "book",
   "storageType" : "sql",
@@ -301,7 +321,7 @@ THE SAME DATA MODELS DESCRIPTION(.json files) WILL BE USEFUL FOR GENERATING BOTH
 
 Fields *`label`* and *`sublabel`* in the specification are only needed by the GUI generator, but backend generator will only read required information, therefore extra fields such as *`label`* and *`sublabel`* will be ignored by the backend generator.
 Example:
-```
+```jsonc
 //book.json
 {
  "model" : "Book",
@@ -329,27 +349,3 @@ Example:
 ## Testing
 
 For relevant files see `package.json` (section scripts), directories `.test` and `docker`. Test framework is `mocha` and `chai`.
-
-### Unit tests
-
-Run all existing unit tests with
-```
-npm run test-unit
-```
-
-### Integration tests
-
-#### Requirements
-
-You need to be on a \*nix operating system, and have bash and Docker installed and running.
-
-Integration tests are carried out using Docker to setup a GraphQL web server and generate code for example data models. The last step of the setup is to create databases and migrate schemas. After that the server is started using `localhost:3000`, which can than be accessed using HTTP. Solely via such HTTP connections the generated API (GraphQL web server) is tested, just as a user might be doing with e.g. `curl`.
-
-All related Docker files are stored in `./docker`; especially `docker-compose-test.yml`.
-
-The test pipeline is defined and executed in `./test/integration-test.bash` for reasons of simplicity. The actual integration tests are written using `mocha` and can be found in `./test/mocha_integration_test.js`, which is invoked by the above bash script.
-
-To ecexute the integration tests run
-```
-npm run test-integration
-```
diff --git a/docker/Dockerfile.graphql_server b/docker/Dockerfile.graphql_server
@@ -3,10 +3,9 @@ FROM node:14.3.0-stretch-slim
 # Create app directory
 WORKDIR /usr/src/app
 
+ENV JQ_PATH=/usr/bin/jq
+
 # Clone the skeleton project and install dependencies
-RUN apt-get update && apt-get install -y git procps autoconf libtool make &&\
-  git clone --branch master https://github.com/ScienceDb/graphql-server.git . && \
-  chmod u+x ./migrateDbAndStartServer.sh && \
-  npm install --save
+RUN apt-get update && apt-get install -y git procps autoconf libtool make jq
 
 EXPOSE 3000
diff --git a/docker/data_models_storage_config1.json b/docker/data_models_storage_config1.json
@@ -0,0 +1,11 @@
+{
+  "default-sql": {
+    "storageType": "sql",
+    "username": "sciencedb",
+    "password": "sciencedb",
+    "database": "sciencedb_development",
+    "host": "gql_postgres1",
+    "dialect": "postgres",
+    "operatorsAliases": false
+  }
+}
diff --git a/docker/data_models_storage_config2.json b/docker/data_models_storage_config2.json
@@ -0,0 +1,11 @@
+{
+  "default-sql": {
+    "storageType": "sql",
+    "username": "sciencedb",
+    "password": "sciencedb",
+    "database": "sciencedb_development",
+    "host": "gql_postgres2",
+    "dialect": "postgres",
+    "operatorsAliases": false
+  }
+}
diff --git a/docker/docker-compose-test.yml b/docker/docker-compose-test.yml
@@ -17,27 +17,16 @@ services:
     build:
       context: .
       dockerfile: Dockerfile.graphql_server
-
     volumes:
-      - ./integration_test_run/instance1/models/adapters:/usr/src/app/models/adapters
-      - ./integration_test_run/instance1/schemas:/usr/src/app/schemas
-      - ./integration_test_run/instance1/resolvers:/usr/src/app/resolvers
-      - ./integration_test_run/instance1/models/sql:/usr/src/app/models/sql
-      - ./integration_test_run/instance1/models/distributed:/usr/src/app/models/distributed
-      - ./integration_test_run/instance1/models/zendro-server:/usr/src/app/models/zendro-server
-      - ./integration_test_run/instance1/models/generic:/usr/src/app/models/generic
-      - ./integration_test_run/instance1/migrations:/usr/src/app/migrations
-      - ./integration_test_run/instance1/validations:/usr/src/app/validations
-      - ./integration_test_run/instance1/patches:/usr/src/app/patches
-      - ./sequelize_config_instance1.json:/usr/src/app/config/config.json
-
+      - ./integration_test_run/servers/instance1:/usr/src/app
+      - ./data_models_storage_config1.json:/usr/src/app/config/data_models_storage_config.json
     ports:
       - "3000:3000"
-
     environment:
       PORT: 3000
       REQUIRE_SIGN_IN: "false"
       LIMIT_RECORDS: 25
+      JQ_PATH: /usr/bin/jq
     # Await POSTGRES role and DB creation, migrate schema, then start web server:
     networks:
       - integrationtest
@@ -47,7 +36,6 @@ services:
       - /bin/sh
       - -c
       - |
-        npm install
         ./migrateDbAndStartServer.sh
 
   gql_ncbi_sim_srv1:
@@ -96,17 +84,8 @@ services:
       dockerfile: Dockerfile.graphql_server
 
     volumes:
-      - ./integration_test_run/instance2/models/adapters:/usr/src/app/models/adapters
-      - ./integration_test_run/instance2/schemas:/usr/src/app/schemas
-      - ./integration_test_run/instance2/resolvers:/usr/src/app/resolvers
-      - ./integration_test_run/instance2/models/sql:/usr/src/app/models/sql
-      - ./integration_test_run/instance2/models/distributed:/usr/src/app/models/distributed
-      - ./integration_test_run/instance2/models/zendro-server:/usr/src/app/models/zendro-server
-      - ./integration_test_run/instance2/models/generic:/usr/src/app/models/generic
-      - ./integration_test_run/instance2/migrations:/usr/src/app/migrations
-      - ./integration_test_run/instance2/validations:/usr/src/app/validations
-      - ./integration_test_run/instance2/patches:/usr/src/app/patches
-      - ./sequelize_config_instance2.json:/usr/src/app/config/config.json
+      - ./integration_test_run/servers/instance2:/usr/src/app
+      - ./data_models_storage_config2.json:/usr/src/app/config/data_models_storage_config.json
 
     ports:
       - "3030:3030"
@@ -115,6 +94,7 @@ services:
       PORT: 3030
       REQUIRE_SIGN_IN: "false"
       LIMIT_RECORDS: 25
+      JQ_PATH: /usr/bin/jq
     # Await POSTGRES role and DB creation, migrate schema, then start web server:
     networks:
       - integrationtest
@@ -123,7 +103,6 @@ services:
       - /bin/sh
       - -c
       - |
-        npm install
         ./migrateDbAndStartServer.sh
 
 networks: