Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename BEs to V3 and V4 #127

Merged
merged 2 commits into from
Apr 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env
Original file line number Diff line number Diff line change
@@ -1 +1 @@
BE_VERSION=backend
BE_VERSION=v3
2 changes: 1 addition & 1 deletion .github/workflows/docker_compose_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
- lint
strategy:
matrix:
BE_VERSION: [backend, backendnext]
BE_VERSION: [v3, v4]
steps:
- uses: actions/checkout@v4
- name: Test docker-compose.yaml
Expand Down
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Note: older versions might not contain certain functionality (e.g. archival mock

By running `docker-compose up -d` these steps take place:
1. a [mongodb**](./services/mongodb/) container is created with some intial data.
2. the SciCat [backend v3*](./services/backend/) container is created and connected to (1).
2. the SciCat [backend v3*](./services/backendv3/) container is created and connected to (1).
3. the SciCat [frontend**](./services/frontend/) container is created and connected to (2).
4. the SciCat [PaN searchapi](./services/searchapi/) container is created and connected to (2).
5. a reverse [proxy](./services/proxy) container is created and routes traffic to (2), (3) and (4) through localhost subdomains, in the form: `http://${service}.localhost` (for the ones of need). The frontend is available at simply `http://localhost`.
Expand All @@ -36,7 +36,7 @@ Here below we show the dependencies (if `B` depends on `A`, then we visualize as
graph TD
subgraph services
subgraph backend
backends[backend*/backendnext*]
backends[v3*/v4*]
end
mongodb[mongodb**] --> backend
backend --> frontend[frontend**]
Expand All @@ -48,20 +48,20 @@ graph TD
proxy -.- searchapi
```

We flag with `*` the services which have extra internal dependencies, which are not shared across the two backend versions, and with `**` the ones which have a dependency on the `BE_VERSION` value. To view them, refer to the service README.
We flag with `*` the services which have extra internal dependencies, which are not shared across the two backend versions, and with `**` the ones which have an explicit dependency on the `BE_VERSION` value. To view them, refer to the service README.

## Select the BE version to use

The user can select what backend version to use, by setting the `BE_VERSION` environment variable (either `backend` or `backendnext`), [either](https://docs.docker.com/compose/environment-variables/envvars-precedence/) setting it in the shell or changing the [.env](./.env#L1) file. If this variable is blank, the system will default to `backendnext`. The services with `**` have a dependency on the `BE_VERSION` value. For any value of `BE_VERSION`, the `backend` is available at `http://backend.localhost`.
The user can select what backend version to use, by setting the `BE_VERSION` environment variable (either `v3` or `v4`), [either](https://docs.docker.com/compose/environment-variables/envvars-precedence/) setting it in the shell or changing the [.env](./.env#L1) file. If this variable is blank, the system will default to `v4`. The services with `**` have a dependency on the `BE_VERSION` value. For any value of `BE_VERSION`, the `backend` is available at `http://backend.localhost`.

For example, by running:

```sh
export BE_VERSION=backendnext
export BE_VERSION=v4
docker-compose up -d
```

Service (2) of the [default setup](README.md#default-setup) is replaced with the [backendnext* service](./services/backendnext/) and then steps from (1) to (5) are run.
Service (2) of the [default setup](README.md#default-setup) is replaced with the [v4* service](./services/backendv4/) and then steps from (1) to (5) are run.

After optionally setting the `BE_VERSION`, one can still select the services to run as described [here](README.md#select-the-services).

Expand Down Expand Up @@ -105,16 +105,16 @@ After any configuration change, `docker-compose up -d` must be rerun, to allow l

## Add a new service

To add a new service (see the [backend](./services/backend/) for an extensive example):
To add a new service (see the [backend v4](./services/backendv4/) for an extensive example):
1. create a dedicated folder in the [services](./services/) one
2. name it as the service
3. create the `docker-compose.yaml` file with the required dependencies (if any)
4. eventually include any service in (3) which is specific to the service and not shared across the global setup
5. eventually add the condition on the backend version (e.g. [here](./services/frontend/docker-compose.yaml#L14))
6. eventually create a `config` folder if it requires configuration
7. add a `README.md` file in the service if needed
7. eventually add a `README.md` file in the service
8. include the reference to (3) to the global [docker-compose include list](docker-compose.yaml#L2)
9. update the main [README.md](README.md) if needed
9. eventually update the main [README.md](README.md)

## General use of SciCat

Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
version: '3'
include:
- ./services/mongodb/docker-compose.yaml
- ./services/${BE_VERSION:-backendnext}/docker-compose.yaml
- ./services/backend${BE_VERSION:-v4}/docker-compose.yaml
- ./services/frontend/docker-compose.yaml
- ./services/searchapi/docker-compose.yaml
- ./services/proxy/docker-compose.yaml
4 changes: 2 additions & 2 deletions services/archivemock/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@ The Archive Mock simulates the interactions of an archival mock with SciCat.

## Service Requirements
- [RabbitMQ](../rabbitmq/)
- [SciCat Backend](../backend/) (configured to use the RabbitMQ instance above for jobs)
- [backend v3](../backendv3/) (configured to use the RabbitMQ instance above for jobs)

## [Config](./config/.env) - Environment Variables

The container uses [environment variables](https://github.com/SwissOpenEM/ScicatArchiveMock?tab=readme-ov-file#utility-scripts) for configuration.

## Default configuraiton

By default, it is configured to connect to the [Scicat Backend](../backend/) container with the `admin` account, and to the [RabbitMQ](../rabbitmq/) container with the `guest` account. It will then handle all archival and retrieval jobs posted to RabbitMQ, and update the corresponding Datasets accordingly in Scicat.
By default, it is configured to connect to the [backend v3](../backendv3/) container with the `admin` account, and to the [RabbitMQ](../rabbitmq/) container with the `guest` account. It will then handle all archival and retrieval jobs posted to RabbitMQ, and update the corresponding Datasets accordingly in Scicat.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [Backend](https://github.com/SciCatProject/backend)
# [Backend v3](https://github.com/SciCatProject/backend)

The SciCat backend v3 is the SciCat metadata catalogue RESTful API layer, built on top of the Loopback framework.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# [Backend next](https://github.com/SciCatProject/scicat-backend-next)
# [Backend v4](https://github.com/SciCatProject/scicat-backend-next)

The SciCat backend-next or v4 is a rewrite of the original backend, built on top of the NestJS framework.
The SciCat backend v4 is a rewrite of the original backend, built on top of the NestJS framework.

## Configuration options

Expand All @@ -25,5 +25,5 @@ Here below we show the internal dependencies of the service, which are not alrea

```mermaid
graph TD
elasticsearch --> backendnext
```
elasticsearch --> backend
```
File renamed without changes.
6 changes: 4 additions & 2 deletions services/frontend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@ For an extensive list of available options see [here](https://scicatproject.gith

## Default configuration

In the default configuration [config](./config/), the frontend is set to call the [backend container](../backend/) which is available on the `backend.localhost`.
In the default configuration [config](./config/), the frontend is set to call the `backend service` available at `backend.localhost` (either [v3](../backendv3/), by default, or [v4](../backendv4/) if specified otherwise by setting `BE_VERSION`).

For an explanation of how setting `BE_VERSION` changes the environment creation see [here](./README.md#dependency-on-be_version).

## Dependency on `BE_VERSION`

The `BE_VERSION` value controls if the `frontend` should connect to the [backend container](../backend/) or the [backendnext container](../backendnext/). This is controlled by merging conditionally the [config.v3.json file](./config/config.v3.json) which is applied when `BE_VERSION=backend` and connects the `frontend` to the [backend container](../backend/).
Since there was a small breaking change from `v3` to `v4`, when connecting to the `backend`, the `BE_VERSION` value controls if [config.v3.json file](./config/config.v3.json), which is applied when `BE_VERSION=v3`, should be included in the configs merge process.
2 changes: 1 addition & 1 deletion services/frontend/config/init.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@

apk update && apk add jq

[[ $BE_VERSION = "backendnext" ]] && CONFIGS=`find /config -name *.json ! -name "config.v3.json" | sort` || CONFIGS=`ls /config/*.json`
[[ $BE_VERSION = "v4" ]] && CONFIGS=`find /config -name *.json ! -name "config.v3.json" | sort` || CONFIGS=`ls /config/*.json`

jq -s 'reduce .[] as $item ({}; . * $item)' $CONFIGS > /usr/share/nginx/html/assets/config.json
2 changes: 1 addition & 1 deletion services/frontend/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@ services:
- -c
- /config/init.sh && nginx -g "daemon off;"
environment:
BE_VERSION: ${BE_VERSION:-backendnext}
BE_VERSION: ${BE_VERSION:-v4}
labels:
- traefik.http.routers.frontend.rule=Host(`localhost`)
10 changes: 5 additions & 5 deletions services/mongodb/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Mongodb

The `mongodb` container is responsible of creating a mongodb container with initial metadata that will later be used by the [backend](../backend/) container.
The `mongodb` container is responsible of creating a mongodb container with initial metadata.

## Configuration options

Expand All @@ -14,10 +14,10 @@ These files are ingested into the database using mongo funcionalities and bypass

## Default configuration

In the default configuration [init.js](./config/init.js), the seeding creates data in the [dacat](./config/init.js#L1) mongodb database used by the [backend container](../backend/)
In the default configuration [init.js](./config/init.js), the seeding creates data in the mongodb database used by the `backend` service (either [v3](../backendv3/), by default, or [v4](../backendv4/) if specified otherwise by setting `BE_VERSION`).

## Dependency on `BE_VERSION`
For an explanation of how setting `BE_VERSION` changes the environment creation see [here](./README.md#dependency-on-be_version).

The seeded database is controlled by the [BE_VERSION environment](./docker-compose.yaml#L9) variable set at startup. This will control what DB is seeded, either the [dacat](./config/init.js#L1) mongodb database used by the [backend container](../backend/) or the [dacat-next](./config/init.js#L1) mongodb database used by the [backendnext container](../backendnext/).
## Dependency on `BE_VERSION`

For an explanation of how setting `BE_VERSION` changes the environment creation see [here](./README.md#dependency-on-be_version).
Since [v3](../backendv3/) and [v4](../backendv4/) connect to two different DBs, the [BE_VERSION](./docker-compose.yaml#L9) environment variable controls [which DB](./config/init.js#L1) should be seeded (`dacat` for [v3](../backendv3/) and `dacat-next` for [v4](../backendv4/)).
2 changes: 1 addition & 1 deletion services/mongodb/config/init.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
db = connect(`mongodb://localhost/${process.env.BE_VERSION === 'backendnext'? 'dacat-next': 'dacat'}`);
db = connect(`mongodb://localhost/${process.env.BE_VERSION === 'v4'? 'dacat-next': 'dacat'}`);
seedFiles = fs.readdirSync('/seed');
seedFiles.forEach((filename) => {
collectionName = filename.replace(/\.json$/, '');
Expand Down
8 changes: 4 additions & 4 deletions services/mongodb/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@ services:
mongodb:
image: mongo:7.0
volumes:
- ${BE_VERSION:-backendnext}_mongodb_data:/data/db
- backend${BE_VERSION:-v4}_mongodb_data:/data/db
- ./config/init.js:/docker-entrypoint-initdb.d/init.js
- ./config/seed:/seed
environment:
BE_VERSION: ${BE_VERSION:-backendnext}
BE_VERSION: ${BE_VERSION:-v4}

volumes:
backendnext_mongodb_data:
backendv3_mongodb_data:
driver: local
backend_mongodb_data:
backendv4_mongodb_data:
driver: local
6 changes: 4 additions & 2 deletions services/searchapi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ The SciCat seachAPI is the SciCat metadata catalogue standardised API for commun

The searchapi configuration is set by the [.env file](./config/.env). For an extensive list of available options see [here](https://github.com/SciCatProject/panosc-search-api).

## Defaul configuration
## Default configuration

In the default configuration [.env file](./config/.env), the searchapi is set to call the [backend container](../backend/) which is available on `backend.localhost`.
In the default configuration [.env file](./config/.env), the searchapi is set to call the `backend service` available at `backend.localhost` (either [v3](../backendv3/), by default, or [v4](../backendv4/) if specified otherwise by setting `BE_VERSION`).

For an explanation of how setting `BE_VERSION` changes the environment creation see [here](./README.md#dependency-on-be_version).