As of 2023/06/15, this repository holds the official version of SciCat backend This is the only version that is officially supported. It is commonly referred to as backend, BE, v4.x or a combination of them.
You can find the old backend (v3.x, loopback implementation) at (https://github.com/SciCatProject/backend). The repo for backend v3.x is archived and read-only
If you have any questions, please feel free to contact any member of the development team, or the SciCat team at ESS.
If you're planning to contribute to this project by adding new functionality, we encourage you to discuss it first in the Discussions tab. This helps ensure that your proposed changes align with the project's goals and prevents duplicate efforts. Here's how you can initiate a discussion:
- Go to the Discussions tab.
- Start a new discussion thread outlining your proposed changes.
- Wait for feedback and consensus before proceeding with creating a pull request.
Thank you for your interest in contributing to our project!
git clone https://github.com/SciCatProject/scicat-backend-next.git
npm install
- Add .env file to project root folder. See Environment variables.
- Optional Add functionalAccounts.json file to project root folder to create local users.
- Optional Add loggers.json file to the root folder and configure multiple loggers.
npm run start:dev
- Go to http://localhost:3000/explorer to get an overview of available endpoints and database schemas.
- To be able to run the e2e tests with the same setup as in the Github actions you will need to run
npm run prepare:local
and after that runnpm run start:dev
. This will start all needed containers and copy some configuration to the right place.
git clone https://github.com/SciCatProject/scicat-backend-next.git
- docker-compose -f docker-compose.dev.yaml up -d
- Optional Mount functionalAccounts.json file to a volume in the container to create local users.
- Optional Mount loggers.json file to a volume in the container to configure multiple loggers.
- Optional change the container env variables
- Attach to the container
npm run start:dev
- Go to http://localhost:3000/explorer to get an overview of available endpoints and database schemas.
- Running the unit tests:
npm run test
- Running the e2e(api) tests:
- First of all run
npm run prepare:local
to prepare the local environment for starting - After that run
npm run test:api
which will start the backend locally and run bothjest
andmocha
e2e(api) tests. - [Optional] If you want to run only the mocha tests you will need to start the backend locally with
npm run start
and then usenpm run test:api:mocha
- [Optional] If you want to run only the jest tests you can use
npm run test:api:jest
There are multiple ways to configure your SciCat instance.
In order to configure a SciCat instance run on barebone OS, there are three options:
- Edit directly the file
src/config/configuration.ts
- Create a
.env
file with your local value of the variables listed in the next session. Only the variables that are required by your installation should be defined. To create your.env
file, you can copy and edit the sample.env.example
provided with in code - Define in your environment all the necessary variables (list provided below), prior running SciCat.
If SciCat runs in a containeraized environment, like docker or kubernetes, you can run the release image and specify your configuration using one of the following two methods:
- create
.env
and mount it directly in your container. - define the necessary environment variables directly in your container.
More information are provided in the official documentation.
Providing a file called functionalAccounts.json at the root of the project, locally or in the container, will automatically create the specified accounts on startup. If this file is not provided, no local users will be created.
Follow the structure of functionalAccounts.json.minimal.example to create your own functionalAccounts.json file.
Providing a file called loggers.json at the root of the project, locally or in the container, and create an external logger class in the src/loggers/loggingProviders/
directory will automatically create specified one or multiple loggers instances.
The loggers.json.example
file in the root directory showcases the example of configuration structure for the one or multiple loggers. logger.service.ts
file contains the configuration handling process logic, and src/loggers/loggingProviders/grayLogger.ts
includes actual usecase of grayLogger.
Valid environment variables for the .env file. See .env.example for examples value formats.
Environment Variable | Type | Optional | Description | Default Value |
---|---|---|---|---|
ADMIN_GROUPS |
string | Yes | Comma-separated list of admin groups with admin permission assigned to the listed users. Example: "admin, ingestor". For more details check: Scicat Documentation | |
CREATE_DATASET_GROUPS |
string | Yes | Comma-separated list of create dataset groups. Users belong to the listed groups can create dataset with/without PID. Example: "group1, group2". For more details check: Scicat Documentation | |
CREATE_DATASET_WITH_PID_GROUPS |
string | Yes | Comma-separated list of create dataset with pid groups. Users belong to the listed groups can create dataset with PID. Example: "group1, group2". For more details check: Scicat Documentation | |
DELETE_GROUPS |
string | Yes | Comma-separated list of delete groups. Users belong to the listed groups can delete any dataset, origDatablocks, datablocks, etc. For more details check: Scicat Documentation | |
DATASET_CREATION_VALIDATION_ENABLED |
boolean | Flag to enable/disable dataset validation to validate if requested new dataset is valid with given regular expression. Preconfigure DATASET_CREATION_VALIDATION_REGEX variable is required. | false | |
DATASET_CREATION_VALIDATION_REGEX |
string | Regular expression validation for new dataset request. | "" | |
PROPOSAL_GROUPS |
string | Yes | Comma-separated list of proposal groups with permission to create any proposals. Example: "proposaladmin, proposalingestor". For more details check: Scicat Documentation | |
SAMPLE_GROUPS |
string | Yes | Comma-separated list of sample groups with permission to create any samples. Example: "sampleadmin, sampleingestor". For more details check: Scicat Documentation | |
ACCESS_GROUPS_GRAPHQL_ENABLED |
string | Yes | Flag to enable/disable the GraphQL service to get access groups. Requires configuration of ACCESS_GROUP_SERVICE_TOKEN , ACCESS_GROUP_SERVICE_API_URL , and ACCESS_GROUP_SERVICE_HANDLER . |
true |
ACCESS_GROUPS_SERVICE_TOKEN |
string | Yes | Authentication token used if access groups are obtained from a third-party service. Not used by the vanilla installation, but only if the instance is customized to use an external service to provide user groups, like the ESS example. | |
ACCESS_GROUP_SERVICE_API_URL |
string | Yes | URL of the service providing the users' access groups. Not used by the vanilla installation, but only if the instance is customized to use an external service to provide user groups, like the ESS example. | |
ACCESS_GROUP_SERVICE_HANDLER |
string | Yes | Configuration property that points to the source of a module. This module provides a specific responseProcessor function for handling GraphQL responses and a query template for making GraphQL requests. | |
ACCESS_GROUPS_STATIC_ENABLED |
string | Yes | Flag to enable/disable automatic assignment of predefined access groups to all users. | true |
ACCESS_GROUPS_STATIC_VALUES |
string | Yes | Comma-separated list of access groups automatically assigned to all users. Example: "scicat, user". | |
ACCESS_GROUPS_OIDCPAYLOAD_ENABLED |
string | Yes | Flag to enable/disable fetching access groups directly from OIDC response. Requires specifying a field via OIDC_ACCESS_GROUPS_PROPERTY to extract access groups. |
false |
DOI_PREFIX |
string | The facility DOI prefix, with trailing slash. | ||
EXPRESS_SESSION_SECRET |
string | Yes | Secret used to set up express session. | |
HTTP_MAX_REDIRECTS |
number | Yes | Max redirects for HTTP requests. | 5 |
HTTP_TIMEOUT |
number | Yes | Timeout for HTTP requests in ms. | 5000 |
JWT_SECRET |
string | The secret for your JWT token, used for authorization. | ||
JWT_EXPIRES_IN |
number | Yes | How long, in seconds, the JWT token is valid. | 3600 |
LDAP_URL |
string | Yes | The URL to your LDAP server. | |
LDAP_BIND_DN |
string | Yes | Bind_DN for your LDAP server. | |
LDAP_BIND_CREDENTIALS |
string | Yes | Credentials for your LDAP server. | |
LDAP_SEARCH_BASE |
string | Yes | Search base for your LDAP server. | |
LDAP_SEARCH_FILTER |
string | Yes | Search filter for your LDAP server. | |
OIDC_ISSUER |
string | Yes | URL of the OIDC server providing the authentication service. Example: https://identity.esss.dk/realm/ess. | |
OIDC_CLIENT_ID |
string | Yes | Identity of the client used to obtain the user token. Example: scicat. | |
OIDC_CLIENT_SECRET |
string | Yes | Secret to provide to the OIDC service to obtain the user token. Example: Aa1JIw3kv3mQlGFWhRrE3gOdkH6xreAwro. | |
OIDC_CALLBACK_URL |
string | Yes | SciCat callback URL to redirect to after a successful login. Example: http://myscicat/api/v3/oidc/callback. | |
OIDC_SCOPE |
string | Yes | Space-separated list of info returned by the OIDC service. Example: "openid profile email". | |
OIDC_SUCCESS_URL |
string | Yes | SciCat Frontend auth-callback URL. Required to pass user credentials to SciCat Frontend after OIDC login. Example: https://myscicatfrontend/auth-callback. | |
OIDC_ACCESS_GROUPS |
string | Yes | Functionality is still unclear. | |
OIDC_ACCESS_GROUPS_PROPERTY |
string | Yes | Target field to get the access groups value from OIDC response. | |
OIDC_USERINFO_MAPPING_FIELD_USERNAME |
string | Yes | Comma-separated list of fields from the OIDC response to use as the user's profile username. Example: OIDC_USERINFO_MAPPING_FIELD_USERNAME="iss, sub" . |
"preferred_username" || "name" |
OIDC_USERINFO_MAPPING_FIELD_DISPLAYNAME |
string | Yes | Field from the OIDC response to use as the user's profile display name. Example: OIDC_USERINFO_MAPPING_FIELD_DISPLAYNAME="preferred_username" . |
"name" |
OIDC_USERINFO_MAPPING_FIELD_EMAIL |
string | Yes | Field from the OIDC response to use as the user's profile email. | "email" |
OIDC_USERINFO_MAPPING_FIELD_FAMILYNAME |
string | Yes | Field from the OIDC response to use as the user's profile family name. | "family_name" |
OIDC_USERINFO_MAPPING_FIELD_ID |
string | Yes | Field from the OIDC response to use as the user's profile ID. | "sub" || "user_id" |
OIDC_USERINFO_MAPPING_FIELD_THUMBNAILPHOTO |
string | Yes | Field from the OIDC response to use as the user's profile thumbnail photo. | "thumbnailPhoto" |
OIDC_USERINFO_MAPPING_FIELD_PROVIDER |
string | Yes | Field from the OIDC response to use as the user's profile provider. | "iss" |
OIDC_USERINFO_MAPPING_FIELD_GROUP |
string | Yes | Field from the OIDC response to use as the user's profile group. | "groups" |
OIDC_USERQUERY_OPERATOR |
string | Yes | Specifies the operator ("or" or "and") for UserModel.findOne queries, determining the logic used to match fields like "username" or "email". Example: UserModel.findOne({$or: {"username":"testUser", "email":"test@test.com"}}) . |
"or" |
OIDC_USERQUERY_FILTER |
string | Yes | Defines key-value pairs for UserModel.findOne queries, using a "key:value" format. Example: OIDC_USERQUERY_FILTER="username:sub, email:email" . |
"username:username, email:email" |
LOGBOOK_ENABLED |
string | Yes | Flag to enable/disable the Logbook endpoints. Values "yes" or "no". | "no" |
LOGBOOK_BASE_URL |
string | Yes | The base URL to the Logbook API. Only required if Logbook is enabled. | |
METADATA_KEYS_RETURN_LIMIT |
number | Yes | The return limit for the /Datasets/metadataKeys endpoint. |
|
METADATA_PARENT_INSTANCES_RETURN_LIMIT |
number | Yes | The return limit of Datasets to extract metadata keys from for the /Datasets/metadataKeys endpoint. |
|
MONGODB_URI |
string | The URI for your MongoDB instance. | ||
OAI_PROVIDER_ROUTE |
string | Yes | URI to OAI provider, used for the /publisheddata/:id/resync endpoint. |
|
PID_PREFIX |
string | The facility PID prefix, with trailing slash. | ||
PUBLIC_URL_PREFIX |
string | The base URL to the facility Landing Page. | ||
PORT |
number | Yes | The port on which you want to access the app. | 3000 |
RABBITMQ_ENABLED |
string | Yes | Flag to enable/disable RabbitMQ consumer. Values "yes" or "no". | "no" |
RABBITMQ_HOSTNAME |
string | Yes | The hostname of the RabbitMQ message broker. Only required if RabbitMQ is enabled. | |
RABBITMQ_USERNAME |
string | Yes | The username used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. | |
RABBITMQ_PASSWORD |
string | Yes | The password used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. | |
REGISTER_DOI_URI |
string | URI to the organization that registers the facility's DOIs. | ||
REGISTER_METADATA_URI |
string | URI to the organization that registers the facility's published data metadata. | ||
SITE |
string | The name of your site. | ||
SMTP_HOST |
string | Yes | Host of SMTP server. | |
SMTP_MESSAGE_FROM |
string | Yes | Email address that emails should be sent from. | |
SMTP_PORT |
string | Yes | Port of SMTP server. | |
SMTP_SECURE |
string | Yes | Secure of SMTP server. | |
POLICY_PUBLICATION_SHIFT |
integer | Yes | Embargo period expressed in years. | 3 years |
POLICY_RETENTION_SHIFT |
integer | Yes | Retention period (how long the facility will hold on to data) expressed in years. | -1 (indefinitely) |
ELASTICSEARCH_ENABLED |
string | Flag to enable/disable the Elasticsearch endpoints. Values "yes" or "no". | "no" | |
ES_HOST |
string | Host of Elasticsearch server instance. | ||
ES_USERNAME |
string | Yes | Elasticsearch username. | "elastic" |
ES_PASSWORD |
string | Elasticsearch password. | ||
MONGODB_COLLECTION |
string | Collection name to be mapped into specified Elasticsearch index. Used for data synchronization between MongoDB and Elasticsearch index. | ||
ES_MAX_RESULT |
number | Maximum records that can be indexed into Elasticsearch. | 10000 | |
ES_FIELDS_LIMIT |
number | The total number of fields in an index. | 1000 | |
ES_REFRESH |
string | If set to wait_for , Elasticsearch will wait till data is inserted into the specified index before returning a response. |
false | |
LOGGERS_CONFIG_FILE |
string | The file name for loggers configuration, located in the project root directory. | "loggers.json" |
Where the current SciCat Backend accepts id fields in the database named pid
, doi
, or similar, this implementation requires there to be an id field of the form _id
on every document. It is therefore necessary to run a database migration script towards MongoDB instance from a place where you have access to it and can install migrate-mongo
package.
All database migration scripts located in the migrations folder. To start the migration use:
npm run migrate:db:up
For down migration use:
npm run migrate:db:down
For listing applied DB migrations:
npm run migrate:db:status
For the full documentation please go to the SciCat home page and follow the documentation link
Following are the post that I found useful working on the migration:
- Schema and DTOs: https://betterprogramming.pub/how-to-use-data-transfer-objects-dto-for-validation-in-nest-js-7ff95309f650
- Validation:
- Swagger and OpenAPI:
Scicat Backend controls new releases with the GitHub-tag-and-release
GitHub Action. This workflow is triggered by push events to the release branch. It automates the version bumping and release processes based on semantic commit messages. Full documentation of the action package can be found on github-tag-action
The image below shows visualized workflow.
Caution
Any push to the release
branch initiates the workflow.
Semantic Versioning:
-
The version is automatically bumped according to the semantic PR titles, using the semantic-release conventions:
fix:
prefixed titles generate a patch release.feat:
prefixed titles generate a minor release.BREAKING CHANGE:
in the commit message triggers a major release.
Auto-generated Release Notes:
The release log is automatically populated with all commit messages since the last tag, providing a detailed changelog for the release. By default semantic-release uses Angular Commit Message Conventions.
In order to generate detailed changelog for the release, the following type for the commit message
/PR title
should be used:
- feat: message - A new feature
- fix: message - A bug fix
- docs: message - Documentation only changes
- style: message - Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
- refactor: message - A code change that neither fixes a bug nor adds a feature
- perf: message - A code change that improves performance
- test: message - Adding missing or correcting existing tests
- chore: message - Changes to the build process or auxiliary tools and libraries such as documentation generation
This project is licensed under the GPL License - see the LICENSE file for details
Scicat is developed at PSI, ESS and MAXIV with in-kind funding from ESS and from the European Union Framework Programme for Research and Innovation Horizon 2020, under grant agreement 676548, “BrightnESS”.