From 646416d977403273ca06f2faeef915998e6970b1 Mon Sep 17 00:00:00 2001 From: Chris Gwilliams Date: Wed, 8 Nov 2023 15:50:19 +0100 Subject: [PATCH 1/4] add: Elasticsearch docs first draft --- Development/v4.x/backend/configuration.md | 315 ++++++++++-------- .../v4.x/elasticsearch/elasticsearch.md | 211 ++++++++++++ 2 files changed, 390 insertions(+), 136 deletions(-) create mode 100644 Development/v4.x/elasticsearch/elasticsearch.md diff --git a/Development/v4.x/backend/configuration.md b/Development/v4.x/backend/configuration.md index 180963d6..a9676395 100644 --- a/Development/v4.x/backend/configuration.md +++ b/Development/v4.x/backend/configuration.md @@ -10,329 +10,372 @@ If you are compiling the application from source, you can edit the file _serc/co This is complete the list of environment variable that can be used to configure SciCat backend. The list is compiled according to the configuration class defined in _src/config/configuration.ts_ -- ADMIN\_GROUPS: +- ADMIN_GROUPS: list of groups that have admin priviliges _default_: "" _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- DELETE\_GROUPS: +- DELETE_GROUPS: list of groups that are allowed to delete content _default_: "" _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- CREATE\_DATASET\_GROUPS: +- CREATE_DATASET_GROUPS: list of non admin groups that are allowed to create datasets without pid. The pid is assigned by the system. If set to "all", all users can create a dataset belonging to any of the groups they belong to. _default_: "#all" _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- CREATE\_DATASET\_WITH\_PID\_GROUPS: +- CREATE_DATASET_WITH_PID_GROUPS: list of non admin groups that are allowed to create datasets with explicit pid. If set to "#all", all users can create a dataset belonging to any of the groups they belong to and with esplicit pid. If the pid verification is enabled, pid will be validated agains the specification passed. _default_: "" _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- CREATE\_DATASET\_PRIVILEGED\_GROUPS: +- CREATE_DATASET_PRIVILEGED_GROUPS: list of non admin groups that are allowed to create datasets for groups they do not belong to. If set to "#all", all users can create a dataset belonging to any group with explicit pid. If the pid verification is enabled, pid will be validated agains the specification passed. _default_: "" - _format_: comma separated list of strings. Leading and trailing spaces are trimmed + _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- ACCESS\_GROUPS\_STATIC\_VALUES: - List of groups assigned by default to all users. Used in the vanilla implementation for easy configuration. +- ACCESS_GROUPS_STATIC_VALUES: + List of groups assigned by default to all users. Used in the vanilla implementation for easy configuration. If you do not want or need to assign any default group, it should be set to empty string "". Default value: "" - _format_: Comman separated list of strings. Leading and trailing spaces are trimmed - _example_: "group1,group2,group3,..." + _format_: Comman separated list of strings. Leading and trailing spaces are trimmed + _example_: "group1,group2,group3,..." -- ACCESS\_GROUP\_SERVICE\_TOKEN: - Access token needed to access the API specified in ACCESS_GROUP_SERVICE_API_URL, used to - retrieve access groups from a third party system. - _format_: string +- ACCESS_GROUP_SERVICE_TOKEN: + Access token needed to access the API specified in ACCESS*GROUP_SERVICE_API_URL, used to + retrieve access groups from a third party system. + \_format*: string -- ACCESS\_GROUP\_SERVICE\_API\_URL: +- ACCESS_GROUP_SERVICE_API_URL: Well formed url of the service API used to provide access groups. Only one value is allowed. - _format_: string - _example_: "https://my.access.group/service/api/url" - -- DOI\_PREFIX: + _format_: string + _example_: "https://my.access.group/service/api/url" +- DOI_PREFIX: The facility DOI prefix, with trailing slash. _default_: "" - _format_: string + _format_: string -- EXPRESS\_SESSION\_SECRET: +- EXPRESS_SESSION_SECRET: Secret used to set up express session. _default_: "" - _format_: string + _format_: string -- LOGOUT\_URL: +- LOGOUT_URL: URL specified upon successful logout. It is returned in the json object for the frontend, or third party UI, to be used locally. _default_: "" - _format_: string + _format_: string -- HTTP\_MAX\_REDIRECTS: +- HTTP_MAX_REDIRECTS: Max number of redirects for http requests. _default_: 5 - _format_: integer + _format_: integer -- HTTP\_TIMEOUT: +- HTTP_TIMEOUT: Timeout from http requests in ms. _default_: 5000 - _format_: integer - -- JWT\_SECRET: + _format_: integer +- JWT_SECRET: The secret used to create any JWT token, used for authorization. - _default_: "" - _format_: string + _default_: "" + _format_: string -- JWT\_EXPIRES\_IN: +- JWT_EXPIRES_IN: Expiration time of any JWT token in seconds. - _default_: 3600 (s) - _format_: integer + _default_: 3600 (s) + _format_: integer -- JWT\_NEVER\_EXPIRES: +- JWT_NEVER_EXPIRES: Length of time that the never expiring jwt token will last. _default_: 100y - _format_: string as in number of years + _format_: string as in number of years -- LDAP\_URL: +- LDAP_URL: Full URI (including port) of your local LDAP server, if this is your selected authentication method. _default_: No default _example_: ldaps://ldap.server.com:636/ - _format_: string + _format_: string -- LDAP\_BIND\_DN: +- LDAP_BIND_DN: Bind DN to access information on your LDAP server. _default_: No default - _format_: string + _format_: string -- LDAP\_BIND\_CREDENTIALS: +- LDAP_BIND_CREDENTIALS: Credentials associated with your bind DN to acccess your LDAP server. _default_: No default - _format_: string + _format_: string -- LDAP\_SEARCH\_BASE: +- LDAP_SEARCH_BASE: Search base for your LDAP server. _default_: No default - _format_: string + _format_: string -- LDAP\_SEARCH\_FILTER: +- LDAP_SEARCH_FILTER: Search filter for you LDAP server. _default_: No default _format_: string - _example_: "(LDAPUsername={{username}})" + _example_: "(LDAPUsername={{username}})" -- LDAP\_MODE: +- LDAP_MODE: type of ldap server we are communicating with - ___NEEDS TO BE UPDATED. Not sure which other values are accepted___ + **_NEEDS TO BE UPDATED. Not sure which other values are accepted_** _default_: ad _format_: string - _acceptable values_: ad - -- LDAP\_EXTERNAL\_ID: + _acceptable values_: ad +- LDAP_EXTERNAL_ID: LDAP matching field that provides the external id _default_: sAMAccountName - _format_: string + _format_: string -- LDAP\_USERNAME: +- LDAP_USERNAME: LDAP field providing the username _default_: displayName - _format_: string + _format_: string -- OIDC\_ISSUER: +- OIDC_ISSUER: Full URL of your OIDC identity provider _default_: No default _format_: string - _example_: "https://identity.your.facility/your/realm" + _example_: "https://identity.your.facility/your/realm" -- OIDC\_CLIENT\_ID: +- OIDC_CLIENT_ID: Client id used to convert OIDC code to OIDC token. This is assigned in the OIDC service when the token is generated _default_: No default _format_: string - _example_: "scicat" + _example_: "scicat" -- OIDC\_CLIENT\_SECRET: +- OIDC_CLIENT_SECRET: Token used to convert OIDC code to OIDC token. This is assigned in the OIDC service when the token is generated - _example_: "90f1268..." + _example_: "90f1268..." -- OIDC\_CALLBACK\_URL: +- OIDC_CALLBACK_URL: URL of the endpoint that is called when the authentication has been executed with the OIDC service. _default_: No default _format_: string - _example_: "http://localhost:3000/api/v3/oidc/callback" + _example_: "http://localhost:3000/api/v3/oidc/callback" -- OIDC\_SCOPE: +- OIDC_SCOPE: Information returned by the OIDC service together with token _default_: No default _format_: string - _example_: "openid profile email" + _example_: "openid profile email" -- OIDC\_SUCCESS\_URL: +- OIDC_SUCCESS_URL: Frontend URL that the user is directed to after a successful authentication. It must be a valid frontend URL. _default_: No default _format_: string - _example_: "http://localhost:3000/Datasets" + _example_: "http://localhost:3000/Datasets" -- OIDC\_ACCESS\_GROUPS: +- OIDC_ACCESS_GROUPS: field used to retrieve access groups from the OIDC service. It is not used in the vanilla implementation. _default_: No default _format_: string - _example_: "access_groups" + _example_: "access_groups" -- OIDC\_ACCESS\_GROUPS\_PROPERTY: +- OIDC_ACCESS_GROUPS_PROPERTY: name of the OIDC property used to retrieve the users groups from OIDC. _default_: none - _format_: string + _format_: string -- OIDC\_AUTO\_LOGOUT: +- OIDC_AUTO_LOGOUT: if enabled, when login out from SciCat, we logout from OIDC also. _default_: false - _format_: boolean + _format_: boolean -- OIDC\_RETURN\_URL: +- OIDC_RETURN_URL: URL the user is redirected after a successful logout _default_: none - _format_: string + _format_: string -- LOGBOOK\_ENABLED: +- LOGBOOK_ENABLED: Flag to enable/disable the Logbook endpoints. accept values: "yes", "no" _default_: no - _format_: string + _format_: string -- LOGBOOK\_BASE\_URL: +- LOGBOOK_BASE_URL: The base URL to the SciChat wrapper API. Only required if Logbook is enabled. - _default_: "http://localhost:3030/scichatapi" - _format_: string + _default_: "http://localhost:3030/scichatapi" + _format_: string -- LOGBOOK\_USERNAME: +- LOGBOOK_USERNAME: The username used to authenticate to the SciChat wrapper API. Only required if Logbook is enabled. _default_: No default - _format_: string + _format_: string -- LOGBOOK\_PASSWORD: - The password used to authenticate to the SciChat wrapper API. Only required if Logbook is enabled. - _default_: No default - _format_: string +- LOGBOOK_PASSWORD: + The password used to authenticate to the SciChat wrapper API. Only required if Logbook is enabled. + _default_: No default + _format_: string -- METADATA\_KEYS\_RETURN\_LIMIT: +- METADATA_KEYS_RETURN_LIMIT: The maximum number of keys returned by the `/Datasets/metadataKeys` endpoint. _default_: No default - _format_: integer + _format_: integer -- METADATA\_PARENT\_INSTANCES\_RETURN\_LIMIT: +- METADATA_PARENT_INSTANCES_RETURN_LIMIT: The maximum number of Datasets used to extract metadata keys in the `/Datasets/metadataKeys` endpoint. _default_: No default - _format_: integer + _format_: integer -- MONGODB\_URI: +- MONGODB_URI: The URI for your MongoDB instance. _default_: No default - _format_: string "mongodb://:@:27017/" - -- OAI\_PROVIDER\_ROUTE: + _format_: string "mongodb://:@:27017/" +- OAI_PROVIDER_ROUTE: URI to OAI provider, which is used in the `/publisheddata/:id/resync` endpoint. _default_: no default - _format_: string + _format_: string -- PID\_PREFIX: +- PID_PREFIX: The facility PID prefix, with trailing slash. _default_: no default - _format_: string + _format_: string -- PUBLIC\_URL\_PREFIX: +- PUBLIC_URL_PREFIX: The base URL to the facility Landing Page. _default_: No default _format_: string - _example_: "https://doi.ess.eu/detail/" + _example_: "https://doi.ess.eu/detail/" - PORT: The port on which the backend listen on. _default_: 3000 - _format_: integer + _format_: integer -- RABBITMQ\_ENABLED: +- RABBITMQ_ENABLED: Flag to enable/disable RabbitMQ consumer. accepted values: "yes", "no" _deprecated_. Will be removed in future releases. _default_: no - _format_: string - -- RABBITMQ\_HOSTNAME: + _format_: string +- RABBITMQ_HOSTNAME: The hostname of the RabbitMQ message broker. Only required if RabbitMQ is enabled. _deprecated_. Will be removed in future releases. _default_: no default - _default_: string + _default_: string -- RABBITMQ\_USERNAME: +- RABBITMQ_USERNAME: The username used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string + _format_: string -- RABBITMQ\_PASSWORD: - The password used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is - enabled. +- RABBITMQ_PASSWORD: + The password used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is + enabled. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string + _format_: string -- REGISTER\_DOI\_URI: +- REGISTER_DOI_URI: URI to the organization that registers the facilities DOIs. _default_: no default _format_: string - _example_: "https://mds.test.datacite.org/doi" + _example_: "https://mds.test.datacite.org/doi" -- REGISTER\_METADATA\_URI: +- REGISTER_METADATA_URI: URI to the organization that registers the facilities published data metadata. _default_: no default _format_: string - _example_: ="https://mds.test.datacite.org/metadata" + _example_: ="https://mds.test.datacite.org/metadata" -- DOI\_USERNAME: +- DOI_USERNAME: Username used to authenticate on the DOI site _default_: no default - _format_: string + _format_: string -- DOI\_PASSWORD: +- DOI_PASSWORD: Password used to authenticate on the DOI site _default_: no default - _format_: string + _format_: string -- SITE: +- SITE: The name of your site. _default_: no default - _format_: string + _format_: string -- SMTP\_HOST: - Host of SMTP server. +- SMTP_HOST: + Host of SMTP server. _deprecated_. Will be removed in future releases. - _default_: no default - _format_: string + _default_: no default + _format_: string -- SMTP\_MESSAGE\_FROM: +- SMTP_MESSAGE_FROM: Email address that emails should be sent from. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string, email + _format_: string, email -- SMTP\_PORT: +- SMTP_PORT: Port of SMTP server. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string + _format_: string -- SMTP\_SECURE: +- SMTP_SECURE: Secure of SMTP server. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string + _format_: string -- POLICY\_PUBLICATION\_SHIFT: +- POLICY_PUBLICATION_SHIFT: Number of years that needs to elapse before the dataset is made publicly acceessible _default_: 3 - _format_: integer + _format_: integer -- POLICY\_RETENTION\_SHIFT: - Number of years that the datasets are kept online before are archived or deleted. A negative value means that they are never archived/deleted +- POLICY_RETENTION_SHIFT: + Number of years that the datasets are kept online before are archived or deleted. A negative value means that they are never archived/deleted _default_: -1 - _format_: integer + _format_: integer + +- ELASTICSEARCH_ENABLED: + Flag to enable/disable the ElasticSearch service + accept values: "yes", "no" + _default_: no default + _format_: string + +- ES_HOST: + The base URL to the Elasticsearch cluster. Use `http` if xpack.security is disabled + _default_: no default + _format_: string + _example_: "https://localhost:9200" or "http://localhost:9200" + +- MONGODB_COLLECTION: + Collection name to be mapped into specified Elasticsearch index + _default_: no default + _format_: string + +- ES_MAX_RESULT: + Maximum records can be indexed into Elasticsearch. + _default_: 10000 + _format_: number + +- ES_FIELDS_LIMIT: + The total number of fields in an index. + _default_: 1000 + _format_: number + +- ES_INDEX: + The total number of fields in an index. + _default_: no default + _format_: string + +- ES_REFRESH: + The total number of fields in an index. + accept values: true, false, "wait_for" + _default_: false + _format_: boolean or string + +- ES_USERNAME: + Elasticsearch cluster username. + _default_: no default, optional. + _format_: string + +- ELASTIC_PASSWORD: + Elasticsearch cluster password. + _default_: no default. + _format_: string diff --git a/Development/v4.x/elasticsearch/elasticsearch.md b/Development/v4.x/elasticsearch/elasticsearch.md new file mode 100644 index 00000000..1b43155c --- /dev/null +++ b/Development/v4.x/elasticsearch/elasticsearch.md @@ -0,0 +1,211 @@ +# Getting started + +To be able to start application with Elasticsearch you will need to: + +1. Start with `npm prepare:local` to executes docker-compose file which pull and run Elasticsearch cluster as a docker container. +2. Start the application with `npm run start` + +# Check the .env file + +Following environment variables are required to run Elasticsearch with Scicat Backend: + +| `Variables` | Values | Description | +| ------------------------ | :--------------------- | :------------------------------------------------------------------------------------------------------------- | +| `ELASTICSEARCH_ENABLED` | `yes` or `no` | Flag to enable/disable the ElasticSearch service | +| `ES_HOST ` | https://localhost:9200 | Use `http` if `xpack.security.enabled` set to `false` | +| `ES_USERNAME` (optional) | elastic | Elasticsearch cluster username. Can be customized with docker image environment values with `ELASTIC_USERNAME` | +| `ES_PASSWORD` | password | Elasticsearch cluster password. Can be customized with docker image environment values `ELASTIC_PASSWORD` | +| `MONGODB_COLLECTION ` | Dataset | Collection name to be mapped into specified Elasticsearch index | +| `ES_MAX_RESULT ` | 210000 | Maximum records can be indexed into Elasticsearch. If not set, default is `10000` | +| `ES_FIELDS_LIMIT ` | 400000 | The total number of fields in an index. If not set, default is `1000` | +| `ES_INDEX ` | dataset | Setting default index for the application | +| `ES_REFRESH ` | wait_for | Wait for the change to become visible - CRUD actions | +| | | + +## \*\* NOTE \*\* + +`ES_REFRESH="wait_for"` is used for better testing flow. For production it should be set to false. Detailed docs about this setting can be found: [docs-refresh](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-refresh.html) + +# Steps to synchronize data with Elasitcsearch + +To enable search queries in Elasticsearch, you should adhere to the following steps: + +1. Open Swagger page `http://localhost:3000/explorer`, Authorize with `admin` token +2. Open elastic-search endpoints and execute `create-index`. + _**NOTE:**_ This step can be skipped if you want to use default index `dataset` +3. execute `sync-database` to sync data into the given index. + +If you see the following response, it means Elasticsearch is ready for use. + +``` +{ + "total": 21670, + "failed": 0, + "retry": 0, + "successful": 21670, + "noop": 0, + "time": 4777, + "bytes": 279159021, + "aborted": false +} +``` + +# Index, analysis and mapping settings + +Index settings in Elasticsearch are used to define various configurations and behaviors for an index. Index settings are vital for optimizing Elasticsearch's performance, relevance of search results, and resource management, ensuring that the index operates efficiently and effectively according to the specific needs of the application using it. + +Pre-defined index settings & dynamic mapping for nested objects can be found `/src/elastic-search/configuration/indexSetting.ts` + +Fields to be mapped into Elasticsearch index can be found `/src/elastic-search/configuration/datasetFieldMapping.ts` + +**Index settings:** + +``` +export const defaultElasticSettings = { + index: { + max_result_window: process.env.ES_MAX_RESULT || 2000000, + number_of_replicas: 0, + mapping: { + total_fields: { + limit: process.env.ES_FIELDS_LIMIT || 2000000, + }, + nested_fields: { + limit: 1000, + }, + }, + }, + analysis: { + analyzer: { + autocomplete: { + tokenizer: "autocomplete", + filter: ["lowercase"], + }, + autocomplete_search: { + tokenizer: "lowercase", + }, + }, + tokenizer: { + autocomplete: autocomplete_tokenizer, + }, + }, +} as IndicesIndexSettingsAnalysis; +``` + +| `Object` | Description | +| ----------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `index` | It contains settings that determine how an Elasticsearch index works, like how many results you can get from a search and how many copies of the data it should store. More details in [index modules](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html) | +| `index.max_result_window` | Determines the maximum number of documents returned in a query. `Default is 10000` | +| `index.number_of_replicas` | Sets the number of replica shards. `Default is 1` | +| `index.mapping.total_fields.limit` | Sets the maximum number of fields an index can have. `Default is 1000` | +| `index.mapping.nested_fields.limit` | Sets the maximum number of nested fields. `Default is 50` | +| `analysis` | It defines how text data is processed before it is indexed, including custom analyzers and tokenizers which control the conversion of text into tokens or terms. More details in [analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analyzer.html) and [analysis](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/analysis.html) | + +
+ +**Dynamic mapping:** + +Dynamic mapping is currently only used for scientific field. More details in [Dynamic template](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/dynamic-templates.html) + +``` +export const dynamic_template: Record[] = [ + { + string_as_keyword: { + path_match: "scientificMetadata.*.*", + match_mapping_type: "string", + mapping: { + type: "keyword", + ignore_above: 256, + }, + }, + }, + { + long_as_double: { + path_match: "scientificMetadata.*.*", + match_mapping_type: "long", + mapping: { + type: "double", + coerce: true, + ignore_malformed: true, + }, + }, + }, + { + date_as_keyword: { + path_match: "scientificMetadata.*.*", + match_mapping_type: "date", + mapping: { + type: "keyword", + ignore_above: 256, + }, + }, + }, +]; +``` + +# APIs + +`NOTE: All requests require admin permissions.` + +For search query integration it is suggested to use client search service directly, which can be found `/src/elastic-search/elastic-search.service.ts` + +**Following table is existing controller endpoints:** + +| endpoint | parameters | request | description | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------------------------------------------- | +| /api/v3/elastic-search/create-index?index= | `string` | query | Create an index with this name | +| /api/v3/elastic-search/sync-database?index= | `string` | query | Sync datasets into elastic search with this index | +| /api/v3/elastic-search/search | `{ "text": "", "ownerGroup": [], "creationLocation": [], "type": [], "keywords": [], "isPublished": false, "scientific": [{"lhs": "", "relation": "", "rhs": 0, "unit": ""}]}` | body | Search query | +| /api/v3/elastic-search/delete-index | `string` | query | Delete an index with this name | +| /api/v3/elastic-search/get-index | `string` | query | Get an index settings with this name | +| /api/v3/elastic-search/update-index | `to be updated` | to be updated | to be updated | + +# Setting up a local Elasticsearch cluster + +## Docker Configuration + +For local development, you can spin up an Elasticsearch cluster using Docker with the following configuration: + +_Elasticsearch image configuration can be found in `./CI/E2E/docker-compose-local.yaml`_ + +``` +es01: + image: docker.elastic.co/elasticsearch/elasticsearch:8.8.2 + ports: + - 9200:9200 + environment: + - xpack.security.enabled=false + - node.name=es01 + - ES_JAVA_OPTS=-Xms2g -Xmx2g + - cluster.name=es-cluster + - cluster.initial_master_nodes=es01 + - ELASTIC_PASSWORD=password + - bootstrap.memory_lock=true + mem_limit: 4g + ulimits: + memlock: + soft: -1 + hard: -1 +``` + +| **`Settings`** | **`Description`** | +| ---------------------------- | -------------------------------------------------------------------------------------- | +| Image Version | Docker image for Elasticsearch: `docker.elastic.co/elasticsearch/elasticsearch:8.8.2`. | +| Port | Container-to-host port mapping: `9200:9200`. | +| mem_limit | Maximum container memory, influenced by `ES_JAVA_OPTS`. | +| ulimits.memlock (soft, hard) | Memory locking limits: `-1` for unlimited. | +| **`Variables`** | **`Description`** | +| xpack.security.enabled | Toggle for Elasticsearch security; `false` disables authentication for development. | +| node.name | Unique Elasticsearch node name; essential for cluster node identification. | +| ES_JAVA_OPTS | JVM heap memory settings for Elasticsearch performance (e.g., `-Xms2g -Xmx2g`). | +| cluster.name | Name of the Elasticsearch cluster for node grouping. | +| ELASTIC_PASSWORD | Password for the 'elastic' user; not needed if security is disabled. | +| bootstrap.memory_lock | Prevents swapping Elasticsearch memory to disk when enabled. | + +## Using Helm for Kubernetes + +For those looking to deploy on Kubernetes, refer to the official [Helm chart documentation](https://github.com/elastic/helm-charts/tree/main/elasticsearch) for Elasticsearch. Helm charts provide a more production-ready setup and can manage complex configurations. + +**More configuration information:** + +- [More complicated examples with docker-compose configuration](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/docker.html) +- [More service configures for production](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/important-settings.html) From 87724673efc5c164053d692bbcdeab08e3d62baf Mon Sep 17 00:00:00 2001 From: Max Novelli Date: Tue, 14 Nov 2023 15:36:07 +0100 Subject: [PATCH 2/4] Fixed underscores with backslash --- Development/v4.x/backend/configuration.md | 135 +++++++++++----------- 1 file changed, 69 insertions(+), 66 deletions(-) diff --git a/Development/v4.x/backend/configuration.md b/Development/v4.x/backend/configuration.md index a9676395..1e2ca818 100644 --- a/Development/v4.x/backend/configuration.md +++ b/Development/v4.x/backend/configuration.md @@ -10,46 +10,46 @@ If you are compiling the application from source, you can edit the file _serc/co This is complete the list of environment variable that can be used to configure SciCat backend. The list is compiled according to the configuration class defined in _src/config/configuration.ts_ -- ADMIN_GROUPS: +- ADMIN\_GROUPS: list of groups that have admin priviliges _default_: "" _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- DELETE_GROUPS: +- DELETE\_GROUPS: list of groups that are allowed to delete content _default_: "" _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- CREATE_DATASET_GROUPS: +- CREATE\_DATASET\_GROUPS: list of non admin groups that are allowed to create datasets without pid. The pid is assigned by the system. If set to "all", all users can create a dataset belonging to any of the groups they belong to. _default_: "#all" _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- CREATE_DATASET_WITH_PID_GROUPS: +- CREATE\_DATASET\_WITH\_PID\_GROUPS: list of non admin groups that are allowed to create datasets with explicit pid. If set to "#all", all users can create a dataset belonging to any of the groups they belong to and with esplicit pid. If the pid verification is enabled, pid will be validated agains the specification passed. _default_: "" _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- CREATE_DATASET_PRIVILEGED_GROUPS: +- CREATE\_DATASET\_PRIVILEGED\_GROUPS: list of non admin groups that are allowed to create datasets for groups they do not belong to. If set to "#all", all users can create a dataset belonging to any group with explicit pid. If the pid verification is enabled, pid will be validated agains the specification passed. _default_: "" _format_: comma separated list of strings. Leading and trailing spaces are trimmed -- ACCESS_GROUPS_STATIC_VALUES: +- ACCESS\_GROUPS\_STATIC\_VALUES: List of groups assigned by default to all users. Used in the vanilla implementation for easy configuration. If you do not want or need to assign any default group, it should be set to empty string "". Default value: "" _format_: Comman separated list of strings. Leading and trailing spaces are trimmed _example_: "group1,group2,group3,..." -- ACCESS_GROUP_SERVICE_TOKEN: +- ACCESS\_GROUP\_SERVICE\_TOKEN: Access token needed to access the API specified in ACCESS*GROUP_SERVICE_API_URL, used to retrieve access groups from a third party system. \_format*: string -- ACCESS_GROUP_SERVICE_API_URL: +- ACCESS\_GROUP\_SERVICE\_API\_URL: Well formed url of the service API used to provide access groups. Only one value is allowed. _format_: string _example_: "https://my.access.group/service/api/url" @@ -58,22 +58,22 @@ The list is compiled according to the configuration class defined in _src/config _default_: "" _format_: string -- EXPRESS_SESSION_SECRET: +- EXPRESS\_SESSION\_SECRET: Secret used to set up express session. _default_: "" _format_: string -- LOGOUT_URL: +- LOGOUT\_URL: URL specified upon successful logout. It is returned in the json object for the frontend, or third party UI, to be used locally. _default_: "" _format_: string -- HTTP_MAX_REDIRECTS: +- HTTP\_MAX\_REDIRECTS: Max number of redirects for http requests. _default_: 5 _format_: integer -- HTTP_TIMEOUT: +- HTTP\_TIMEOUT: Timeout from http requests in ms. _default_: 5000 _format_: integer @@ -82,160 +82,162 @@ The list is compiled according to the configuration class defined in _src/config _default_: "" _format_: string -- JWT_EXPIRES_IN: +- JWT\_EXPIRES\_IN: Expiration time of any JWT token in seconds. _default_: 3600 (s) _format_: integer -- JWT_NEVER_EXPIRES: +- JWT\_NEVER\_EXPIRES: Length of time that the never expiring jwt token will last. _default_: 100y _format_: string as in number of years -- LDAP_URL: +- LDAP\_URL: Full URI (including port) of your local LDAP server, if this is your selected authentication method. _default_: No default _example_: ldaps://ldap.server.com:636/ _format_: string -- LDAP_BIND_DN: +- LDAP\_BIND\_DN: Bind DN to access information on your LDAP server. _default_: No default _format_: string -- LDAP_BIND_CREDENTIALS: +- LDAP\_BIND\_CREDENTIALS: Credentials associated with your bind DN to acccess your LDAP server. _default_: No default _format_: string -- LDAP_SEARCH_BASE: +- LDAP\_SEARCH\_BASE: Search base for your LDAP server. _default_: No default _format_: string -- LDAP_SEARCH_FILTER: +- LDAP\_SEARCH\_FILTER: Search filter for you LDAP server. _default_: No default _format_: string _example_: "(LDAPUsername={{username}})" -- LDAP_MODE: +- LDAP\_MODE: type of ldap server we are communicating with **_NEEDS TO BE UPDATED. Not sure which other values are accepted_** _default_: ad _format_: string - _acceptable values_: ad -- LDAP_EXTERNAL_ID: + _acceptable values_: ad + +- LDAP\_EXTERNAL\_ID: LDAP matching field that provides the external id _default_: sAMAccountName _format_: string -- LDAP_USERNAME: +- LDAP\_USERNAME: LDAP field providing the username _default_: displayName _format_: string -- OIDC_ISSUER: +- OIDC\_ISSUER: Full URL of your OIDC identity provider _default_: No default _format_: string _example_: "https://identity.your.facility/your/realm" -- OIDC_CLIENT_ID: +- OIDC\_CLIENT\_ID: Client id used to convert OIDC code to OIDC token. This is assigned in the OIDC service when the token is generated _default_: No default _format_: string _example_: "scicat" -- OIDC_CLIENT_SECRET: +- OIDC\_CLIENT\_SECRET: Token used to convert OIDC code to OIDC token. This is assigned in the OIDC service when the token is generated _example_: "90f1268..." -- OIDC_CALLBACK_URL: +- OIDC\_CALLBACK\_URL: URL of the endpoint that is called when the authentication has been executed with the OIDC service. _default_: No default _format_: string _example_: "http://localhost:3000/api/v3/oidc/callback" -- OIDC_SCOPE: +- OIDC\_SCOPE: Information returned by the OIDC service together with token _default_: No default _format_: string _example_: "openid profile email" -- OIDC_SUCCESS_URL: +- OIDC\_SUCCESS\_URL: Frontend URL that the user is directed to after a successful authentication. It must be a valid frontend URL. _default_: No default _format_: string _example_: "http://localhost:3000/Datasets" -- OIDC_ACCESS_GROUPS: +- OIDC\_ACCESS\_GROUPS: field used to retrieve access groups from the OIDC service. It is not used in the vanilla implementation. _default_: No default _format_: string _example_: "access_groups" -- OIDC_ACCESS_GROUPS_PROPERTY: +- OIDC\_ACCESS\_GROUPS\_PROPERTY: name of the OIDC property used to retrieve the users groups from OIDC. _default_: none _format_: string -- OIDC_AUTO_LOGOUT: +- OIDC\_AUTO\_LOGOUT: if enabled, when login out from SciCat, we logout from OIDC also. _default_: false _format_: boolean -- OIDC_RETURN_URL: +- OIDC\_RETURN\_URL: URL the user is redirected after a successful logout _default_: none _format_: string -- LOGBOOK_ENABLED: +- LOGBOOK\_ENABLED: Flag to enable/disable the Logbook endpoints. accept values: "yes", "no" _default_: no _format_: string -- LOGBOOK_BASE_URL: +- LOGBOOK\_BASE\_URL: The base URL to the SciChat wrapper API. Only required if Logbook is enabled. _default_: "http://localhost:3030/scichatapi" _format_: string -- LOGBOOK_USERNAME: +- LOGBOOK\_USERNAME: The username used to authenticate to the SciChat wrapper API. Only required if Logbook is enabled. _default_: No default _format_: string -- LOGBOOK_PASSWORD: +- LOGBOOK\_PASSWORD: The password used to authenticate to the SciChat wrapper API. Only required if Logbook is enabled. _default_: No default _format_: string -- METADATA_KEYS_RETURN_LIMIT: +- METADATA\_KEYS\_RETURN\_LIMIT: The maximum number of keys returned by the `/Datasets/metadataKeys` endpoint. _default_: No default _format_: integer -- METADATA_PARENT_INSTANCES_RETURN_LIMIT: +- METADATA\_PARENT\_INSTANCES\_RETURN\_LIMIT: The maximum number of Datasets used to extract metadata keys in the `/Datasets/metadataKeys` endpoint. _default_: No default _format_: integer -- MONGODB_URI: +- MONGODB\_URI: The URI for your MongoDB instance. _default_: No default _format_: string "mongodb://:@:27017/" -- OAI_PROVIDER_ROUTE: + +- OAI\_PROVIDER\_ROUTE: URI to OAI provider, which is used in the `/publisheddata/:id/resync` endpoint. _default_: no default _format_: string -- PID_PREFIX: +- PID\_PREFIX: The facility PID prefix, with trailing slash. _default_: no default _format_: string -- PUBLIC_URL_PREFIX: +- PUBLIC\_URL\_PREFIX: The base URL to the facility Landing Page. _default_: No default _format_: string @@ -246,49 +248,50 @@ The list is compiled according to the configuration class defined in _src/config _default_: 3000 _format_: integer -- RABBITMQ_ENABLED: +- RABBITMQ\_ENABLED: Flag to enable/disable RabbitMQ consumer. accepted values: "yes", "no" _deprecated_. Will be removed in future releases. _default_: no _format_: string -- RABBITMQ_HOSTNAME: + +- RABBITMQ\_HOSTNAME: The hostname of the RabbitMQ message broker. Only required if RabbitMQ is enabled. _deprecated_. Will be removed in future releases. _default_: no default _default_: string -- RABBITMQ_USERNAME: +- RABBITMQ\_USERNAME: The username used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. _deprecated_. Will be removed in future releases. _default_: no default _format_: string -- RABBITMQ_PASSWORD: +- RABBITMQ\_PASSWORD: The password used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. _deprecated_. Will be removed in future releases. _default_: no default _format_: string -- REGISTER_DOI_URI: +- REGISTER\_DOI\_URI: URI to the organization that registers the facilities DOIs. _default_: no default _format_: string _example_: "https://mds.test.datacite.org/doi" -- REGISTER_METADATA_URI: +- REGISTER\_METADATA\_URI: URI to the organization that registers the facilities published data metadata. _default_: no default _format_: string _example_: ="https://mds.test.datacite.org/metadata" -- DOI_USERNAME: +- DOI\_USERNAME: Username used to authenticate on the DOI site _default_: no default _format_: string -- DOI_PASSWORD: +- DOI\_PASSWORD: Password used to authenticate on the DOI site _default_: no default _format_: string @@ -298,84 +301,84 @@ The list is compiled according to the configuration class defined in _src/config _default_: no default _format_: string -- SMTP_HOST: +- SMTP\_HOST: Host of SMTP server. _deprecated_. Will be removed in future releases. _default_: no default _format_: string -- SMTP_MESSAGE_FROM: +- SMTP\_MESSAGE\_FROM: Email address that emails should be sent from. _deprecated_. Will be removed in future releases. _default_: no default _format_: string, email -- SMTP_PORT: +- SMTP\_PORT: Port of SMTP server. _deprecated_. Will be removed in future releases. _default_: no default _format_: string -- SMTP_SECURE: +- SMTP\_SECURE: Secure of SMTP server. _deprecated_. Will be removed in future releases. _default_: no default _format_: string -- POLICY_PUBLICATION_SHIFT: +- POLICY\_PUBLICATION\_SHIFT: Number of years that needs to elapse before the dataset is made publicly acceessible _default_: 3 _format_: integer -- POLICY_RETENTION_SHIFT: +- POLICY\_RETENTION\_SHIFT: Number of years that the datasets are kept online before are archived or deleted. A negative value means that they are never archived/deleted _default_: -1 _format_: integer -- ELASTICSEARCH_ENABLED: +- ELASTICSEARCH\_ENABLED: Flag to enable/disable the ElasticSearch service accept values: "yes", "no" _default_: no default _format_: string -- ES_HOST: +- ES\_HOST: The base URL to the Elasticsearch cluster. Use `http` if xpack.security is disabled _default_: no default _format_: string _example_: "https://localhost:9200" or "http://localhost:9200" -- MONGODB_COLLECTION: +- MONGODB\_COLLECTION: Collection name to be mapped into specified Elasticsearch index _default_: no default _format_: string -- ES_MAX_RESULT: +- ES\_MAX\_RESULT: Maximum records can be indexed into Elasticsearch. _default_: 10000 _format_: number -- ES_FIELDS_LIMIT: +- ES\_FIELDS\_LIMIT: The total number of fields in an index. _default_: 1000 _format_: number -- ES_INDEX: +- ES\_INDEX: The total number of fields in an index. _default_: no default _format_: string -- ES_REFRESH: +- ES\_REFRESH: The total number of fields in an index. accept values: true, false, "wait_for" _default_: false _format_: boolean or string -- ES_USERNAME: +- ES\_USERNAME: Elasticsearch cluster username. _default_: no default, optional. _format_: string -- ELASTIC_PASSWORD: +- ELASTIC\_PASSWORD: Elasticsearch cluster password. _default_: no default. _format_: string From 200c9b4a400d0db554353b8fa3aba94091edb373 Mon Sep 17 00:00:00 2001 From: Max Novelli Date: Tue, 14 Nov 2023 15:38:42 +0100 Subject: [PATCH 3/4] Added Elastic Search configuration to main menu --- SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUMMARY.md b/SUMMARY.md index 66d98177..ecc3f32c 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -48,6 +48,7 @@ * [OrigDatablocks](Development/v4.x/backend/authorization/authorization_origdatablocks.md) * [Jobs](Development/v4.x/backend/authorization/authorization_jobs.md) * [Users](Development/v4.x/backend/authorization/authorization_users.md) + * [Elastic Search](Development/v4.x/elasticsearch/elasticsearch.md) * [Frontend](Development/v4.x/frontend/configuration.md) * [Theming](Development/v4.x/Theming.md) * [Login and Accounts](Development/v4.x/Login.md) From e8b8245d606e262f06891fea02c56cd390aabd09 Mon Sep 17 00:00:00 2001 From: Max Novelli Date: Tue, 14 Nov 2023 15:50:35 +0100 Subject: [PATCH 4/4] fixed markdown spaces --- Development/v4.x/backend/configuration.md | 211 +++++++++++----------- 1 file changed, 106 insertions(+), 105 deletions(-) diff --git a/Development/v4.x/backend/configuration.md b/Development/v4.x/backend/configuration.md index 1e2ca818..45a10a3d 100644 --- a/Development/v4.x/backend/configuration.md +++ b/Development/v4.x/backend/configuration.md @@ -13,115 +13,116 @@ The list is compiled according to the configuration class defined in _src/config - ADMIN\_GROUPS: list of groups that have admin priviliges _default_: "" - _format_: comma separated list of strings. Leading and trailing spaces are trimmed + _format_: comma separated list of strings. Leading and trailing spaces are trimmed - DELETE\_GROUPS: list of groups that are allowed to delete content _default_: "" - _format_: comma separated list of strings. Leading and trailing spaces are trimmed + _format_: comma separated list of strings. Leading and trailing spaces are trimmed - CREATE\_DATASET\_GROUPS: - list of non admin groups that are allowed to create datasets without pid. The pid is assigned by the system. If set to "all", all users can create a dataset belonging to any of the groups they belong to. + list of non admin groups that are allowed to create datasets without pid. The pid is assigned by the system. If set to "all", all users can create a dataset belonging to any of the groups they belong to. _default_: "#all" - _format_: comma separated list of strings. Leading and trailing spaces are trimmed + _format_: comma separated list of strings. Leading and trailing spaces are trimmed - CREATE\_DATASET\_WITH\_PID\_GROUPS: - list of non admin groups that are allowed to create datasets with explicit pid. If set to "#all", all users can create a dataset belonging to any of the groups they belong to and with esplicit pid. + list of non admin groups that are allowed to create datasets with explicit pid. If set to "#all", all users can create a dataset belonging to any of the groups they belong to and with esplicit pid. If the pid verification is enabled, pid will be validated agains the specification passed. _default_: "" - _format_: comma separated list of strings. Leading and trailing spaces are trimmed + _format_: comma separated list of strings. Leading and trailing spaces are trimmed - CREATE\_DATASET\_PRIVILEGED\_GROUPS: - list of non admin groups that are allowed to create datasets for groups they do not belong to. If set to "#all", all users can create a dataset belonging to any group with explicit pid. + list of non admin groups that are allowed to create datasets for groups they do not belong to. If set to "#all", all users can create a dataset belonging to any group with explicit pid. If the pid verification is enabled, pid will be validated agains the specification passed. _default_: "" - _format_: comma separated list of strings. Leading and trailing spaces are trimmed + _format_: comma separated list of strings. Leading and trailing spaces are trimmed - ACCESS\_GROUPS\_STATIC\_VALUES: List of groups assigned by default to all users. Used in the vanilla implementation for easy configuration. If you do not want or need to assign any default group, it should be set to empty string "". Default value: "" _format_: Comman separated list of strings. Leading and trailing spaces are trimmed - _example_: "group1,group2,group3,..." + _example_: "group1,group2,group3,..." - ACCESS\_GROUP\_SERVICE\_TOKEN: - Access token needed to access the API specified in ACCESS*GROUP_SERVICE_API_URL, used to - retrieve access groups from a third party system. - \_format*: string + Access token needed to access the API specified in ACCESS\_GROUP\_SERVICE\_API\_URL, used to retrieve access groups from a third party system. + _format*: string - ACCESS\_GROUP\_SERVICE\_API\_URL: Well formed url of the service API used to provide access groups. Only one value is allowed. _format_: string _example_: "https://my.access.group/service/api/url" + - DOI_PREFIX: The facility DOI prefix, with trailing slash. _default_: "" - _format_: string + _format_: string - EXPRESS\_SESSION\_SECRET: Secret used to set up express session. _default_: "" - _format_: string + _format_: string - LOGOUT\_URL: URL specified upon successful logout. It is returned in the json object for the frontend, or third party UI, to be used locally. _default_: "" - _format_: string + _format_: string - HTTP\_MAX\_REDIRECTS: Max number of redirects for http requests. _default_: 5 - _format_: integer + _format_: integer - HTTP\_TIMEOUT: Timeout from http requests in ms. _default_: 5000 _format_: integer + - JWT_SECRET: The secret used to create any JWT token, used for authorization. _default_: "" - _format_: string + _format_: string - JWT\_EXPIRES\_IN: Expiration time of any JWT token in seconds. _default_: 3600 (s) - _format_: integer + _format_: integer - JWT\_NEVER\_EXPIRES: Length of time that the never expiring jwt token will last. _default_: 100y - _format_: string as in number of years + _format_: string as in number of years - LDAP\_URL: Full URI (including port) of your local LDAP server, if this is your selected authentication method. _default_: No default - _example_: ldaps://ldap.server.com:636/ - _format_: string + _example_: ldaps://ldap.server.com:636/ + _format_: string - LDAP\_BIND\_DN: Bind DN to access information on your LDAP server. _default_: No default - _format_: string + _format_: string - LDAP\_BIND\_CREDENTIALS: Credentials associated with your bind DN to acccess your LDAP server. _default_: No default - _format_: string + _format_: string - LDAP\_SEARCH\_BASE: Search base for your LDAP server. - _default_: No default - _format_: string + _default_: No default + _format_: string - LDAP\_SEARCH\_FILTER: Search filter for you LDAP server. _default_: No default - _format_: string - _example_: "(LDAPUsername={{username}})" + _format_: string + _example_: "(LDAPUsername={{username}})" - LDAP\_MODE: type of ldap server we are communicating with - **_NEEDS TO BE UPDATED. Not sure which other values are accepted_** + **_NEEDS TO BE UPDATED. Not sure which other values are accepted_** _default_: ad _format_: string _acceptable values_: ad @@ -129,256 +130,256 @@ The list is compiled according to the configuration class defined in _src/config - LDAP\_EXTERNAL\_ID: LDAP matching field that provides the external id _default_: sAMAccountName - _format_: string + _format_: string - LDAP\_USERNAME: LDAP field providing the username _default_: displayName - _format_: string + _format_: string - OIDC\_ISSUER: Full URL of your OIDC identity provider _default_: No default _format_: string - _example_: "https://identity.your.facility/your/realm" + _example_: "https://identity.your.facility/your/realm" - OIDC\_CLIENT\_ID: Client id used to convert OIDC code to OIDC token. This is assigned in the OIDC service when the token is generated _default_: No default _format_: string - _example_: "scicat" + _example_: "scicat" -- OIDC\_CLIENT\_SECRET: +- OIDC\_CLIENT\_SECRET: Token used to convert OIDC code to OIDC token. This is assigned in the OIDC service when the token is generated - _example_: "90f1268..." + _example_: "90f1268..." - OIDC\_CALLBACK\_URL: - URL of the endpoint that is called when the authentication has been executed with the OIDC service. - _default_: No default - _format_: string - _example_: "http://localhost:3000/api/v3/oidc/callback" + URL of the endpoint that is called when the authentication has been executed with the OIDC service. + _default_: No default + _format_: string + _example_: "http://localhost:3000/api/v3/oidc/callback" - OIDC\_SCOPE: Information returned by the OIDC service together with token _default_: No default - _format_: string - _example_: "openid profile email" + _format_: string + _example_: "openid profile email" - OIDC\_SUCCESS\_URL: Frontend URL that the user is directed to after a successful authentication. It must be a valid frontend URL. _default_: No default _format_: string - _example_: "http://localhost:3000/Datasets" + _example_: "http://localhost:3000/Datasets" - OIDC\_ACCESS\_GROUPS: field used to retrieve access groups from the OIDC service. It is not used in the vanilla implementation. _default_: No default _format_: string - _example_: "access_groups" + _example_: "access_groups" - OIDC\_ACCESS\_GROUPS\_PROPERTY: name of the OIDC property used to retrieve the users groups from OIDC. _default_: none - _format_: string + _format_: string - OIDC\_AUTO\_LOGOUT: if enabled, when login out from SciCat, we logout from OIDC also. _default_: false - _format_: boolean + _format_: boolean - OIDC\_RETURN\_URL: URL the user is redirected after a successful logout _default_: none - _format_: string + _format_: string - LOGBOOK\_ENABLED: Flag to enable/disable the Logbook endpoints. accept values: "yes", "no" - _default_: no - _format_: string + _default_: no + _format_: string - LOGBOOK\_BASE\_URL: The base URL to the SciChat wrapper API. Only required if Logbook is enabled. _default_: "http://localhost:3030/scichatapi" - _format_: string + _format_: string - LOGBOOK\_USERNAME: The username used to authenticate to the SciChat wrapper API. Only required if Logbook is enabled. _default_: No default - _format_: string + _format_: string - LOGBOOK\_PASSWORD: The password used to authenticate to the SciChat wrapper API. Only required if Logbook is enabled. _default_: No default - _format_: string + _format_: string - METADATA\_KEYS\_RETURN\_LIMIT: The maximum number of keys returned by the `/Datasets/metadataKeys` endpoint. _default_: No default - _format_: integer + _format_: integer - METADATA\_PARENT\_INSTANCES\_RETURN\_LIMIT: The maximum number of Datasets used to extract metadata keys in the `/Datasets/metadataKeys` endpoint. _default_: No default - _format_: integer + _format_: integer - MONGODB\_URI: The URI for your MongoDB instance. _default_: No default - _format_: string "mongodb://:@:27017/" + _format_: string "mongodb://:@:27017/" - OAI\_PROVIDER\_ROUTE: URI to OAI provider, which is used in the `/publisheddata/:id/resync` endpoint. _default_: no default - _format_: string + _format_: string - PID\_PREFIX: The facility PID prefix, with trailing slash. _default_: no default - _format_: string + _format_: string - PUBLIC\_URL\_PREFIX: The base URL to the facility Landing Page. _default_: No default _format_: string - _example_: "https://doi.ess.eu/detail/" + _example_: "https://doi.ess.eu/detail/" - PORT: The port on which the backend listen on. _default_: 3000 - _format_: integer + _format_: integer - RABBITMQ\_ENABLED: Flag to enable/disable RabbitMQ consumer. accepted values: "yes", "no" _deprecated_. Will be removed in future releases. _default_: no - _format_: string + _format_: string - RABBITMQ\_HOSTNAME: The hostname of the RabbitMQ message broker. Only required if RabbitMQ is enabled. _deprecated_. Will be removed in future releases. _default_: no default - _default_: string + _default_: string - RABBITMQ\_USERNAME: The username used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string + _format_: string - RABBITMQ\_PASSWORD: The password used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string + _format_: string - REGISTER\_DOI\_URI: URI to the organization that registers the facilities DOIs. _default_: no default _format_: string - _example_: "https://mds.test.datacite.org/doi" + _example_: "https://mds.test.datacite.org/doi" - REGISTER\_METADATA\_URI: URI to the organization that registers the facilities published data metadata. _default_: no default _format_: string - _example_: ="https://mds.test.datacite.org/metadata" + _example_: ="https://mds.test.datacite.org/metadata" - DOI\_USERNAME: Username used to authenticate on the DOI site _default_: no default - _format_: string + _format_: string - DOI\_PASSWORD: Password used to authenticate on the DOI site _default_: no default - _format_: string + _format_: string - SITE: The name of your site. _default_: no default - _format_: string + _format_: string - SMTP\_HOST: Host of SMTP server. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string + _format_: string - SMTP\_MESSAGE\_FROM: Email address that emails should be sent from. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string, email + _format_: string, email - SMTP\_PORT: Port of SMTP server. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string + _format_: string - SMTP\_SECURE: Secure of SMTP server. _deprecated_. Will be removed in future releases. _default_: no default - _format_: string + _format_: string - POLICY\_PUBLICATION\_SHIFT: Number of years that needs to elapse before the dataset is made publicly acceessible _default_: 3 - _format_: integer + _format_: integer - POLICY\_RETENTION\_SHIFT: Number of years that the datasets are kept online before are archived or deleted. A negative value means that they are never archived/deleted _default_: -1 - _format_: integer + _format_: integer - ELASTICSEARCH\_ENABLED: Flag to enable/disable the ElasticSearch service accept values: "yes", "no" - _default_: no default - _format_: string + _default_: no default + _format_: string - ES\_HOST: - The base URL to the Elasticsearch cluster. Use `http` if xpack.security is disabled - _default_: no default - _format_: string - _example_: "https://localhost:9200" or "http://localhost:9200" + The base URL to the Elasticsearch cluster. Use `http` if xpack.security is disabled + _default_: no default + _format_: string + _example_: "https://localhost:9200" or "http://localhost:9200" - MONGODB\_COLLECTION: - Collection name to be mapped into specified Elasticsearch index - _default_: no default - _format_: string - -- ES\_MAX\_RESULT: - Maximum records can be indexed into Elasticsearch. - _default_: 10000 - _format_: number + Collection name to be mapped into specified Elasticsearch index + _default_: no default + _format_: string + +- ES\_MAX\_RESULT: + Maximum records can be indexed into Elasticsearch. + _default_: 10000 + _format_: number -- ES\_FIELDS\_LIMIT: - The total number of fields in an index. - _default_: 1000 - _format_: number +- ES\_FIELDS\_LIMIT: + The total number of fields in an index. + _default_: 1000 + _format_: number - ES\_INDEX: - The total number of fields in an index. - _default_: no default - _format_: string + The total number of fields in an index. + _default_: no default + _format_: string - ES\_REFRESH: - The total number of fields in an index. - accept values: true, false, "wait_for" - _default_: false - _format_: boolean or string + The total number of fields in an index. + accept values: true, false, "wait_for" + _default_: false + _format_: boolean or string - ES\_USERNAME: - Elasticsearch cluster username. - _default_: no default, optional. - _format_: string - -- ELASTIC\_PASSWORD: - Elasticsearch cluster password. - _default_: no default. - _format_: string + Elasticsearch cluster username. + _default_: no default, optional. + _format_: string + +- ELASTIC\_PASSWORD: + Elasticsearch cluster password. + _default_: no default. + _format_: string