diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
index 2eadacb..d971c0e 100644
--- a/.github/workflows/docs.yml
+++ b/.github/workflows/docs.yml
@@ -8,7 +8,7 @@ jobs:
max-parallel: 4
matrix:
os: [ubuntu-latest]
- python-version: [3.9]
+ python-version: ["3.11"]
runs-on: ${{ matrix.os }}
diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md
index 5607677..9426b08 100644
--- a/docs/guides/troubleshooting.md
+++ b/docs/guides/troubleshooting.md
@@ -2,6 +2,10 @@
This guide aims to provide general tips on troubleshooting and restoring services to a functional state.
+> NOTE:
+> Please also check questions other users have faced: [sensitive-data-archive Discussion](https://github.com/neicnordic/sensitive-data-archive/discussions?discussions_q=)
+> We encourage starting a discussion there as it will help us improve current guides.
+
## After deployment checklist
After having deployed the SDA services in a `FederatedEGA` setup, the following steps can be followed to ensure that everything is up and running correctly.
diff --git a/docs/structure.md b/docs/structure.md
index 87df9e7..f7a9f05 100644
--- a/docs/structure.md
+++ b/docs/structure.md
@@ -39,7 +39,7 @@ Inter-communication between services
There are three main ways that the system is passing on information and persist state in the system:
1. through AMQP messages sent from and to micro services;
-2. changes in the database of the status of a file being processed via the the `sda-pipeline`;
+2. changes in the database of the status of a file being processed via the the `SDA - Sensitive Data Archive` components;
3. location and state of files in either of the three file storage areas.
### AMQP messaging - Rabbit MQ
diff --git a/docs/submission.md b/docs/submission.md
index ac9e91d..32af8c2 100644
--- a/docs/submission.md
+++ b/docs/submission.md
@@ -43,7 +43,7 @@ Structure of the message and its contents are described in
SDA RabbitMQ-->>Intercept: Intercept reads message
Intercept -->> SDA RabbitMQ: Forwards ingest message
to queue
alt Ingest is successful
- Intercept->>Ingest: msg: [sda][ingest] begin ingestion
+ SDA RabbitMQ->>Ingest: msg: [sda][ingest] begin ingestion
activate Ingest
Ingest->>SDA Database: mark ingested
Note over Ingest: store file in Archive
@@ -68,7 +68,7 @@ Structure of the message and its contents are described in
Central EGA RabbitMQ-->>SDA RabbitMQ: federated msg: [from_cega][accession type]
SDA RabbitMQ-->>Intercept: Intercept reads message
Intercept -->> SDA RabbitMQ: Forwards accession ID message
to queue
- Intercept->>Finalize: msg: [sda][accession] map file to accession ID
+ SDA RabbitMQ->>Finalize: msg: [sda][accession] map file to accession ID
alt Finalize is successful
activate Finalize
note right of Finalize: Finalize makes the file backup
@@ -82,9 +82,9 @@ Structure of the message and its contents are described in
SDA RabbitMQ-->>Central EGA RabbitMQ: shovel msg:[to_cega][files.completed]
Central EGA RabbitMQ-->>SDA RabbitMQ: federated msg: [from_cega][mappings type]
SDA RabbitMQ-->>Intercept: Intercept reads message
- Intercept -->> SDA RabbitMQ: Forwards mapper message
to queue
- Intercept->>Mapper: msg: [sda][mappings] begin ingestion
- alt Mapper is successful
+ Intercept -->> SDA RabbitMQ: Forwards mapper message of type mapping
to queue
+ SDA RabbitMQ->>Mapper: msg: [sda][mappings] map dataset to file accession ID
+ alt Mapper Mapper creates dataset ID to file accession ID mapping
activate Mapper
Mapper->>SDA Database: map file to dataset accession ID
Mapper->>Inbox: remove file from inbox
@@ -92,6 +92,28 @@ Structure of the message and its contents are described in
Mapper-->>SDA RabbitMQ: msg: error
SDA RabbitMQ-->>Central EGA RabbitMQ: shovel msg:[to_cega][files.error]
end
+ Central EGA RabbitMQ-->>SDA RabbitMQ: federated msg: [from_cega][release type]
+ SDA RabbitMQ-->>Intercept: Intercept reads message
+ Intercept -->> SDA RabbitMQ: Forwards mapper message
to queue
+ SDA RabbitMQ->>Mapper: msg: [sda][mappings] release dataset
+ alt Mapper flags dataset ready for release
+ activate Mapper
+ Mapper->>SDA Database: flag dataset ready for release
+ else Error occurred in mapper process
+ Mapper-->>SDA RabbitMQ: msg: error
+ SDA RabbitMQ-->>Central EGA RabbitMQ: shovel msg:[to_cega][files.error]
+ end
+ Central EGA RabbitMQ-->>SDA RabbitMQ: federated msg: [from_cega][deprecate type]
+ SDA RabbitMQ-->>Intercept: Intercept reads message
+ Intercept -->> SDA RabbitMQ: Forwards mapper message
to queue
+ SDA RabbitMQ->>Mapper: msg: [sda][mappings] deprecate dataset
+ alt Mapper flags dataset as deprecated
+ activate Mapper
+ Mapper->>SDA Database: flag dataset as deprecated
+ else Error occurred in mapper process
+ Mapper-->>SDA RabbitMQ: msg: error
+ SDA RabbitMQ-->>Central EGA RabbitMQ: shovel msg:[to_cega][files.error]
+ end
deactivate Mapper
```
@@ -135,7 +157,7 @@ At this stage, the associated decryption key is retrieved. If decryption
completes and the checksum is valid, a message of completion is sent to
`CentralEGA`: Ingestion completed.
-> **Important**
+> Important:
> If a file disappears or is overwritten in the inbox before ingestion is completed, ingestion may not be possible.
Should any of the aforementioned steps result in an error, the workflow is terminated, and the error is logged. If the error is attributed to user misuse, such as providing an incorrect checksum or tampering with the encrypted file, it is reported to `CentralEGA` for display in the Submission Interface.
diff --git a/readthedocs.yaml b/readthedocs.yaml
index 0e0e3d0..e2645f9 100644
--- a/readthedocs.yaml
+++ b/readthedocs.yaml
@@ -7,9 +7,9 @@ version: 2
# Set the version of Python and other tools you might need
build:
- os: ubuntu-20.04
+ os: ubuntu-22.04
tools:
- python: "3.9"
+ python: "3.11"
mkdocs:
configuration: mkdocs.yml