diff --git a/docs/digifeeds/overview.md b/docs/digifeeds/overview.md index 179a0be..bb54e4f 100644 --- a/docs/digifeeds/overview.md +++ b/docs/digifeeds/overview.md @@ -65,3 +65,87 @@ sequenceDiagram Script->>Script: Logs summary of script run Script->>+Prometheus Pushgateway: Sends metrics ``` + +## Process Barcodes in K8s + +```mermaid +sequenceDiagram + Workflow->>+Input S3 Folder: Gets the list of barcodes from zips + par For every Barcode + Workflow ->> Digifeeds DB: Get or add item for the given Barcode + Digifeeds DB ->> Workflow: Returns Item information + alt Item has added_to_digifeeds_set status + Workflow ->> Workflow: Succeed and continue + else Item does not have added_to_digifeeds_set status + critical Add Barcode to digifeeds Alma Set + Workflow->>Alma API: Add Barcode to Alma Set + option Barcode Doesn't exist + Alma API ->> Workflow: Returns Barcode not found error + Workflow ->> Digifeeds DB: Add not_found_in_alma status to Item + Workflow ->> Workflow: Error and exit for this barcode + option Barcode is already there + Alma API ->> Workflow: Returns Barcode already exists in set error + Workflow ->> Digifeeds DB: Adds added_to_digifeeds_set status to Item + option Success + Alm API ->> Workflow: Successfully added to set + Workflow->>Digifeeds DB: Adss added_to_digifeeds_set status to item + end + end + Workflow ->> Workflow: Check if the Item has an in_zephir status + alt It does + Workflow ->> Workflow: Succeed and continue + else It does not + Workflow ->> Zephir Bib API: Does Zephir have a corresponding record for the Barcode? + alt It does not + Zephir Bib API -->> Workflow: 404 or some other error + Workflow ->> Workflow: Log that the Item is not in Zephir.
Succeed and Continue + else It does + Zephir Bib API -->> Workflow: 200 Success + Workflow ->> Digifeeds DB: add in_zephir status to Item + Workflow ->> Workflow: Succeed and Continue + end + end + Workflow ->> Workflow: check if the Item has in_zephir status AND
has had it for more than two weeks + alt It does not meet those conditions + Workflow ->> Workflow: Log that the Item is not in Zephir. Succeed and Continue + else It does meet those conditions + Workflow ->> Digifeeds DB: Add copying_start status to Item + Workflow ->> Google Drive: Copy the corresponding zip from S3 to the Google Drive + Workflow ->> Digifeeds DB: Add copying_end status to Item + Workflow ->> Input S3 Folder: Move the corresponding zip
to the Processed Folder in the bucket + Workflow ->> Digifeeds DB: Add pending_deletion status to Item + end + end + Workflow->>+Prometheus Pushgateway: Sends metrics +``` + +## Process Barcodes High Level Overview + +```mermaidjs +flowchart TD + A(Get list of barcodes from zips in S3 Bucket) + B1(Add Barcode to Digifeeds Alma Set) + B2(Check Zephir for Barcode) + B3(Move zip from S3 to Google Drive if Ready) + + C1(Add Barcode to Digifeeds Alma Set) + C2(Check Zephir for Barcode) + C3(Move zip from S3 to Google Drive if Ready) + + D(Send metrics to Prometheus) + + A --> B1 + A --> C1 + subgraph "barcode 1" + B1 --> B2 + B2 --> B3 + end + subgraph "barcode 2" + C1 --> C2 + C2 --> C3 + end + + B3 --> D + C3 --> D + +```