From dbfb2bcfc610eab6b8c60b4cf350034e0b58f397 Mon Sep 17 00:00:00 2001
From: mpc <matthewcoole@gmail.com>
Date: Mon, 21 Oct 2024 13:47:32 +0100
Subject: [PATCH] Updated readme /w mermaid format dag

---
 README.md | 67 +++++++++++++++++++++----------------------------------
 dag.md    | 23 -------------------
 2 files changed, 26 insertions(+), 64 deletions(-)
 delete mode 100644 dag.md

diff --git a/README.md b/README.md
index 20d03ac..3ccf9c1 100644
--- a/README.md
+++ b/README.md
@@ -28,48 +28,33 @@ This pipeline is defined in [`dvc.yaml`](dvc.yaml) and can be viewed with the co
 ```shell
 dvc dag
 ```
+or it can be output to mermaid format to display in markdown:
+```shell
+dvc dag -md
 ```
-                                  +----------------+                       
-                                  | fetch-metadata |                       
-                                  +----------------+                       
-                                  **               **                      
-                               ***                   ***                   
-                             **                         **                 
-                +------------------+            +-----------------------+  
-                | extract-metadata |            | fetch-supporting-docs |  
-                +------------------+            +-----------------------+  
-                                  **               **                      
-                                    ***         ***                        
-                                       **     **                           
-                                    +------------+                         
-                                    | chunk-data |                         
-                                    +------------+                         
-                                           *                               
-                                           *                               
-                                           *                               
-                                +-------------------+                      
-                                | create-embeddings |                      
-                                +-------------------+                      
-                                           *                               
-                                           *                               
-                                           *                               
-+------------------+            +--------------------+                     
-| generate-testset |            | upload-to-docstore |                     
-+------------------+            +--------------------+                     
-                  **              **                                       
-                    ***        ***                                         
-                       **    **                                            
-                +------------------+                                       
-                | run-rag-pipeline |                                       
-                +------------------+                                       
-                          *                                                
-                          *                                                
-                          *                                                
-                    +----------+                                           
-                    | evaluate |                                           
-                    +----------+  
+```mermaid
+flowchart TD
+	node1["chunk-data"]
+	node2["create-embeddings"]
+	node3["evaluate"]
+	node4["extract-metadata"]
+	node5["fetch-metadata"]
+	node6["fetch-supporting-docs"]
+	node7["generate-testset"]
+	node8["run-rag-pipeline"]
+	node9["upload-to-docstore"]
+	node1-->node2
+	node2-->node9
+	node4-->node1
+	node5-->node4
+	node5-->node6
+	node6-->node1
+	node7-->node8
+	node8-->node3
+	node9-->node8
+	node10["data/evaluation-sets.dvc"]
+	node11["data/synthetic-datasets.dvc"]
 ```
-
 > Note: To re-run the `fetch-supporting-docs` stage of the pipeline you will need to request access to the [Legilo](https://legilo.eds-infra.ceh.ac.uk/) service from the EDS dev team and provide your `username` and `password` in a `.env` file.
 
 ## Running Experiments
@@ -104,4 +89,4 @@ Notes on the use of Data Version Control and Continuous Machine Learning:
 
 ### vLLM
 Notes on running models with vLLM:
-- [vLLM](vllm.md)
\ No newline at end of file
+- [vLLM](vllm.md)
diff --git a/dag.md b/dag.md
deleted file mode 100644
index 91e16fe..0000000
--- a/dag.md
+++ /dev/null
@@ -1,23 +0,0 @@
-```mermaid
-flowchart TD
-	node1["chunk-data"]
-	node2["create-embeddings"]
-	node3["evaluate"]
-	node4["extract-metadata"]
-	node5["fetch-metadata"]
-	node6["fetch-supporting-docs"]
-	node7["generate-testset"]
-	node8["run-rag-pipeline"]
-	node9["upload-to-docstore"]
-	node1-->node2
-	node2-->node9
-	node4-->node1
-	node5-->node4
-	node5-->node6
-	node6-->node1
-	node7-->node8
-	node8-->node3
-	node9-->node8
-	node10["data/evaluation-sets.dvc"]
-	node11["data/synthetic-datasets.dvc"]
-```