Skip to content

Commit

Permalink
Updated readme /w mermaid format dag
Browse files Browse the repository at this point in the history
  • Loading branch information
matthewcoole committed Oct 21, 2024
1 parent e9b504f commit dbfb2bc
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 64 deletions.
67 changes: 26 additions & 41 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,48 +28,33 @@ This pipeline is defined in [`dvc.yaml`](dvc.yaml) and can be viewed with the co
```shell
dvc dag
```
or it can be output to mermaid format to display in markdown:
```shell
dvc dag -md
```
+----------------+
| fetch-metadata |
+----------------+
** **
*** ***
** **
+------------------+ +-----------------------+
| extract-metadata | | fetch-supporting-docs |
+------------------+ +-----------------------+
** **
*** ***
** **
+------------+
| chunk-data |
+------------+
*
*
*
+-------------------+
| create-embeddings |
+-------------------+
*
*
*
+------------------+ +--------------------+
| generate-testset | | upload-to-docstore |
+------------------+ +--------------------+
** **
*** ***
** **
+------------------+
| run-rag-pipeline |
+------------------+
*
*
*
+----------+
| evaluate |
+----------+
```mermaid
flowchart TD
node1["chunk-data"]
node2["create-embeddings"]
node3["evaluate"]
node4["extract-metadata"]
node5["fetch-metadata"]
node6["fetch-supporting-docs"]
node7["generate-testset"]
node8["run-rag-pipeline"]
node9["upload-to-docstore"]
node1-->node2
node2-->node9
node4-->node1
node5-->node4
node5-->node6
node6-->node1
node7-->node8
node8-->node3
node9-->node8
node10["data/evaluation-sets.dvc"]
node11["data/synthetic-datasets.dvc"]
```

> Note: To re-run the `fetch-supporting-docs` stage of the pipeline you will need to request access to the [Legilo](https://legilo.eds-infra.ceh.ac.uk/) service from the EDS dev team and provide your `username` and `password` in a `.env` file.
## Running Experiments
Expand Down Expand Up @@ -104,4 +89,4 @@ Notes on the use of Data Version Control and Continuous Machine Learning:

### vLLM
Notes on running models with vLLM:
- [vLLM](vllm.md)
- [vLLM](vllm.md)
23 changes: 0 additions & 23 deletions dag.md

This file was deleted.

0 comments on commit dbfb2bc

Please sign in to comment.