From 5dd7a3ed8a5c967ae29fd9927c4ceddcc86493a3 Mon Sep 17 00:00:00 2001 From: Caelan U Date: Thu, 2 Jun 2022 15:47:18 -0700 Subject: [PATCH] add Argo Workflows to Pipeline frameworks & libraries --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index b7a8459..50cba33 100644 --- a/README.md +++ b/README.md @@ -12,8 +12,9 @@ Pipeline frameworks & libraries * [Airflow](https://github.com/airbnb/airflow) - Python-based workflow system created by AirBnb. * [Anduril](http://www.anduril.org/anduril/site/) - Component-based workflow framework for scientific data analysis. * [Antha](https://www.antha-lang.org/) - High-level language for biology. +* [Argo Workflows](https://argoproj.github.io/argo-workflows/) - Container-native workflow engine for orchestrating parallel data processing, ML, or CI jobs on Kubernetes. * [AWE](https://github.com/MG-RAST/AWE/) - Workflow and resource management system with CWL support. -* [Balsam](https://github.com/argonne-lcf/balsam) - Python-based high throughput task and workflow engine. +* [Balsam](https://github.com/argonne-lcf/balsam) - Python-based high throughput task and workflow engine. * [Bds](http://pcingola.github.io/BigDataScript/) - Scripting language for data pipelines. * [BioMake](https://github.com/evoldoers/biomake) - GNU-Make-like utility for managing builds and complex workflows. * [BioQueue](https://github.com/liyao001/BioQueue) - Explicit framework with web monitoring and resource estimation. @@ -60,7 +61,7 @@ Pipeline frameworks & libraries * [Loom](https://github.com/StanfordBioinformatics/loom) - Tool for running bioinformatics workflows locally or in the cloud. * [Longbow](http://www.hecbiosim.ac.uk/longbow) - Job proxying tool for biomolecular simulations. * [Luigi](https://github.com/spotify/luigi) - Python module that helps you build complex pipelines of batch jobs. -* [Maestro](https://github.com/LLNL/maestrowf) - YAML based HPC workflow execution tool. +* [Maestro](https://github.com/LLNL/maestrowf) - YAML based HPC workflow execution tool. * [Makeflow](http://ccl.cse.nd.edu/software/makeflow/) - Workflow engine for executing large complex workflows on clusters. * [Mara](https://github.com/mara/data-integration) - A lightweight, opinionated ETL framework, halfway between plain scripts and Apache Airflow. * [Mario](https://github.com/intentmedia/mario) - Scala library for defining data pipelines. @@ -156,7 +157,7 @@ Workflow platforms * [VisTrails](http://www.vistrails.org/) - Scientific workflow and provenance management system. * [Wings](http://www.wings-workflows.org) - Semantic workflow system utilizing Pegasus as execution system. * [Watchdog](https://github.com/klugem/watchdog) - Workflow management system for the automated and distributed analysis of large-scale experimental data. -* [FlowHub](https://www.flowhub.com.cn) - FlowHub is a new workflow cloud platform. +* [FlowHub](https://www.flowhub.com.cn) - FlowHub is a new workflow cloud platform. Workflow languages ------------------- @@ -175,7 +176,7 @@ Workflow standardization initiatives * [Workflow Patterns Library](http://www.workflowpatterns.com/patterns) * [ResearchObject.org](http://www.researchobject.org) -ETL & Data orchestration +ETL & Data orchestration ------------------------ * [DVC](https://dvc.org) - Data version control system for ML project with lightweight pipeline support. * [lakeFS](https://github.com/treeverse/lakeFS) - Repeatable, atomic and versioned data lake on top of object storage.