Merge branch 'master' into anandhu-eng-patch-1

.github/workflows/_publish.yaml_ → .github/workflows/publish-docs.yml

@@ -1,13 +1,16 @@
 name: Publish documentation site
 
-
 on:
   release:
     types: [published]
   push:
     branches:
-      - main
+      - master
       - docs
+    paths:
+      - docs/**
+      - mkdocs.yml
+      - README.md
 
 jobs:
 

CONTRIBUTING.md

@@ -14,12 +14,18 @@ Modify the project in your own fork and issue a pull request once you want other
 to take a look at what you have done and discuss the proposed changes. 
 Ensure that cla-bot and other checks pass for your Pull requests.
 
-Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://cKnowledge.org/gfursin),
-sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. 
-Since then, this open-source technology (CM, CM4MLOps, CM4ABTF, CM4Research, etc)
-is being developed as a community effort thanks to all our volunteers, collaborators and contributors
-listed here in alphabetical order:
+Collective Knowledge (CK) and Collective Mind (CM)
+were created by [Grigori Fursin](https://arxiv.org/abs/2406.16791),
+sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons 
+to benefit everyone. Since then, this open-source automation technology
+(CM/CMX, CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, etc) is being extended 
+as a community effort thanks to all our volunteers, collaborators 
+and contributors listed here in alphabetical order:
 
+* @Henryfzh
+* @Leonard226
+* @Oseltamivir
+* @Submandarine
 * Resmi Arjun
 * Omar Benjelloun (Google)
 * Alice Cheng (Nvidia)
@@ -62,7 +68,7 @@ listed here in alphabetical order:
 * Arun Tejusve Raghunath Rajan (Cruise)
 * Vijay Janapa Reddi (Harvard University)
 * Andrew Reusch (OctoML)
-* Anandhu S (Kerala Technical University)
+* Anandhu Sooraj (Kerala Technical University)
 * Sergey Serebryakov (HPE)
 * Warren Schultz (Principled Technologies)
 * Amrutha Sheleenderan (Kerala Technical University)

README.md

@@ -11,40 +11,52 @@
 
 ### About
 
-[Collective Knowledge (CK)](https://cKnowledge.org) in an educational project
-to help researchers and engineers automate their repetitive, tedious and time-consuming tasks
-to build, run, benchmark and optimize AI, ML and other applications and systems 
-across diverse and continuously changing models, data, software and hardware.
+[Collective Knowledge (CK, CM, CM4MLOps, CM4MLPerf and CMX)](https://cKnowledge.org) 
+is an educational community project to learn how to run AI, ML and other emerging workloads 
+in the most efficient and cost-effective way across diverse models, data sets, software and hardware.
 
 CK consists of several sub-projects:
 
-* [Collective Mind framework (CM)](cm) - a very light-weight Python-based framework with minimal dependencies
-  to help users implement, share and reuse cross-platform automation recipes to 
-  build, benchmark and optimize applications on any platform
-  with any software and hardware. 
-
-  * [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference)
+* [Collective Mind framework (CM)](cm) - a very lightweight Python-based framework with minimal dependencies
+  intended to help researchers and engineers automate their repetitive, tedious and time-consuming tasks
+  to build, run, benchmark and optimize AI, ML and other applications and systems 
+  across diverse and continuously changing models, data, software and hardware.
 
   * [CM4MLOPS](https://github.com/mlcommons/cm4mlops) - 
     a collection of portable, extensible and technology-agnostic automation recipes
-    with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications 
+    with a common CLI and Python API (CM scripts) to unify and automate 
+    all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications 
     on diverse platforms with any software and hardware: see [online catalog at CK playground](https://access.cknowledge.org/playground/?action=scripts),
     [online MLCommons catalog](https://docs.mlcommons.org/cm4mlops/scripts) 
 
+  * [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference)
+
   * [CM4ABTF](https://github.com/mlcommons/cm4abtf) - a unified CM interface and automation recipes
     to run automotive benchmark across different models, data sets, software and hardware from different vendors.
 
-* [CMX (the next generation of CM)](cm/docs/cmx) - we are developing the next generation of CM 
-  to make it simpler and more flexible based on user feedback. Please follow 
-  this project [here](https://github.com/orgs/mlcommons/projects/46).
+* [CMX (the next generation of CM, CM4MLOps and CM4MLPerf)](cm/docs/cmx) - 
+    we are developing the next generation of CM 
+    to make it simpler and more flexible based on user feedback. Please follow 
+    this project [here]( https://github.com/orgs/mlcommons/projects/46 ).
+
 
 * [Collective Knowledge Playground](https://access.cKnowledge.org) - a unified platform
   to list CM scripts similar to PYPI, aggregate AI/ML Systems benchmarking results in a reproducible format with CM workflows, 
   and organize [public optimization challenges and reproducibility initiatives](https://access.cknowledge.org/playground/?action=challenges) 
   to co-design more efficient and cost-effiective software and hardware for emerging workloads.
 
+  * [CM4MLPerf-results](https://github.com/mlcommons/cm4mlperf-results) - 
+    a simplified and unified representation of the past MLPerf results 
+    for further visualization and analysis using [CK graphs](https://access.cknowledge.org/playground/?action=experiments)
+    (*the new version is coming soon*).
+
+
 * [Artifact Evaluation](https://cTuning.org/ae) - automating artifact evaluation and reproducibility initiatives at ML and systems conferences.
 
+### Deprecated and archived projects 
+
+* [CM-MLOps](cm-mlops) - now [CM4MLOps](cm4mlops)
+* [CK automation framework v1 and v2](ck) - now [CM](cm)
 
 
 ### License
@@ -56,31 +68,51 @@ CK consists of several sub-projects:
 * Copyright (c) 2021-2024 MLCommons
 * Copyright (c) 2014-2021 cTuning foundation
 
-### Motivation and long-term vision
+### Author
 
-You can learn more about the motivation behind these projects from the following articles and presentations:
+* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning)
 
-* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] 
+### Maintainers
+
+* [Collective Mind (CM)](cm): [Grigori Fursin](https://cKnowledge.org/gfursin)
+* CM4MLOps (CM automation recipes): [Arjun Suresh](https://github.com/arjunsuresh) and [Anandhu Sooraj](https://github.com/anandhu-eng)
+* CMX (the next generation of CM, CM4MLOps and CM4MLPerf): [Grigori Fursin](https://cKnowledge.org/gfursin)
+
+### Citing our project
+
+If you found the CM automation framework helpful, kindly reference this article:
+[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ].
+
+To learn more about the motivation behind CK and CM technology, please explore the following presentations:
+
+* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ]
 * ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] 
-* ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]
+* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]
+* Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ]
+
 
 ### CM Documentation
 
 * [CM installation GUI](https://access.cknowledge.org/playground/?action=install)
-* [CM Getting Started Guide and FAQ](docs/getting-started.md)
-  * [Common CM interface to run MLPerf inference benchmarks](docs/mlperf/inference)
-  * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](docs/tutorials/common-interface-to-reproduce-research-projects.md)
+* [CM Getting Started Guide and FAQ](https://github.com/mlcommons/ck/tree/master/docs/getting-started.md)
+  * [Common CM interface to run MLPerf inference benchmarks](https://github.com/mlcommons/ck/tree/master/docs/mlperf/inference)
+  * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](https://github.com/mlcommons/ck/tree/master/docs/tutorials/common-interface-to-reproduce-research-projects.md)
   * [CM automation recipes for MLOps and DevOps](https://access.cknowledge.org/playground/?action=scripts)
-  * [Other CM tutorials](docs/tutorials)
-* [Full documentation](docs/README.md)
-* [CM development tasks](docs/taskforce.md#current-tasks)
-* [CM and CK history](docs/history.md)
+  * [Other CM tutorials](https://github.com/mlcommons/ck/tree/master/docs/tutorials)
+* [Full documentation](https://github.com/mlcommons/ck/tree/master/docs/README.md)
+* [CM taskforce](https://github.com/mlcommons/ck/tree/master/docs/taskforce.md)
+* [CMX, CM and CK history](https://github.com/mlcommons/ck/tree/master/docs/history.md)
 
 
 ### Acknowledgments
 
-Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://cKnowledge.org/gfursin),
-sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. 
-Since then, this open-source technology (CM, CM4MLOps, CM4MLPerf, CM4ABTF, CM4Research, etc)
-is being developed as a community effort thanks to all our
-[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! 
+The open-source Collective Knowledge project (CK, CM, CM4MLOps/CM4MLPerf, 
+CM4Research and CMX) was created by [Grigori Fursin](https://cKnowledge.org/gfursin)
+and sponsored by cTuning.org, OctoAI and HiPEAC.
+Grigori donated CK to MLCommons to benefit the community
+and to advance its development as a collaborative, community-driven effort.
+
+We thank [MLCommons](https://mlcommons.org), [FlexAI](https://flex.ai) 
+and [cTuning](https://cTuning.org) for supporting this project,
+as well as our dedicated [volunteers and collaborators](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)
+for their feedback and contributions!

ck/README.md

@@ -8,6 +8,9 @@
   behind CK in this [ACM TechTalk](https://www.youtube.com/watch?v=7zpeIVwICa4) 
   and the [journal article](https://doi.org/10.1098/rsta.2020.0211).**
 
+We keep this directory for backwards compatibility to let users reproduce past projects relying on this repository.
+
+
 <br>
 <br>
 <br>

cm-mlops/README.md

@@ -1,3 +1,5 @@
 **This repository is archived. In April 2024, we have separated this CM repo from the CM framework 
   and moved it to a [standalone mlcommons@cm4mlops repository](https://github.com/mlcommons/cm4mlops/tree/dev)
   following the suggesion from our users and MLCommons members.**
+
+We keep this directory for backwards compatibility to let users reproduce past projects relying on this repository.

cm/CHANGES.md

@@ -1,3 +1,54 @@
+## V3.4.4.1
+   - added tabulate as requirement
+
+## V3.4.4
+   - improved error reporting in utils.load_json and utils.load_yaml
+   - added utils.substitute_template
+   - enhanced 'cm pull repo' based on user feedback
+
+## V3.4.3
+   - fixed pyproject.toml
+   - fixed docs generation
+   - improved  cm pull repo with branch and SHA (#1355)
+
+## V3.4.2
+   - added utils.flatten_dict
+   - added utils.safe_int
+   - added utils.safe_float
+   - added utils.get_set
+   - added utils.digits
+
+## V3.4.1
+   - reduced Python min version in pyproject.toml to 3.7 for backwards compatibility
+
+## V3.4.0
+   - added pyproject.toml to install CM and deprecated setup.py
+
+## V3.3.4
+   - added utils.path2: 
+     add quotes if spaces in path
+   - added utils.update_dict_with_flat_key: 
+     update dictionary via flat key (x.y.z)
+   - added utils.get_value_from_dict_with_flat_key
+     get value from dict via flat key (x.y.z)
+   - added utils.load_module
+     universal python module loader
+
+## V3.3.3
+   - fixed CM logger issue
+
+## V3.3.2
+   - moved "CMX stack error" to -log
+   - improved error reporting
+
+## V3.3.1
+   - allow "cm/cmx pull repo {URL}" along with "cm/cmx pull repo --url={URL}" 
+   - CMX: added "automation_full_path" to -log
+
+## V3.2.9
+   - fixed minor bug with JSON console output
+   - fixed minor bug with console in cmind.x function
+
 ## V3.2.8
    - fixed Docker test for mlperf-inference with cm4mlops
    - minor improvements

cm/MANIFEST.in

@@ -0,0 +1,3 @@
+include CHANGES.md
+include CONTRIBUTING.md
+include COPYRIGHT.txt

cm/README.md

@@ -9,18 +9,20 @@
 
 ### About
 
-Collective Mind (CM) is a small, modular, cross-platform and decentralized workflow automation framework 
-with a human-friendly interface to make it easier to build, run, benchmark and optimize applications 
-across diverse models, data sets, software and hardware.
+Collective Mind (CM) is a small [Python package](https://pypi.org/project/cmind)
+with a unified CLI and API designed for creating and managing 
+portable and technology-agnostic automations for MLOps, DevOps and ResearchOps.
 
+It is intended to make it easier to build, run, benchmark and optimize applications 
+across diverse models, data sets, software and hardware.
+
 CM is a part of [Collective Knowledge (CK)](https://github.com/mlcommons/ck) - 
-an educational community project to learn how to run emerging workloads 
+an educational community project to learn how to run AI, ML and other emerging workloads 
 in the most efficient and cost-effective way across diverse 
-and continuously changing systems.
+and continuously changing systems using the MLPerf benchmarking methodology.
 
-CM includes a collection of portable, extensible and technology-agnostic automation recipes
-with a common API and CLI (aka CM scripts) to unify and automate different steps 
-required to compose, run, benchmark and optimize complex ML/AI applications 
+CM includes a [collection of portable, extensible and technology-agnostic automation recipes](https://access.cknowledge.org/playground/?action=scripts)
+(aka CM scripts) to unify and automate different steps required to compose, run, benchmark and optimize complex ML/AI applications 
 on any platform with any software and hardware. 
 
 CM scripts extend the concept of `cmake` with simple Python automations, native scripts
@@ -44,12 +46,16 @@ from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors:
   and simple JSON/YAML descriptions instead of inventing new workflow languages;
 * must have the same interface to run all automations natively, in a cloud or inside containers.
 
+### Author and maintainer
+
+* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning)
+
 ### Resources
 
-* CM v2.x (stable version 2022-cur): [installation on Linux, Windows, MacOS](https://access.cknowledge.org/playground/?action=install) ; 
+* CM v2.x (2022-cur) (stable): [installation on Linux, Windows, MacOS](https://access.cknowledge.org/playground/?action=install) ; 
   [docs](https://docs.mlcommons.org/ck) ; [popular commands](https://github.com/mlcommons/ck/tree/master/cm/docs/demos/some-cm-commands.md) ; 
   [getting started guide](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md)
-* CM v3.x (prototype 2024-cur): [docs](https://github.com/mlcommons/ck/tree/master/cm/docs/cmx)
+* CM v3.x aka CMX (2024-cur) (stable): [docs](https://github.com/orgs/mlcommons/projects/46)
 * MLPerf inference benchmark automated via CM
   * [Run MLPerf for submissions](https://docs.mlcommons.org/inference)
   * [Run MLPerf at the Student Cluster Competition'24](https://docs.mlcommons.org/inference/benchmarks/text_to_image/reproducibility/scc24)
@@ -61,21 +67,26 @@ from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors:
 
 [Apache 2.0](LICENSE.md)
 
-### Citing CM and CM4MLOps
+### Citing CM project
 
-If you found CM useful, please cite this article: 
+If you found CM automations useful, please cite this article: 
 [ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ].
 
-You can learn more about the motivation behind these projects from the following articles and presentations:
+You can learn more about the motivation behind these projects from the following presentations:
 
-* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] 
+* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ]
 * ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] 
 * ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]
 
 ### Acknowledgments
 
-The Collective Mind framework (CM) was created by [Grigori Fursin](https://cKnowledge.org/gfursin),
-sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. 
-Since then, this open-source technology (CM, CM4MLOps, CM4MLPerf, CM4ABTF, CM4Research, etc)
-is being developed as a community effort thanks to all our
-[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! 
+The Collective Mind (CM) automation framework was originally 
+developed by [Grigori Fursin](https://cKnowledge.org/gfursin), 
+as a part of the [Collective Knowledge educational initiative](https://cKnowledge.org),
+sponsored by [cTuning.org](https://cTuning.org) and [cKnowledge.org](https://cKnowledge.org), 
+and contributed to MLCommons for the benefit of all. 
+
+This open-source technology, including CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, and more, 
+is a collaborative project supported by [MLCommons](https://mlcommons.org), 
+[FlexAI](https://flex.ai), [cTuning](https://cTuning.org)
+and our [amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! 

cm/cmind/__init__.py

@@ -2,7 +2,7 @@
 #
 # Written by Grigori Fursin
 
-__version__ = "3.2.8"
+__version__ = "3.4.4"
 
 from cmind.core import access
 from cmind.core import x

cm/cmind/cli.py

@@ -193,21 +193,6 @@ def run_experiment(argv = None):
 
     return run(['run', 'experiment'] + argv)
 
-############################################################
-def run_ff(argv = None):
-    """
-    """
-
-    # Access CMX
-    if argv is None:
-        argv = sys.argv[1:]
-
-    x = 'lex'
-    if len(argv) >0 and argv[0].startswith('_misc'):
-        x = argv[0]
-
-    return runx(['run', 'f'+x+'.flow'] + argv)
-
 ############################################################
 def parse(cmd):
     """

cm/cmind/config.py

@@ -37,7 +37,7 @@ def __init__(self, config_file = None):
                      "flag_help2": "help",
 
                      "error_prefix": "CM error:",
-                     "error_prefix2": "CMX detected an issue",
+                     "error_prefix2": "CMX detected a problem",
                      "info_cli": "cm {action} {automation} {artifact(s)} {flags} @input.yaml @input.json",
                      "info_clix": "cmx {action} {automation} {artifact(s)} {CMX control flags (-)} {CMX automation flags (--)}",