diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e012953e8a..277e33d512 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,11 +14,11 @@ Modify the project in your own fork and issue a pull request once you want other to take a look at what you have done and discuss the proposed changes. Ensure that cla-bot and other checks pass for your Pull requests. -Collective Knowledge (CK) and Collective Mind (CM) +Collective Knowledge (CK), Collective Mind (CM) and Common Metadata eXchange (CMX) were created by [Grigori Fursin](https://arxiv.org/abs/2406.16791), sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. Since then, this open-source automation technology -(CM/CMX, CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, etc) is being extended +(CM/CMX, CM4MLOps/MLPerf automations, CM4ABTF, CM4Research, etc) is being extended as a community effort thanks to all our volunteers, collaborators and contributors listed here in alphabetical order: diff --git a/COPYRIGHT.txt b/COPYRIGHT.txt index 7b62e067df..9c84f3fbe1 100644 --- a/COPYRIGHT.txt +++ b/COPYRIGHT.txt @@ -1,5 +1,5 @@ -Copyright (c) 2021-2024 MLCommons +© 2021-2025 MLCommons. All Rights Reserved. -Grigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone. +Grigori Fursin, the cTuning foundation and OctoML donated the CK and CM projects to MLCommons to benefit everyone and continue development as a community effort. Copyright (c) 2014-2021 cTuning foundation diff --git a/HISTORY.CM.md b/HISTORY.CM.md index e14f8b8356..ce9a2e43ed 100644 --- a/HISTORY.CM.md +++ b/HISTORY.CM.md @@ -23,6 +23,12 @@ aimed at benefiting the broader community and fostering its ongoing development [ref4](https://github.com/mlcommons/ck/tree/master/ck), [ref5](https://github.com/mlcommons/ck-mlops). + Prepared MLCommons proposal for the creation of the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md), + aimed at fostering community-driven support for CK and CM developments to benefit everyone. + +* Jan, 2022: hired Arjun Suresh at OctoML to support and maintain CK1 framework and help prepare OctoML's MLPerf submissions using CK1. + Meanwhile, transitioned to focusing on CM and CM-MLOps development, building upon the prototypes created in 2021. + * Mar 1, 2022: started developing cm-mlops: [ref](https://github.com/octoml/cm-mlops/commit/0ae94736a420dfa84f7417fc62d323303b8760c6). * Mar 24, 2022: after successfully stabilizing the initial prototype of CM, donated it to MLCommons to benefit the entire community: @@ -55,7 +61,7 @@ aimed at benefiting the broader community and fostering its ongoing development [ref3](https://github.com/octoml/cm-mlops/blob/7692240becd6397a96c3975388913ea082002e7a/CONTRIBUTIONS). * May 11, 2022: After successfully prototyping CM and CM-MLOps, deprecated the CK1 framework in favor of CM. - Welcomed Arjun as a maintainer and tester for CM and CM-MLOps: + Transferred Arjun Suresh to the CM project as a maintainer and tester for CM and CM-MLOps: [ref](https://github.com/octoml/cm-mlops/blob/17405833665bc1e93820f9ff76deb28a0f543bdb/CONTRIBUTIONS). Created a [file](https://github.com/mlcommons/ck/blob/master/cm-mlops/CHANGES.md) @@ -75,17 +81,47 @@ aimed at benefiting the broader community and fostering its ongoing development [ref1](https://github.com/mlcommons/ck/commit/2c5d5c5c944ae5f252113c62af457c7a4c5e877a#diff-faac2c4ecfd0bfb928dafc938d3dad5651762fbb504a2544752a337294ee2573R224), [ref2](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#author-and-coordinator). - Welcomed Arjun Suresh as a contributor to CM: [ref](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#contributors-in-alphabetical-order). + Welcomed Arjun Suresh as a contributor to CM automations for MLPerf: [ref](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#contributors-in-alphabetical-order). * From September 2022: coordinated community development of CM and CM4MLOps - to [modularize and automate MLPerf](https://docs.mlcommons.org/inference) + to [modularize and automate MLPerf benchmarks](https://docs.mlcommons.org/inference) and support [reproducibility initiatives at ML and Systems conferences](https://cTuning.or/ae) - through the MLCommons Task Force on Automation and Reproducibility. + through the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md). + + * Directed and financed the creation of (CM) automations to streamline the MLPerf power measurement processes. + + * Proposed to use MLPerf benchmarks for the Student Cluster Competition, led the developments + and prepared a tutorial to run MLPerf inference at SCC'22 via CM: [ref](https://github.com/mlcommons/ck/blob/master/docs/tutorials/sc22-scc-mlperf.md) + +* April 2023: departed OctoML to focus on the development of the [CK playground](https://access.cKnowledge.org) and CM automations + to make Mlperf accessible to everyone. Hired Arjun Suresh to help with developments. + + * Initiated and funded development of the [MLPerf explorer](https://github.com/ctuning/q2a-mlperf-visualizer) + to improve visualization of results + +* August 2023: organized the 1st mass-scale MLPerf community submission of 12217 inference benchmark v3.1 results + out of total 13351 results (including 90% of all power results) across diverse models, software and hardware + from different vendors via [open challenges](https://access.cknowledge.org/playground/?action=challenges) funded by cTuning.org : + [LinkedIn article](https://www.linkedin.com/pulse/new-milestone-make-mlperf-benchmarks-accessible-everyone-fursin/) + with results visualized by the [MLPerf explorer](https://github.com/ctuning/q2a-mlperf-visualizer), + [CM4MLOps challenges at GitHub](https://github.com/mlcommons/cm4mlops/tree/main/challenge). + +* February, 2024: proposed to use CM to automate [MLPerf automotive benchmark (ABTF)](https://mlcommons.org/working-groups/benchmarks/automotive/). + + * moved my prototypes of the CM automation for ABTF to cm4abtf repo: [ref](https://github.com/mlcommons/cm4abtf/commit/f92b9f464de89a38a4bde149290dede2d94c8631) + * led further CM4ABTF developments funded by cTuning.org. * Starting in April 2024, began the gradual transfer of ongoing maintenance and enhancement responsibilities for CM and CM4MLOps, including MLPerf automations, to MLCommons. Welcomed Anandhu Sooraj as a maintainer and contributor to CM4MLOps with MLPerf automations. -For more details, please refer to this [white paper](https://arxiv.org/abs/2406.16791) -and [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339). +* Took a break from all development activities. + +* July 2024: started prototyping the next generation of CM (CMX and CMX4MLOps) with simpler interfaces + based on user feedback while maintaining backward compatibility. + +* 2025: continue developing CMX and CMX4MLOPs to make it easier to run and customize MLPerf inference, training + and other benchmarks across diverse models, datasets, software and hardware. +For more details, please refer to the [white paper](https://arxiv.org/abs/2406.16791) +and the [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339). diff --git a/README.md b/README.md index 164faed253..69fa3296ec 100755 --- a/README.md +++ b/README.md @@ -18,7 +18,14 @@ in the most efficient and cost-effective way across diverse models, data sets, s It includes the following sub-projects. -### Collective Mind (CM) +### Collective Minds (CM) + +The Collective Mind (CM) project, or Collective Minds, facilitates the +decomposition of complex software systems into portable, reusable, and +interconnected automation recipes. These recipes are developed and +continuously improved by the community. + +#### CM automation framework The [Collective Mind framework](https://github.com/mlcommons/ck/tree/master/cm) is a lightweight, Python-based toolset featuring @@ -30,12 +37,21 @@ and other applications across diverse and continuously changing models, data, so Collective Mind is continuously enhanced through public and private Git repositories with CM automation recipes and artifacts accessible via unified CM interface. -The CM architecture diagram is available for viewing +#### CMX automation framework + +[CMX](https://github.com/mlcommons/ck/tree/master/cmx) is the next evolution +of the Collective Mind framework designed to enhance simplicity, flexibility, and extensibility of automations +based on user feedback. It is backwards compatible with CM, released along with CM +in the [cmind package](https://pypi.org/project/cmind/) and can serve as drop-in replacement for CM. + +The CM/CMX architecture diagram is available for viewing [here](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png). -### Notable Collective Mind repositories -#### CM4MLOps + +### Notable CM use cases + +#### MLOps and MLPerf automations [CM4MLOPS repository powered by CM](https://github.com/mlcommons/ck/tree/master/cm-mlops) - a collection of portable, extensible and technology-agnostic automation recipes @@ -57,27 +73,19 @@ while keeping backward compatibility. See the [online documentation](https://docs.mlcommons.org/inference) at MLCommons to run MLPerf inference benchmarks across diverse systems using CM. -#### CM4ABTF +#### MLCommons ABTF automation [CM4ABTF repository powered by CM](https://github.com/mlcommons/cm4abtf) - a collection of portable automations and CM scripts to run the upcoming automotive MLPerf benchmark across different models, data sets, software and hardware from different vendors. -#### CM4MLPerf-results +#### MLPerf results visualization [CM4MLPerf-results powered by CM](https://github.com/mlcommons/cm4mlperf-results) - a simplified and unified representation of the past MLPerf results in the CM format for further visualization and analysis using [CK graphs](https://access.cknowledge.org/playground/?action=experiments). -#### CM4Research - -[CM4Research repository powered by CM](https://github.com/ctuning/cm4research) - -a unified interface designed to streamline the preparation, execution, and reproduction of experiments in research projects. - - -### Projects powered by Collective Mind - #### Collective Knowledge Playground [Collective Knowledge Playground](https://access.cKnowledge.org) - @@ -97,18 +105,23 @@ collaboratively enhance the efficiency and cost-effectiveness of AI systems. leveraging the Collective Mind framework to automate artifact evaluation and support reproducibility efforts at ML and systems conferences. +* [CM4Research repository powered by CM](https://github.com/ctuning/cm4research) - +a unified interface designed to streamline the preparation, execution, and reproduction of experiments in research projects. + -## Incubator +## Legacy projects -[CMX](https://github.com/mlcommons/ck/tree/master/cmx) - the next evolution of the Collective Mind framework, -designed to enhance simplicity, flexibility, and extensibility of automations -based on user feedback. Follow the project's progress [here]( https://github.com/orgs/mlcommons/projects/46 ). +### CM-MLOps (now CM4MLOps) +You can find CM-MLOps original dev directory [here](https://github.com/mlcommons/ck/tree/master/cm-mlops). +We moved it to [CM4MLOps](https://github.com/mlcommons/ck/tree/master/cm4mlops) in 2024. +In 2025, we aggregate all CM and CMX automations in the [new CMX4MLOps repository](https://github.com/mlcommons/ck/tree/master/cmx4mlops). -## Archived projects +### CK automation framework v1 and v2 -* [CM-MLOps](https://github.com/mlcommons/ck/tree/master/cm-mlops) - now [CM4MLOps](https://github.com/mlcommons/ck/tree/master/cm4mlops) -* [CK automation framework v1 and v2](https://github.com/mlcommons/ck/tree/master/ck) - now [CM](https://github.com/mlcommons/ck/tree/master/cm) +You can find the original CK automation framework v1 and v2 directory [here](https://github.com/mlcommons/ck/tree/master/ck). +It was deprecated for the [CM framework](https://github.com/mlcommons/ck/tree/master/cm) +and later for the [CMX workflow automation framework (backwards compatible with CM)](https://github.com/mlcommons/ck/tree/master/cmx) ## License @@ -117,17 +130,17 @@ based on user feedback. Follow the project's progress [here]( https://github.com ## Copyright -* Copyright (c) 2021-2024 MLCommons -* Copyright (c) 2014-2021 cTuning foundation +Copyright (c) 2021-2024 MLCommons + +Grigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone. + +Copyright (c) 2014-2021 cTuning foundation ## Author * [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning) -## Citing Collective Mind and Collective Knowledge - -If you found the CM automation framework helpful, kindly reference this article: -[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. +## Long-term vision To learn more about the motivation behind CK and CM technology, please explore the following presentations: @@ -136,10 +149,9 @@ To learn more about the motivation behind CK and CM technology, please explore t * ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] * Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ] +## Documentation -## CM Documentation - -* [Collective Mind white paper](https://arxiv.org/abs/2406.16791) +* [White paper](https://arxiv.org/abs/2406.16791) * [CM/CMX architecture](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png) * [CM/CMX installation GUI](https://access.cknowledge.org/playground/?action=install) * [CM Getting Started Guide and FAQ](https://github.com/mlcommons/ck/tree/master/docs/getting-started.md) @@ -149,13 +161,12 @@ To learn more about the motivation behind CK and CM technology, please explore t * [Other CM tutorials](https://github.com/mlcommons/ck/tree/master/docs/tutorials) * [Full documentation](https://github.com/mlcommons/ck/tree/master/docs/README.md) * [CM taskforce](https://github.com/mlcommons/ck/tree/master/docs/taskforce.md) -* [CK, CM and CMX history](https://github.com/mlcommons/ck/tree/master/docs/history.md) +* History: [CK](https://github.com/mlcommons/ck/tree/master/docs/history.md), [CM and CM automations for MLOps and MLPerf](https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md) ### Acknowledgments -The open-source Collective Knowledge project (CK, CM, CM4MLOps/CM4MLPerf, -CM4Research and CMX) was created by [Grigori Fursin](https://cKnowledge.org/gfursin) +This open-source project was created by [Grigori Fursin](https://cKnowledge.org/gfursin) and sponsored by cTuning.org, OctoAI and HiPEAC. Grigori donated CK to MLCommons to benefit the community and to advance its development as a collaborative, community-driven effort. @@ -164,3 +175,6 @@ We thank [MLCommons](https://mlcommons.org), [FlexAI](https://flex.ai) and [cTuning](https://cTuning.org) for supporting this project, as well as our dedicated [volunteers and collaborators](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) for their feedback and contributions! + +If you found the CM automations helpful, kindly reference this article: +[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. diff --git a/cm-mlops/COPYRIGHT.md b/cm-mlops/COPYRIGHT.md new file mode 100644 index 0000000000..5aa453ab8c --- /dev/null +++ b/cm-mlops/COPYRIGHT.md @@ -0,0 +1,5 @@ +# Copyright Notice + +© 2022-2025 MLCommons. All Rights Reserved. + +Grigori Fursin, the cTuning foundation and OctoML donated the CK and CM projects to MLCommons to benefit everyone and continue development as a community effort. diff --git a/cm-mlops/COPYRIGHT.txt b/cm-mlops/COPYRIGHT.txt deleted file mode 100644 index d4e3029cae..0000000000 --- a/cm-mlops/COPYRIGHT.txt +++ /dev/null @@ -1,5 +0,0 @@ -Copyright (c) 2022-2023 MLCommons - -Grigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone. - -Since 2024, we use a separate Github repository for further developments: https://github.com/mlcommons/cm4mlops diff --git a/cm/CHANGES.md b/cm/CHANGES.md index ada7097f8a..a7a6ec8c93 100644 --- a/cm/CHANGES.md +++ b/cm/CHANGES.md @@ -1,3 +1,6 @@ +## V3.5.3 + - fixed `cm info repo` and `cmx info repo`: https://github.com/mlcommons/ck/issues/1378 + ## V3.5.2 - added `-i` flag to print memory and disk use before running CM/CMX commands: https://github.com/mlcommons/ck/issues/1375 diff --git a/cm/COPYRIGHT.txt b/cm/COPYRIGHT.txt index f69429f9ea..35a95f3eba 100644 --- a/cm/COPYRIGHT.txt +++ b/cm/COPYRIGHT.txt @@ -1,4 +1,4 @@ -Copyright (c) 2022-2024 MLCommons +Copyright (c) 2022-2025 MLCommons Grigori Fursin, the cTuning foundation and OctoML donated CM and CM4MLOps to MLCommons to benefit everyone. diff --git a/cm/HISTORY.md b/cm/HISTORY.md deleted file mode 100644 index e14f8b8356..0000000000 --- a/cm/HISTORY.md +++ /dev/null @@ -1,91 +0,0 @@ -This document narrates the history of the creation and design of CM and CM4MLOps (also known as CK2) -by [Grigori Fursin](https://cKnowledge.org/gfursin). It also highlights the donation of this open-source technology to MLCommons, -aimed at benefiting the broader community and fostering its ongoing development as a collaborative, community-driven initiative: - -* Jan 28, 2021: After delivering an invited ACM TechTalk'21 about the Collective Knowledge framework (CK1) - and reproducibility initiatives for conferences, as well as CK-MLOps and MLPerf automations, - Grigori received lots of positive feedback and suggestions for improvements to workflow automations: - https://learning.acm.org/techtalks/reproducibility. - - Following this, Grigori began prototyping CK2 (later CM) to streamline CK1, CK-MLOps and MLPerf benchmarking. - The goal was to dramatically simplify CK1 workflows by introducing just a few core and portable automations, - which eventually evolved into `CM script` and `CM cache`. - - At that time, the cTuning foundation hosted CK1 and all the prototypes for the CM framework at https://github.com/ctuning/ck: - [ref1](https://github.com/mlcommons/ck/commit/9e57934f4999db23052531e92160772ab831463a), - [ref2](https://github.com/mlcommons/ck/tree/9e57934f4999db23052531e92160772ab831463a), - [ref3](https://github.com/mlcommons/ck/tree/9e57934f4999db23052531e92160772ab831463a/incubator). - -* Sep 23, 2021: donated CK1, CK-MLOps, MLPerf automations and early prototypes of CM from the cTuning repository to MLCommons: - [ref1](https://web.archive.org/web/20240803140223/https://octo.ai/blog/octoml-joins-the-community-effort-to-democratize-mlperf-inference-benchmarking), - [ref2](https://github.com/mlcommons/ck/tree/228f80b0bf44610c8244ff0c3f6bec5bbd25aa6c/incubator), - [ref3](https://github.com/mlcommons/ck/tree/695c3843fd8121bbdde6c453cd6ec9503986b0c6?tab=readme-ov-file#author-and-coordinator), - [ref4](https://github.com/mlcommons/ck/tree/master/ck), - [ref5](https://github.com/mlcommons/ck-mlops). - -* Mar 1, 2022: started developing cm-mlops: [ref](https://github.com/octoml/cm-mlops/commit/0ae94736a420dfa84f7417fc62d323303b8760c6). - -* Mar 24, 2022: after successfully stabilizing the initial prototype of CM, donated it to MLCommons to benefit the entire community: - [ref1](https://github.com/mlcommons/ck/tree/c7918ad544f26b6c499c2fc9c07431a9640fca5a/ck2), - [ref2](https://github.com/mlcommons/ck/tree/c7918ad544f26b6c499c2fc9c07431a9640fca5a/ck2#coordinators), - [ref3](https://github.com/mlcommons/ck/commit/3c146cb3c75a015363f7a96758adf6dcc43032d6), - [ref4](https://github.com/mlcommons/ck/commit/3c146cb3c75a015363f7a96758adf6dcc43032d6#diff-d97f0f6f5a32f16d6ed18b9600ffc650f7b25512685f7a2373436c492c6b52b3R48). - -* Apr 6, 2022: started transitioning previous MLOps and MLPerf automations from the mlcommons/ck-mlops format - to the new CM format using the cm-mlops repository (will be later renamed to cm4mlops): - [ref1](https://github.com/octoml/cm-mlops/commit/d1efdc30fb535ce144020d4e88f3ed768c933176), - [ref2](https://github.com/octoml/cm-mlops/blob/d1efdc30fb535ce144020d4e88f3ed768c933176/CONTRIBUTIONS). - -* Apr 22, 2022: began architecting "Intelligent Components" in the CM-MLOps repository, - which will be renamed to `CM Script` at a later stage: - [ref1](https://github.com/octoml/cm-mlops/commit/b335c609c47d2c547afe174d9df232652d57f4f8), - [ref2](https://github.com/octoml/cm-mlops/tree/b335c609c47d2c547afe174d9df232652d57f4f8), - [ref3](https://github.com/octoml/cm-mlops/blob/b335c609c47d2c547afe174d9df232652d57f4f8/CONTRIBUTIONS). - - At the same time, prototyped other core CM automations, including IC, Docker, and Experiment: - [ref1](https://github.com/octoml/cm-mlops/tree/b335c609c47d2c547afe174d9df232652d57f4f8/automation), - [ref2](https://github.com/mlcommons/ck/commits/master/?before=7f66e2438bfe21b4ce2d08326a5168bb9e3132f6+7001). - -* Apr 28, 2022: donated CM-MLOps to MLCommons, which was later renamed to CM4MLOps: - [ref](https://github.com/mlcommons/ck/commit/456e4861056c0e39c4d689c03da91f90a44be058). - -* May 9, 2022: developed the initial set of core IC automations for MLOps (aka CM scripts): - [ref1](https://github.com/octoml/cm-mlops/commit/4a4a027f4088ce7e7abcec29c39d98981bf09d4c), - [ref2](https://github.com/octoml/cm-mlops/tree/4a4a027f4088ce7e7abcec29c39d98981bf09d4c), - [ref3](https://github.com/octoml/cm-mlops/blob/7692240becd6397a96c3975388913ea082002e7a/CONTRIBUTIONS). - -* May 11, 2022: After successfully prototyping CM and CM-MLOps, deprecated the CK1 framework in favor of CM. - Welcomed Arjun as a maintainer and tester for CM and CM-MLOps: - [ref](https://github.com/octoml/cm-mlops/blob/17405833665bc1e93820f9ff76deb28a0f543bdb/CONTRIBUTIONS). - - Created a [file](https://github.com/mlcommons/ck/blob/master/cm-mlops/CHANGES.md) - to document and track our public developments at MLCommons. - -* Jun 8, 2022: renamed the 'IC' automation to the more intuitive 'CM script' automation. - [ref1](https://github.com/mlcommons/ck/tree/5ca4e2c33e58a660ac20a545d8aa5143ab6e8e81/cm-devops/automation/script), - [ref2](https://github.com/mlcommons/ck/tree/5ca4e2c33e58a660ac20a545d8aa5143ab6e8e81), - [ref3](https://github.com/octoml/cm-mlops/commit/7910fb7ffc62a617d987d2f887d6f9981ff80187). - -* Jun 16, 2022: prototyped the `CM cache` automation to facilitate caching and reuse of the outputs from CM scripts: - [ref1](https://github.com/mlcommons/ck/commit/1f81aae8cebd5567ec4ca55f693beaf32b49fb48), - [ref2](https://github.com/mlcommons/ck/tree/1f81aae8cebd5567ec4ca55f693beaf32b49fb48), - [ref3](https://github.com/mlcommons/ck/tree/1f81aae8cebd5567ec4ca55f693beaf32b49fb48?tab=readme-ov-file#contacts). - -* Sep 6, 2022: delivered CM demo to run MLPerf while deprecating CK1 automations for MLPerf: - [ref1](https://github.com/mlcommons/ck/commit/2c5d5c5c944ae5f252113c62af457c7a4c5e877a#diff-faac2c4ecfd0bfb928dafc938d3dad5651762fbb504a2544752a337294ee2573R224), - [ref2](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#author-and-coordinator). - - Welcomed Arjun Suresh as a contributor to CM: [ref](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#contributors-in-alphabetical-order). - -* From September 2022: coordinated community development of CM and CM4MLOps - to [modularize and automate MLPerf](https://docs.mlcommons.org/inference) - and support [reproducibility initiatives at ML and Systems conferences](https://cTuning.or/ae) - through the MLCommons Task Force on Automation and Reproducibility. - -* Starting in April 2024, began the gradual transfer of ongoing maintenance and enhancement - responsibilities for CM and CM4MLOps, including MLPerf automations, to MLCommons. - Welcomed Anandhu Sooraj as a maintainer and contributor to CM4MLOps with MLPerf automations. - -For more details, please refer to this [white paper](https://arxiv.org/abs/2406.16791) -and [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339). - diff --git a/cm/README.md b/cm/README.md index 569127634f..5cfb4b8aca 100644 --- a/cm/README.md +++ b/cm/README.md @@ -22,22 +22,34 @@ an educational community project to learn how to run AI, ML and other emerging w in the most efficient and cost-effective way across diverse and ever-evolving systems using the MLPerf benchmarking methodology. -## Collective Mind architecture +#### CMX automation framework + +[CMX](https://github.com/mlcommons/ck/tree/master/cmx) - the next evolution +of the [Collective Mind framework](https://github.com/mlcommons/ck/tree/master/cm) +designed to enhance simplicity, flexibility, and extensibility of automations +based on user feedback. It is backwards compatible with CM, released along with CM +in the [cmind package](https://pypi.org/project/cmind/) and can serve as drop-in replacement for CM. + +The CM/CMX architecture diagram is available for viewing +[here](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png). + + +## CM and CMX architecture The diagram below illustrates the primary classes, functions, and internal automations within the Collective Mind framework: -![](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png) +![](https://cKnowledge.org/images/cm-diagram-v3.5.1.png) The CM API documentation is available [here](https://cknowledge.org/docs/cm/api/cmind.html). -## Collective Mind repositories +## Projects powered by Collective Mind -Collective Mind is continuously enhanced through public and private CM4* Git repositories, +Collective Mind is continuously enhanced through public and private Git repositories, which serve as the unified interface for various collections of reusable automations and artifacts. The most notable projects and repositories powered by CM are: -#### CM4MLOps +#### Automations for MLOps and MLPerf [CM4MLOPS repository powered by CM](https://github.com/mlcommons/cm4mlops) - a collection of portable, extensible and technology-agnostic automation recipes @@ -173,11 +185,13 @@ from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors: [Apache 2.0](LICENSE.md) -### Citing Collective Mind +### Citing Collective Mind project -If you found CM automations, please cite this article: +If you found CM automations help, please cite this article: [ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. +History: [CM and CM automations for MLOps and MLPerf](https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md). + You can learn more about the motivation behind these projects from the following presentations: * "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] @@ -191,7 +205,7 @@ as a part of the [Collective Knowledge educational initiative](https://cKnowledg sponsored by [cTuning.org](https://cTuning.org) and [cKnowledge.org](https://cKnowledge.org), and contributed to MLCommons for the benefit of all. -This open-source technology, including CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, and more, +This open-source technology, including CM4MLOps and MLPerf automations, CM4ABTF, CM4Research, and more, is a collaborative project supported by [MLCommons](https://mlcommons.org), [FlexAI](https://flex.ai), [cTuning](https://cTuning.org) and our [amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! diff --git a/cm/cmind/__init__.py b/cm/cmind/__init__.py index 619e652099..a90f5ed8e3 100644 --- a/cm/cmind/__init__.py +++ b/cm/cmind/__init__.py @@ -1,8 +1,15 @@ # Collective Mind init # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md -__version__ = "3.5.2" +__version__ = "3.5.3" from cmind.core import access from cmind.core import x diff --git a/cm/cmind/__main__.py b/cm/cmind/__main__.py index c0349fe282..973a4fac5b 100644 --- a/cm/cmind/__main__.py +++ b/cm/cmind/__main__.py @@ -1,6 +1,13 @@ # Collective Mind CLI entry point # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import cmind.cli as cli diff --git a/cm/cmind/artifact.py b/cm/cmind/artifact.py index f22a36c03b..9359b01286 100644 --- a/cm/cmind/artifact.py +++ b/cm/cmind/artifact.py @@ -1,6 +1,13 @@ # Collective Mind artifact # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os diff --git a/cm/cmind/automation.py b/cm/cmind/automation.py index b96b08449f..023c2d9f86 100644 --- a/cm/cmind/automation.py +++ b/cm/cmind/automation.py @@ -1,6 +1,13 @@ # Collective Mind automation # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os @@ -1195,6 +1202,8 @@ def info(self, i): cid1 = '' + + for artifact in lst: if console and len(lst)>1: print ('****************************************************') diff --git a/cm/cmind/cli.py b/cm/cmind/cli.py index 7e03d2347c..e925b797d4 100644 --- a/cm/cmind/cli.py +++ b/cm/cmind/cli.py @@ -1,6 +1,13 @@ # Collective Mind command line wrapper # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import sys diff --git a/cm/cmind/config.py b/cm/cmind/config.py index 4071601901..a67046c105 100644 --- a/cm/cmind/config.py +++ b/cm/cmind/config.py @@ -1,6 +1,13 @@ # Collective Mind configuration # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os diff --git a/cm/cmind/core.py b/cm/cmind/core.py index 1d4543b318..3f5e9cbcfa 100644 --- a/cm/cmind/core.py +++ b/cm/cmind/core.py @@ -1,6 +1,13 @@ # Collective Mind core functions # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md from cmind.config import Config from cmind.repos import Repos diff --git a/cm/cmind/index.py b/cm/cmind/index.py index 5207a998b1..7564872c84 100644 --- a/cm/cmind/index.py +++ b/cm/cmind/index.py @@ -1,6 +1,13 @@ # Collective Mind index # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os import json diff --git a/cm/cmind/net.py b/cm/cmind/net.py index 5cbc46368e..ee0caf191c 100644 --- a/cm/cmind/net.py +++ b/cm/cmind/net.py @@ -1,6 +1,13 @@ # Reused from the CK framework for compatibility # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import urllib import json diff --git a/cm/cmind/repo.py b/cm/cmind/repo.py index 9cf04c0370..cd70e90973 100644 --- a/cm/cmind/repo.py +++ b/cm/cmind/repo.py @@ -1,6 +1,13 @@ # Collective Mind repository # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os @@ -39,6 +46,7 @@ def __init__(self, path, cfg): # Repo meta self.meta = {} + self.original_meta = {} # for compatibility with CM/CMX universal search ############################################################ def load(self, cmx = False): @@ -70,6 +78,7 @@ def load(self, cmx = False): return r self.meta = r['meta'] + self.original_meta = self.meta if cmx and self.meta.get('prefix_cmx', '') != '': self.path_prefix = self.meta.get('prefix_cmx','') diff --git a/cm/cmind/repo/README.md b/cm/cmind/repo/README.md index 48e5cd22b8..684888d1de 100644 --- a/cm/cmind/repo/README.md +++ b/cm/cmind/repo/README.md @@ -6,3 +6,12 @@ This internal repository contains CM-compatible artifacts and automations that were moved to the CM core to ensure their stability because they are frequently used by the community. + +* Authors: Grigori Fursin +* Contributors: + +* Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +* License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +* History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +* White paper: https://arxiv.org/abs/2406.16791 +* Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md diff --git a/cm/cmind/repo/automation/automation/module.py b/cm/cmind/repo/automation/automation/module.py index d1551df310..29a2d97da1 100644 --- a/cm/cmind/repo/automation/automation/module.py +++ b/cm/cmind/repo/automation/automation/module.py @@ -1,6 +1,13 @@ # CM automation module to add or document all automations # -# Written by Grigori Fursin +# Authors: Grigori Fursin +# Contributors: +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os diff --git a/cm/cmind/repo/automation/automation/module_misc.py b/cm/cmind/repo/automation/automation/module_misc.py index 3ed1c7d1d7..00217e7979 100644 --- a/cm/cmind/repo/automation/automation/module_misc.py +++ b/cm/cmind/repo/automation/automation/module_misc.py @@ -1,6 +1,13 @@ # Documenting CM automations # -# Written by Grigori Fursin +# Authors: Grigori Fursin +# Contributors: +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os from cmind import utils diff --git a/cm/cmind/repo/automation/automation/modulex_dummy.py b/cm/cmind/repo/automation/automation/modulex_dummy.py index 6dc0028a00..7aace68e9e 100644 --- a/cm/cmind/repo/automation/automation/modulex_dummy.py +++ b/cm/cmind/repo/automation/automation/modulex_dummy.py @@ -1,5 +1,3 @@ -# Developer(s): Grigori Fursin - import os from cmind.automation import Automation diff --git a/cm/cmind/repo/automation/ckx/module.py b/cm/cmind/repo/automation/ckx/module.py index e1958da523..ac4ece2161 100644 --- a/cm/cmind/repo/automation/ckx/module.py +++ b/cm/cmind/repo/automation/ckx/module.py @@ -1,6 +1,13 @@ # CM wrapper for older CK framework in case it is still needed # -# Written by Grigori Fursin +# Authors: Grigori Fursin +# Contributors: +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os diff --git a/cm/cmind/repo/automation/core/cm_60cb625a46b38610/misc.py b/cm/cmind/repo/automation/core/cm_60cb625a46b38610/misc.py index 5bc87457b4..741abc7093 100644 --- a/cm/cmind/repo/automation/core/cm_60cb625a46b38610/misc.py +++ b/cm/cmind/repo/automation/core/cm_60cb625a46b38610/misc.py @@ -1,3 +1,14 @@ +# Demo of using sub-modules in CM/CMX +# +# Authors: Grigori Fursin +# Contributors: +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md + import cmind.utils as utils ############################################################ diff --git a/cm/cmind/repo/automation/core/module.py b/cm/cmind/repo/automation/core/module.py index 7aecf41002..fa774a5d11 100644 --- a/cm/cmind/repo/automation/core/module.py +++ b/cm/cmind/repo/automation/core/module.py @@ -1,6 +1,13 @@ # CM core automation with a few universal functions # -# Written by Grigori Fursin +# Authors: Grigori Fursin +# Contributors: +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os diff --git a/cm/cmind/repo/automation/core/module_misc.py b/cm/cmind/repo/automation/core/module_misc.py index 4bcb549cca..7a347d69f1 100644 --- a/cm/cmind/repo/automation/core/module_misc.py +++ b/cm/cmind/repo/automation/core/module_misc.py @@ -1,6 +1,13 @@ # Generate CM UID # -# Written by Grigori Fursin +# Authors: Grigori Fursin +# Contributors: +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import cmind.utils as utils diff --git a/cm/cmind/repo/automation/repo/module.py b/cm/cmind/repo/automation/repo/module.py index 3af99ac81e..44fe29d100 100644 --- a/cm/cmind/repo/automation/repo/module.py +++ b/cm/cmind/repo/automation/repo/module.py @@ -1,6 +1,13 @@ # CM automation to manage CM repositories # -# Written by Grigori Fursin +# Authors: Grigori Fursin +# Contributors: +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os diff --git a/cm/cmind/repos.py b/cm/cmind/repos.py index e75fcf8299..2365513c95 100644 --- a/cm/cmind/repos.py +++ b/cm/cmind/repos.py @@ -1,6 +1,13 @@ # Collective Mind repositories # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os @@ -105,10 +112,13 @@ def load(self, init = False): import pkgutil for mi, name, ispkg in pkgutil.iter_modules(): if name.startswith('cm') and name != 'cmind': - path = os.path.join(mi.path, name, 'repo') - path_cmr = os.path.join(path, 'cmr.yaml') - if os.path.isfile(path_cmr) and path not in self.paths: - self.paths.insert(0, path) + for x in [None, 'repo']: + path = os.path.join(mi.path, name) + if x != None: path = os.path.join(path, x) + path_cmr = os.path.join(path, 'cmr.yaml') + if os.path.isfile(path_cmr) and path not in self.paths: + self.paths.insert(0, path) + break # Check internal repo (will be after local) if self.path_to_internal_repo != '' and os.path.isdir(self.path_to_internal_repo): diff --git a/cm/cmind/utils.py b/cm/cmind/utils.py index a4b7249704..9bf5bc9832 100644 --- a/cm/cmind/utils.py +++ b/cm/cmind/utils.py @@ -2,7 +2,14 @@ # # Some functionality was reused from the CK framework for compatibility # -# Written by Grigori Fursin +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.txt +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# White paper: https://arxiv.org/abs/2406.16791 +# Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md import os diff --git a/cmx4mlops/cmx4mlops/README.md b/cmx4mlops/cmx4mlops/README.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/cmx4mlops/cmx4mlops/VERSION b/cmx4mlops/cmx4mlops/VERSION new file mode 100644 index 0000000000..4e379d2bfe --- /dev/null +++ b/cmx4mlops/cmx4mlops/VERSION @@ -0,0 +1 @@ +0.0.2 diff --git a/cmx4mlops/cmx4mlops/__init__.py b/cmx4mlops/cmx4mlops/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/incubator/cmx4mlops/cmr.yaml b/cmx4mlops/cmx4mlops/cmr.yaml similarity index 67% rename from incubator/cmx4mlops/cmr.yaml rename to cmx4mlops/cmx4mlops/cmr.yaml index 13493b9e90..0a67a3b123 100644 --- a/incubator/cmx4mlops/cmr.yaml +++ b/cmx4mlops/cmx4mlops/cmr.yaml @@ -3,10 +3,12 @@ uid: 428611a6db02407f git: true -version: "0.5.1" +version: "0.0.2" author: "Grigori Fursin" install_python_requirements: false -min_cm_version: "3.4.4" +min_cm_version: "3.5.3" + +prefix: repo diff --git a/cmx4mlops/cmx4mlops/repo/README.md b/cmx4mlops/cmx4mlops/repo/README.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/cmx4mlops/cmx4mlops/repo/automation/cmx-demo/_cm.yaml b/cmx4mlops/cmx4mlops/repo/automation/cmx-demo/_cm.yaml new file mode 100644 index 0000000000..ce472f5f7e --- /dev/null +++ b/cmx4mlops/cmx4mlops/repo/automation/cmx-demo/_cm.yaml @@ -0,0 +1,16 @@ +alias: cmx-demo +uid: 7099bc46c47f4a5e + +automation_alias: automation +automation_uid: bbeb15d8f0a944a4 + +tags: +- automation +- cmx +- demo + +authors: +- 'Grigori Fursin' + +contributors: +- 'Grigori Fursin' diff --git a/cmx4mlops/cmx4mlops/repo/automation/cmx-demo/modulex.py b/cmx4mlops/cmx4mlops/repo/automation/cmx-demo/modulex.py new file mode 100644 index 0000000000..3d17c08f7e --- /dev/null +++ b/cmx4mlops/cmx4mlops/repo/automation/cmx-demo/modulex.py @@ -0,0 +1,83 @@ +# Demo of the CMX interfaces +# +# Author(s): Grigori Fursin +# Contributor(s): +# +# Copyright: https://github.com/mlcommons/ck/blob/master/COPYRIGHT.md +# License: https://github.com/mlcommons/ck/blob/master/LICENSE.md +# +# White paper: https://arxiv.org/abs/2406.16791 +# History: https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md +# +# CK and CM project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md +# + +import os + +from cmind.automation import Automation +from cmind import utils + +class CAutomation(Automation): + """ + Automation actions + """ + + ############################################################ + def __init__(self, cmind, automation_file): + super().__init__(cmind, __file__) + + ############################################################ + def test(self, i): + """ + Test automation + + Args: + i (dict): CM input dict + + action (str): CM action + automation (str): CM automation + artifact (str): CM artifact + artifacts (list): list of extra CM artifacts + + control: (dict): various CM control + (out) (str): if 'con', output to console + ... + + (flags) + ... + + Returns: + (CM return dict): + + * return (int): return code == 0 if no error and >0 if error + * (error) (str): error string if return>0 + + * Output from this automation action + + """ + + # Access CM + print (self.cmind) + + # Print self path + print (self.path) + + # Print self meta + print (self.meta) + + # Print self automation module path + print (self.automation_file_path) + + # Print self automation module path + print (self.full_path) + + # Print self artifact + print (self.artifact) + + # Print input + + import json + print (json.dumps(i, indent=2)) + + return {'return':0} + diff --git a/incubator/cmx4mlops/README.md b/incubator/cmx4mlops/README.md deleted file mode 100644 index a0990367ef..0000000000 --- a/incubator/cmx4mlops/README.md +++ /dev/null @@ -1 +0,0 @@ -TBD diff --git a/incubator/cmx4mlperf-results/README.md b/incubator/cmx4mlperf-results/README.md deleted file mode 100644 index a0990367ef..0000000000 --- a/incubator/cmx4mlperf-results/README.md +++ /dev/null @@ -1 +0,0 @@ -TBD diff --git a/incubator/cmx4mlperf-results/cmr.yaml b/incubator/cmx4mlperf-results/cmr.yaml deleted file mode 100644 index cfd0b41c3a..0000000000 --- a/incubator/cmx4mlperf-results/cmr.yaml +++ /dev/null @@ -1,12 +0,0 @@ -alias: cmx4mlperf-results -uid: bd6defa59c804a6d - -git: true - -version: "0.0.1" - -author: "Grigori Fursin" - -install_python_requirements: false - -min_cm_version: "3.4.4" diff --git a/requirements.txt b/requirements.txt index f0e34ec5e0..819df0f8e4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -cmind>=3.3.3 +cmind>=3.5.2 pyyaml requests setuptools