diff --git a/.github/workflows/_publish.yaml_ b/.github/workflows/publish-docs.yml similarity index 87% rename from .github/workflows/_publish.yaml_ rename to .github/workflows/publish-docs.yml index 97ef2a8665..368007a7dc 100644 --- a/.github/workflows/_publish.yaml_ +++ b/.github/workflows/publish-docs.yml @@ -1,13 +1,16 @@ name: Publish documentation site - on: release: types: [published] push: branches: - - main + - master - docs + paths: + - docs/** + - mkdocs.yml + - README.md jobs: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 35f7a91bb3..e012953e8a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,12 +14,18 @@ Modify the project in your own fork and issue a pull request once you want other to take a look at what you have done and discuss the proposed changes. Ensure that cla-bot and other checks pass for your Pull requests. -Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://cKnowledge.org/gfursin), -sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. -Since then, this open-source technology (CM, CM4MLOps, CM4ABTF, CM4Research, etc) -is being developed as a community effort thanks to all our volunteers, collaborators and contributors -listed here in alphabetical order: +Collective Knowledge (CK) and Collective Mind (CM) +were created by [Grigori Fursin](https://arxiv.org/abs/2406.16791), +sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons +to benefit everyone. Since then, this open-source automation technology +(CM/CMX, CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, etc) is being extended +as a community effort thanks to all our volunteers, collaborators +and contributors listed here in alphabetical order: +* @Henryfzh +* @Leonard226 +* @Oseltamivir +* @Submandarine * Resmi Arjun * Omar Benjelloun (Google) * Alice Cheng (Nvidia) @@ -62,7 +68,7 @@ listed here in alphabetical order: * Arun Tejusve Raghunath Rajan (Cruise) * Vijay Janapa Reddi (Harvard University) * Andrew Reusch (OctoML) -* Anandhu S (Kerala Technical University) +* Anandhu Sooraj (Kerala Technical University) * Sergey Serebryakov (HPE) * Warren Schultz (Principled Technologies) * Amrutha Sheleenderan (Kerala Technical University) diff --git a/README.md b/README.md index 7b747720ab..4ef2fe81dd 100755 --- a/README.md +++ b/README.md @@ -11,40 +11,52 @@ ### About -[Collective Knowledge (CK)](https://cKnowledge.org) in an educational project -to help researchers and engineers automate their repetitive, tedious and time-consuming tasks -to build, run, benchmark and optimize AI, ML and other applications and systems -across diverse and continuously changing models, data, software and hardware. +[Collective Knowledge (CK, CM, CM4MLOps, CM4MLPerf and CMX)](https://cKnowledge.org) +is an educational community project to learn how to run AI, ML and other emerging workloads +in the most efficient and cost-effective way across diverse models, data sets, software and hardware. CK consists of several sub-projects: -* [Collective Mind framework (CM)](cm) - a very light-weight Python-based framework with minimal dependencies - to help users implement, share and reuse cross-platform automation recipes to - build, benchmark and optimize applications on any platform - with any software and hardware. - - * [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference) +* [Collective Mind framework (CM)](cm) - a very lightweight Python-based framework with minimal dependencies + intended to help researchers and engineers automate their repetitive, tedious and time-consuming tasks + to build, run, benchmark and optimize AI, ML and other applications and systems + across diverse and continuously changing models, data, software and hardware. * [CM4MLOPS](https://github.com/mlcommons/cm4mlops) - a collection of portable, extensible and technology-agnostic automation recipes - with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications + with a common CLI and Python API (CM scripts) to unify and automate + all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications on diverse platforms with any software and hardware: see [online catalog at CK playground](https://access.cknowledge.org/playground/?action=scripts), [online MLCommons catalog](https://docs.mlcommons.org/cm4mlops/scripts) + * [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference) + * [CM4ABTF](https://github.com/mlcommons/cm4abtf) - a unified CM interface and automation recipes to run automotive benchmark across different models, data sets, software and hardware from different vendors. -* [CMX (the next generation of CM)](cm/docs/cmx) - we are developing the next generation of CM - to make it simpler and more flexible based on user feedback. Please follow - this project [here](https://github.com/orgs/mlcommons/projects/46). +* [CMX (the next generation of CM, CM4MLOps and CM4MLPerf)](cm/docs/cmx) - + we are developing the next generation of CM + to make it simpler and more flexible based on user feedback. Please follow + this project [here]( https://github.com/orgs/mlcommons/projects/46 ). + * [Collective Knowledge Playground](https://access.cKnowledge.org) - a unified platform to list CM scripts similar to PYPI, aggregate AI/ML Systems benchmarking results in a reproducible format with CM workflows, and organize [public optimization challenges and reproducibility initiatives](https://access.cknowledge.org/playground/?action=challenges) to co-design more efficient and cost-effiective software and hardware for emerging workloads. + * [CM4MLPerf-results](https://github.com/mlcommons/cm4mlperf-results) - + a simplified and unified representation of the past MLPerf results + for further visualization and analysis using [CK graphs](https://access.cknowledge.org/playground/?action=experiments) + (*the new version is coming soon*). + + * [Artifact Evaluation](https://cTuning.org/ae) - automating artifact evaluation and reproducibility initiatives at ML and systems conferences. +### Deprecated and archived projects + +* [CM-MLOps](cm-mlops) - now [CM4MLOps](cm4mlops) +* [CK automation framework v1 and v2](ck) - now [CM](cm) ### License @@ -56,31 +68,51 @@ CK consists of several sub-projects: * Copyright (c) 2021-2024 MLCommons * Copyright (c) 2014-2021 cTuning foundation -### Motivation and long-term vision +### Author -You can learn more about the motivation behind these projects from the following articles and presentations: +* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning) -* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] +### Maintainers + +* [Collective Mind (CM)](cm): [Grigori Fursin](https://cKnowledge.org/gfursin) +* CM4MLOps (CM automation recipes): [Arjun Suresh](https://github.com/arjunsuresh) and [Anandhu Sooraj](https://github.com/anandhu-eng) +* CMX (the next generation of CM, CM4MLOps and CM4MLPerf): [Grigori Fursin](https://cKnowledge.org/gfursin) + +### Citing our project + +If you found the CM automation framework helpful, kindly reference this article: +[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. + +To learn more about the motivation behind CK and CM technology, please explore the following presentations: + +* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] * ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] -* ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] +* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] +* Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ] + ### CM Documentation * [CM installation GUI](https://access.cknowledge.org/playground/?action=install) -* [CM Getting Started Guide and FAQ](docs/getting-started.md) - * [Common CM interface to run MLPerf inference benchmarks](docs/mlperf/inference) - * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](docs/tutorials/common-interface-to-reproduce-research-projects.md) +* [CM Getting Started Guide and FAQ](https://github.com/mlcommons/ck/tree/master/docs/getting-started.md) + * [Common CM interface to run MLPerf inference benchmarks](https://github.com/mlcommons/ck/tree/master/docs/mlperf/inference) + * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](https://github.com/mlcommons/ck/tree/master/docs/tutorials/common-interface-to-reproduce-research-projects.md) * [CM automation recipes for MLOps and DevOps](https://access.cknowledge.org/playground/?action=scripts) - * [Other CM tutorials](docs/tutorials) -* [Full documentation](docs/README.md) -* [CM development tasks](docs/taskforce.md#current-tasks) -* [CM and CK history](docs/history.md) + * [Other CM tutorials](https://github.com/mlcommons/ck/tree/master/docs/tutorials) +* [Full documentation](https://github.com/mlcommons/ck/tree/master/docs/README.md) +* [CM taskforce](https://github.com/mlcommons/ck/tree/master/docs/taskforce.md) +* [CMX, CM and CK history](https://github.com/mlcommons/ck/tree/master/docs/history.md) ### Acknowledgments -Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://cKnowledge.org/gfursin), -sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. -Since then, this open-source technology (CM, CM4MLOps, CM4MLPerf, CM4ABTF, CM4Research, etc) -is being developed as a community effort thanks to all our -[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! +The open-source Collective Knowledge project (CK, CM, CM4MLOps/CM4MLPerf, +CM4Research and CMX) was created by [Grigori Fursin](https://cKnowledge.org/gfursin) +and sponsored by cTuning.org, OctoAI and HiPEAC. +Grigori donated CK to MLCommons to benefit the community +and to advance its development as a collaborative, community-driven effort. + +We thank [MLCommons](https://mlcommons.org), [FlexAI](https://flex.ai) +and [cTuning](https://cTuning.org) for supporting this project, +as well as our dedicated [volunteers and collaborators](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) +for their feedback and contributions! diff --git a/ck/README.md b/ck/README.md index e282005b94..8b6c51d79c 100644 --- a/ck/README.md +++ b/ck/README.md @@ -8,6 +8,9 @@ behind CK in this [ACM TechTalk](https://www.youtube.com/watch?v=7zpeIVwICa4) and the [journal article](https://doi.org/10.1098/rsta.2020.0211).** +We keep this directory for backwards compatibility to let users reproduce past projects relying on this repository. + +


diff --git a/cm-mlops/README.md b/cm-mlops/README.md index e3333e6a2e..f65fdc2677 100644 --- a/cm-mlops/README.md +++ b/cm-mlops/README.md @@ -1,3 +1,5 @@ **This repository is archived. In April 2024, we have separated this CM repo from the CM framework and moved it to a [standalone mlcommons@cm4mlops repository](https://github.com/mlcommons/cm4mlops/tree/dev) following the suggesion from our users and MLCommons members.** + +We keep this directory for backwards compatibility to let users reproduce past projects relying on this repository. diff --git a/cm/CHANGES.md b/cm/CHANGES.md index 537ff2f58c..c3fb785ec1 100644 --- a/cm/CHANGES.md +++ b/cm/CHANGES.md @@ -1,3 +1,54 @@ +## V3.4.4.1 + - added tabulate as requirement + +## V3.4.4 + - improved error reporting in utils.load_json and utils.load_yaml + - added utils.substitute_template + - enhanced 'cm pull repo' based on user feedback + +## V3.4.3 + - fixed pyproject.toml + - fixed docs generation + - improved cm pull repo with branch and SHA (#1355) + +## V3.4.2 + - added utils.flatten_dict + - added utils.safe_int + - added utils.safe_float + - added utils.get_set + - added utils.digits + +## V3.4.1 + - reduced Python min version in pyproject.toml to 3.7 for backwards compatibility + +## V3.4.0 + - added pyproject.toml to install CM and deprecated setup.py + +## V3.3.4 + - added utils.path2: + add quotes if spaces in path + - added utils.update_dict_with_flat_key: + update dictionary via flat key (x.y.z) + - added utils.get_value_from_dict_with_flat_key + get value from dict via flat key (x.y.z) + - added utils.load_module + universal python module loader + +## V3.3.3 + - fixed CM logger issue + +## V3.3.2 + - moved "CMX stack error" to -log + - improved error reporting + +## V3.3.1 + - allow "cm/cmx pull repo {URL}" along with "cm/cmx pull repo --url={URL}" + - CMX: added "automation_full_path" to -log + +## V3.2.9 + - fixed minor bug with JSON console output + - fixed minor bug with console in cmind.x function + ## V3.2.8 - fixed Docker test for mlperf-inference with cm4mlops - minor improvements diff --git a/cm/MANIFEST.in b/cm/MANIFEST.in new file mode 100644 index 0000000000..f08d39c547 --- /dev/null +++ b/cm/MANIFEST.in @@ -0,0 +1,3 @@ +include CHANGES.md +include CONTRIBUTING.md +include COPYRIGHT.txt diff --git a/cm/README.md b/cm/README.md index f5ae36387d..ac726975ae 100644 --- a/cm/README.md +++ b/cm/README.md @@ -9,18 +9,20 @@ ### About -Collective Mind (CM) is a small, modular, cross-platform and decentralized workflow automation framework -with a human-friendly interface to make it easier to build, run, benchmark and optimize applications -across diverse models, data sets, software and hardware. +Collective Mind (CM) is a small [Python package](https://pypi.org/project/cmind) +with a unified CLI and API designed for creating and managing +portable and technology-agnostic automations for MLOps, DevOps and ResearchOps. +It is intended to make it easier to build, run, benchmark and optimize applications +across diverse models, data sets, software and hardware. + CM is a part of [Collective Knowledge (CK)](https://github.com/mlcommons/ck) - -an educational community project to learn how to run emerging workloads +an educational community project to learn how to run AI, ML and other emerging workloads in the most efficient and cost-effective way across diverse -and continuously changing systems. +and continuously changing systems using the MLPerf benchmarking methodology. -CM includes a collection of portable, extensible and technology-agnostic automation recipes -with a common API and CLI (aka CM scripts) to unify and automate different steps -required to compose, run, benchmark and optimize complex ML/AI applications +CM includes a [collection of portable, extensible and technology-agnostic automation recipes](https://access.cknowledge.org/playground/?action=scripts) +(aka CM scripts) to unify and automate different steps required to compose, run, benchmark and optimize complex ML/AI applications on any platform with any software and hardware. CM scripts extend the concept of `cmake` with simple Python automations, native scripts @@ -44,12 +46,16 @@ from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors: and simple JSON/YAML descriptions instead of inventing new workflow languages; * must have the same interface to run all automations natively, in a cloud or inside containers. +### Author and maintainer + +* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning) + ### Resources -* CM v2.x (stable version 2022-cur): [installation on Linux, Windows, MacOS](https://access.cknowledge.org/playground/?action=install) ; +* CM v2.x (2022-cur) (stable): [installation on Linux, Windows, MacOS](https://access.cknowledge.org/playground/?action=install) ; [docs](https://docs.mlcommons.org/ck) ; [popular commands](https://github.com/mlcommons/ck/tree/master/cm/docs/demos/some-cm-commands.md) ; [getting started guide](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md) -* CM v3.x (prototype 2024-cur): [docs](https://github.com/mlcommons/ck/tree/master/cm/docs/cmx) +* CM v3.x aka CMX (2024-cur) (stable): [docs](https://github.com/orgs/mlcommons/projects/46) * MLPerf inference benchmark automated via CM * [Run MLPerf for submissions](https://docs.mlcommons.org/inference) * [Run MLPerf at the Student Cluster Competition'24](https://docs.mlcommons.org/inference/benchmarks/text_to_image/reproducibility/scc24) @@ -61,21 +67,26 @@ from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors: [Apache 2.0](LICENSE.md) -### Citing CM and CM4MLOps +### Citing CM project -If you found CM useful, please cite this article: +If you found CM automations useful, please cite this article: [ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. -You can learn more about the motivation behind these projects from the following articles and presentations: +You can learn more about the motivation behind these projects from the following presentations: -* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] +* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] * ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] * ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] ### Acknowledgments -The Collective Mind framework (CM) was created by [Grigori Fursin](https://cKnowledge.org/gfursin), -sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. -Since then, this open-source technology (CM, CM4MLOps, CM4MLPerf, CM4ABTF, CM4Research, etc) -is being developed as a community effort thanks to all our -[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! +The Collective Mind (CM) automation framework was originally +developed by [Grigori Fursin](https://cKnowledge.org/gfursin), +as a part of the [Collective Knowledge educational initiative](https://cKnowledge.org), +sponsored by [cTuning.org](https://cTuning.org) and [cKnowledge.org](https://cKnowledge.org), +and contributed to MLCommons for the benefit of all. + +This open-source technology, including CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, and more, +is a collaborative project supported by [MLCommons](https://mlcommons.org), +[FlexAI](https://flex.ai), [cTuning](https://cTuning.org) +and our [amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! diff --git a/cm/cmind/__init__.py b/cm/cmind/__init__.py index 5b70144f0b..e996b79326 100644 --- a/cm/cmind/__init__.py +++ b/cm/cmind/__init__.py @@ -2,7 +2,7 @@ # # Written by Grigori Fursin -__version__ = "3.2.8" +__version__ = "3.4.4" from cmind.core import access from cmind.core import x diff --git a/cm/cmind/cli.py b/cm/cmind/cli.py index be9ea72a9d..7e03d2347c 100644 --- a/cm/cmind/cli.py +++ b/cm/cmind/cli.py @@ -193,21 +193,6 @@ def run_experiment(argv = None): return run(['run', 'experiment'] + argv) -############################################################ -def run_ff(argv = None): - """ - """ - - # Access CMX - if argv is None: - argv = sys.argv[1:] - - x = 'lex' - if len(argv) >0 and argv[0].startswith('_misc'): - x = argv[0] - - return runx(['run', 'f'+x+'.flow'] + argv) - ############################################################ def parse(cmd): """ diff --git a/cm/cmind/config.py b/cm/cmind/config.py index 7b56442246..4071601901 100644 --- a/cm/cmind/config.py +++ b/cm/cmind/config.py @@ -37,7 +37,7 @@ def __init__(self, config_file = None): "flag_help2": "help", "error_prefix": "CM error:", - "error_prefix2": "CMX detected an issue", + "error_prefix2": "CMX detected a problem", "info_cli": "cm {action} {automation} {artifact(s)} {flags} @input.yaml @input.json", "info_clix": "cmx {action} {automation} {artifact(s)} {CMX control flags (-)} {CMX automation flags (--)}", diff --git a/cm/cmind/core.py b/cm/cmind/core.py index a8e1096315..6c6327b09b 100644 --- a/cm/cmind/core.py +++ b/cm/cmind/core.py @@ -103,6 +103,7 @@ def __init__(self, repos_path = '', debug = False): # Logging self.logger = None + self.xlogger = None # Index self.index = None @@ -172,17 +173,18 @@ def errorx(self, r): message = '' - if not self.logger == None or (module_path != '' and lineno != ''): + if not self.xlogger == None or (module_path != '' and lineno != ''): call_stack = self.state.get('call_stack', []) - if not self.logger == None: + if not self.xlogger == None: self.log(f"x error call stack: {call_stack}", "debug") self.log(f"x error: {r}", "debug") - sys.stderr.write('^'*60 + '\n') +# sys.stderr.write('^'*60 + '\n') + sys.stderr.write('\n') - if not self.logger == None: + if not self.xlogger == None: sys.stderr.write('CMX call stack:\n') for cs in call_stack: @@ -195,7 +197,7 @@ def errorx(self, r): message += self.cfg['error_prefix2'] if module_path != '' and lineno !='': - message += f' in {module_path} ({lineno}):\n\n' + message += f' while running automation {module_path} ({lineno}):\n\n' text = r['error'] text = text[0].upper() + text[1:] @@ -295,7 +297,7 @@ def log(self, s, t = 'info'): None """ - logger = self.logger + logger = self.xlogger if logger != None: if t == 'debug': @@ -903,7 +905,7 @@ def x(self, i, out = None): meta = ii) # Check logging - if self.logger is None: + if self.xlogger is None: log_level = None if use_log == "false": @@ -929,7 +931,7 @@ def x(self, i, out = None): log_level = logging.INFO # Configure - self.logger = logging.getLogger("cmx") + self.xlogger = logging.getLogger("cmx") logging.basicConfig(filename = log_file, filemode = 'w', level = log_level) # Check if force out programmatically (such as from CLI) @@ -942,7 +944,7 @@ def x(self, i, out = None): recursion = self.state.get('recursion', 0) self.state['recursion'] = recursion + 1 - if not self.logger == None: + if not self.xlogger == None: log_action = i.get('action', '') log_automation = i.get('automation', '') log_artifact = i.get('artifact', '') @@ -966,7 +968,7 @@ def x(self, i, out = None): r['return'] = 1 r['error'] = delayed_error - if not self.logger == None: + if not self.xlogger == None: self.log(f"x output: {r}", "debug") self.state['recursion'] = recursion @@ -1341,6 +1343,9 @@ def _x(self, i, control): loaded_common_automation = True # Finalize automation class initialization + if not self.xlogger == None: + self.log(f"x automation_full_path: {automation_full_path}", "info") + initialized_automation = loaded_automation_class(self, automation_full_path) initialized_automation.meta = automation_meta initialized_automation.full_path = automation_full_path @@ -1656,7 +1661,7 @@ def access(i): return cm.access(i) ############################################################ -def x(i): +def x(i, out = None): """ Automatically initialize CM and run automations without the need to initialize and customize CM class. @@ -1668,9 +1673,9 @@ def x(i): global cm if cm is None: - cm=CM() + cm = CM() - return cm.x(i) + return cm.x(i, out = out) ############################################################ def error(i): diff --git a/cm/cmind/repo/automation/repo/module.py b/cm/cmind/repo/automation/repo/module.py index 6446e45596..2b7578eb1b 100644 --- a/cm/cmind/repo/automation/repo/module.py +++ b/cm/cmind/repo/automation/repo/module.py @@ -34,6 +34,7 @@ def pull(self, i): (new_branch) (str): Create new Git branch (checkout) (str): Git checkout (checkout_only) (bool): only checkout existing repo + (dir) (str): use repository in this directory (depth) (int): Git depth (desc) (str): brief repository description (1 line) (prefix) (str): extra directory to keep CM artifacts @@ -54,11 +55,11 @@ def pull(self, i): console = i.get('out') == 'con' - alias = i.get('artifact','') - url = i.get('url','') - desc = i.get('desc','') - prefix = i.get('prefix','') - pat = i.get('pat','') + alias = i.get('artifact', '') + url = i.get('url', '') + desc = i.get('desc', '') + prefix = i.get('prefix', '') + pat = i.get('pat', '') extra_cmd_git = i.get('extra_cmd_git', '') extra_cmd_pip = i.get('extra_cmd_pip', '') @@ -66,6 +67,12 @@ def pull(self, i): checkout_only = i.get('checkout_only', False) skip_zip_parent_dir = i.get('skip_zip_parent_dir', False) + # Check alias is URL + if url == '' and (alias.startswith('https://') or alias.startswith('git@')): + url = alias + alias = '' + + # Process URL and alias if url == '': if alias != '': url = self.cmind.cfg['repo_url_prefix'] @@ -126,13 +133,14 @@ def pull(self, i): branch = i.get('branch', '') new_branch = i.get('new_branch', '') checkout = i.get('checkout', '') + _dir = i.get('dir', '') r = net.request({'get': {'action': 'check-migration-repo-notes', 'repo': url, 'branch': branch, 'checkout': checkout}}) notes = r.get('dict', {}).get('notes','') if notes !='': print (notes) - if alias == 'mlcommons@ck' and branch == '' and checkout == '': + if alias == 'mlcommons@ck' and branch == '' and checkout == '' and _dir == '': print ('=========================================================================') print ('Warning: mlcommons@ck was automatically changed to mlcommons@cm4mlops.') print ('If you want to use older mlcommons@ck repository, use branch or checkout.') @@ -147,6 +155,7 @@ def pull(self, i): 'branch': branch, 'new_branch': new_branch, 'checkout': checkout, + 'dir': _dir, 'depth': i.get('depth', '')}] @@ -156,8 +165,8 @@ def pull(self, i): warnings = [] - if not self.cmind.logger == None: - self.cmind.log(f"x repo log: {pull_repos}", "debug") +# if not self.cmind.xlogger == None: +# self.cmind.log(f"x repo log: {pull_repos}", "debug") for repo in pull_repos: alias = repo['alias'] @@ -167,6 +176,7 @@ def pull(self, i): checkout = repo.get('checkout','') depth = repo.get('depth','') path_to_repo = repo.get('path_to_repo', None) + _dir = repo.get('dir', '') if console: print (self.cmind.cfg['line']) @@ -179,6 +189,8 @@ def pull(self, i): print ('New branch: {}'.format(new_branch)) if checkout!='': print ('Checkout: {}'.format(checkout)) + if _dir!='': + print ('Directory: {}'.format(_dir)) if depth!='' and depth!=None: print ('Depth: {}'.format(str(depth))) print ('') @@ -191,6 +203,7 @@ def pull(self, i): branch = branch, new_branch = new_branch, checkout = checkout, + _dir = _dir, console = console, desc=desc, prefix=prefix, diff --git a/cm/cmind/repos.py b/cm/cmind/repos.py index cfe6baf3e9..e75fcf8299 100644 --- a/cm/cmind/repos.py +++ b/cm/cmind/repos.py @@ -99,6 +99,17 @@ def load(self, init = False): r = utils.save_json(full_path_to_repo_paths, meta = self.paths) if r['return']>0: return r + # Skip internal repos + skip_internal_repos = os.environ.get('CM_CORE_SKIP_INTERNAL_REPOS','').strip().lower() + if skip_internal_repos not in ['1', 'true', 'yes']: + import pkgutil + for mi, name, ispkg in pkgutil.iter_modules(): + if name.startswith('cm') and name != 'cmind': + path = os.path.join(mi.path, name, 'repo') + path_cmr = os.path.join(path, 'cmr.yaml') + if os.path.isfile(path_cmr) and path not in self.paths: + self.paths.insert(0, path) + # Check internal repo (will be after local) if self.path_to_internal_repo != '' and os.path.isdir(self.path_to_internal_repo): self.paths.insert(0, self.path_to_internal_repo) @@ -294,7 +305,7 @@ def process(self, repo_path, mode='add'): return rr ############################################################ - def pull(self, alias, url = '', branch = '', checkout = '', console = False, desc = '', prefix = '', depth = None, + def pull(self, alias, url = '', branch = '', checkout = '', _dir = '', console = False, desc = '', prefix = '', depth = None, path_to_repo = None, checkout_only = False, skip_zip_parent_dir = False, extra_cmd_git = '', extra_cmd_pip = '', new_branch = ''): """ @@ -307,6 +318,7 @@ def pull(self, alias, url = '', branch = '', checkout = '', console = False, des (new_branch) (str): Create new branch (checkout) (str): Git repository checkout (checkout_only) (bool): only checkout existing repo + (_dir) (str): use repository in this directory (depth) (int): Git repository depth (console) (bool): if True, print some info to console (desc) (str): optional repository description @@ -335,6 +347,9 @@ def pull(self, alias, url = '', branch = '', checkout = '', console = False, des if path_to_repo == None: path_to_repo = os.path.join(self.full_path_to_repos, alias) + if _dir != '': + path_to_repo = os.path.join(path_to_repo, _dir) + if console: print ('Local path: '+path_to_repo) print ('') @@ -482,10 +497,14 @@ def pull(self, alias, url = '', branch = '', checkout = '', console = False, des # When checkout only, we do not need -b for branch extra_flag = ' ' if checkout_only else ' -b ' - if branch != '': + if branch != '' and checkout != '': + # if both branch and checkout are specified, we do checkout and set remote branch + cmd = "git checkout -b " + branch + " " + checkout + " && git branch --set-upstream-to=origin/"+ branch + " " + branch + + elif branch != '': cmd = 'git fetch && git checkout ' + branch - if checkout!='': + elif checkout!='': cmd += ' ' + checkout if console: diff --git a/cm/cmind/utils.py b/cm/cmind/utils.py index a631236d01..5ba5bbce1a 100644 --- a/cm/cmind/utils.py +++ b/cm/cmind/utils.py @@ -204,7 +204,7 @@ def load_json(file_name, check_if_exists = False, encoding='utf8'): try: meta = json.load(jf) except Exception as e: - return {'return':4, 'error': format(e)} + return {'return':4, 'error': f'detected problem in {file_name}: {e}'} return {'return':0, 'meta': meta} @@ -274,7 +274,7 @@ def load_yaml(file_name, check_if_exists = False, encoding = 'utf8'): # To support old versions meta = yaml.safe_load(yf) except Exception as e: - return {'return':4, 'error': format(e)} + return {'return':4, 'error': f'detected problem in {file_name}: {e}'} return {'return':0, 'meta': meta} @@ -1947,3 +1947,226 @@ def test_input(i): 'unknown_keys_str': unknown_keys_str} return r + +############################################################################## +def path2(path): + """ + Add quotes if spaces in path + """ + new_path = f'"{path}"' if not path.startswith('"') and ' ' in path else path + + return new_path + +############################################################################## +def update_dict_with_flat_key(key, value, d): + """ + Update dictionary via flat key (x.y.z) + """ + + if '.' in key: + keys = key.split('.') + + new_d = d + + first = True + + for key in keys[:-1]: + if first: + first = False + + if key not in new_d: + new_d[key] = {} + + new_d = new_d[key] + + new_d[keys[-1]] = value + else: + d[key] = value + + return {'return':0} + +############################################################################## +def get_value_from_dict_with_flat_key(key, d): + """ + Get value from dict via flat key (x.y.z) + """ + + if '.' in key: + keys = key.split('.') + new_d = d + for key in keys[:-1]: + if key in new_d: + new_d = new_d[key] + value = new_d.get(keys[-1]) + else: + value = d.get(key) + + return value + +############################################################################## +def load_module(cmind, task_path, sub_module_name): + """ + Universal python module loaders + """ + + import importlib + + sub_module_obj = None + + sub_module_path = os.path.join(task_path, sub_module_name) + if os.path.isfile(sub_module_path): + sub_module_spec = importlib.util.spec_from_file_location(sub_module_name, sub_module_path) + if sub_module_spec == None: + return cmind.prepare_error(1, f"Can\'t load Python module file spec {sub_module_path}") + + try: + sub_module_obj = importlib.util.module_from_spec(sub_module_spec) + sub_module_spec.loader.exec_module(sub_module_obj) + except Exception as e: # pragma: no cover + return cmind.prepare_error(1, f"Can\'t load Python module code {sub_module_path}:\n\n{e}") + + return {'return':0, 'sub_module_obj': sub_module_obj, 'sub_module_path': sub_module_path} + +############################################################################## +def flatten_dict(d, fd = {}, prefix = ''): + """ + Flatten dict ({"x":{"y":"z"}} -> x.y=z) + """ + + + for k in d: + v = d[k] + + if type(v) == list and len(v) == 1 and type(v[0]) == dict: + v = v[0] + + if type(v) == dict: + new_prefix = prefix + k + '.' + flatten_dict(v, fd, new_prefix) + else: + fd[prefix + k] = str(v) + + return + +############################################################################## +def safe_int(i, d): + """ + Get safe int (useful for sorting function) + + Args: + i (any): variable with any type + d (int): default value + + Returns: + (int): returns i if it can be converted to int, or d otherwise + """ + + r = d + try: + r = int(i) + except Exception as e: + pass + + return r + +############################################################################## +def safe_float(i, d): + """ + Get safe float (useful for sorting function) + + Args: + i (any): variable with any type + d (float): default value + + Returns: + (float): returns i if it can be converted to float or d otherwise + + """ + + r = d + try: + r = float(i) + except Exception as e: + pass + + return r + +############################################################################## +def get_set(meta, key, state, keys): + """ + Get value from dict and update in another dict + + Args: + meta (dict): original dict + key (str): key to get value from original dict + state (dict): target dict + keys (list): list of keys to set value in target dict + + Returns: + (Python object): value from original dict or None + + """ + + v = meta.get(key, None) + if v != None: + cur = state + vv = None + for k in keys: + if k == keys[-1]: + vv = cur.get(k, None) + if vv == None: + cur[k] = v + else: + if k not in cur: + cur[k] = {} + cur = cur[k] + + return v + +############################################################################## +def digits(s, first = True): + """ + Get first digits and convert to int + + Args: + s (str): string ("1.3+xyz") + first (bool): if True, choose only first digits, otherwise all + + Returns: + (int): returns int from first digits or 0 + + """ + + v = 0 + + digits = '' + for char in s: + if char.isdigit(): + digits+=char + elif first: + break + + try: + v = int(digits) + except Exception as e: + pass + + return v + +############################################################################## +def substitute_template(template, variables): + """ + Substitutes variables in a template string with values from a dictionary. + + Args: + template (str): The template string with placeholders (e.g., "something-{var1}-something-{var2}"). + vars (dict): A dictionary containing variable-value pairs (e.g., {'var1': 'a', 'var2': 'b'}). + + Returns: + str: The template string with placeholders replaced by the corresponding values. + """ + try: + return template.format(**variables) + except KeyError as e: + return f"Error: Missing value for {e.args[0]} in the vars dictionary." + diff --git a/cm/dev/cmx/README.md b/cm/dev/cmx/README.md deleted file mode 100644 index 09e623a353..0000000000 --- a/cm/dev/cmx/README.md +++ /dev/null @@ -1,6 +0,0 @@ -# Resolving tech. issues - -## Can't run CM console scripts - -`python setup.py install` is gradually deprecated and may result in failing console scripts. -Use `pip install .` instead. diff --git a/cm/docs/cmx/README.md b/cm/docs/cmx/README.md index b7880a55ef..5f54ce2759 100644 --- a/cm/docs/cmx/README.md +++ b/cm/docs/cmx/README.md @@ -1,6 +1,9 @@ # Collective Mind v3 aka CMX -We prototype the next generation of CM. +CMX is the next generation of the CM automation framework +being developed by [Grigori Fursin](https://cKnowledge.org/gfursin) +based on extensive feedback from CM users. + ## Documentation diff --git a/cm/pyproject.toml b/cm/pyproject.toml new file mode 100644 index 0000000000..f0786bfdb6 --- /dev/null +++ b/cm/pyproject.toml @@ -0,0 +1,94 @@ +[build-system] +build-backend = "setuptools.build_meta" +requires = ["setuptools"] + +[project] +name = "cmind" + +authors = [ + {name = "Grigori Fursin", email = "grigori.fursin@ctuning.org"} +] + +maintainers = [ + {name = "Grigori Fursin", email = "grigori.fursin@ctuning.org"} +] + +description = "Collective Mind automation framework (CM)" + +requires-python = ">=3.7" + +dependencies = [ + "pyyaml", + "requests", + "setuptools", + "wheel", + "giturlparse", + "tabulate" +] + +keywords = [ + "cmind", + "cm", + "cmx", + "collective mind", + "automation", + "portability", + "reusability", + "mlops", + "devops", + "vmlops", + "api", + "cli" +] + +license = {text = "Apache 2.0"} + +classifiers = [ + "License :: OSI Approved :: Apache Software License", + "Operating System :: OS Independent", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", +] + +dynamic = ["readme", "version"] + +[tool.setuptools.dynamic] +readme = {file = "README.md", content-type = "text/markdown"} +version = {attr = "cmind.__version__"} + +[tool.setuptools] +zip-safe = false +include-package-data = true + +[tool.setuptools.package-data] +cmind = ["repo/*", + "repo/automation/automation/*", + "repo/automation/ckx/*", + "repo/automation/core/*", + "repo/automation/core/cm_60cb625a46b38610/*", + "repo/automation/repo/*" + ] + +[tool.setuptools.packages.find] +where = ["."] +include = ["cmind"] +exclude = ["docs", "docs.*", "tests", "tests.*"] + + +[project.urls] +Homepage = "https://cKnowledge.org" +Repository = "https://github.com/mlcommons/ck/tree/master/cm" +Issues = "https://github.com/mlcommons/ck/issues" +Changelog = "https://github.com/mlcommons/ck/blob/master/cm/CHANGES.md" + +[project.scripts] +cmind = "cmind.cli:run" +cm = "cmind.cli:run" +cmx = "cmind.cli:runx" +cmr = "cmind.cli:run_script" +cmrd = "cmind.cli:docker_script" +cmg = "cmind.cli:gui_script" +cme = "cmind.cli:run_experiment" diff --git a/cm/requirements.txt b/cm/requirements.txt index 859f84a20b..48a3980c2b 100644 --- a/cm/requirements.txt +++ b/cm/requirements.txt @@ -1,4 +1,6 @@ pyyaml requests setuptools +wheel giturlparse +tabulate diff --git a/cm/setup.py b/cm/setup_.py similarity index 95% rename from cm/setup.py rename to cm/setup_.py index 1a4bccde7d..fcaa548819 100644 --- a/cm/setup.py +++ b/cm/setup_.py @@ -94,14 +94,13 @@ def run(self): 'install': custom_install }, - install_requires=['pyyaml', 'requests', 'setuptools', 'giturlparse'], + install_requires=['pyyaml', 'requests', 'setuptools', 'wheel', 'giturlparse'], entry_points={"console_scripts": [ "cmind = cmind.cli:run", "cm = cmind.cli:run", "cmx = cmind.cli:runx", "cmr = cmind.cli:run_script", - "flexflow = cmind.cli:run_ff", "cmrd = cmind.cli:docker_script", "cmg = cmind.cli:gui_script", "cme = cmind.cli:run_experiment" diff --git a/cm/setup_install.bat b/cm/setup_install.bat deleted file mode 100644 index dee8b9cc12..0000000000 --- a/cm/setup_install.bat +++ /dev/null @@ -1 +0,0 @@ -pip install . diff --git a/cm4abtf/README.md b/cm4abtf/README.md new file mode 100644 index 0000000000..bd335a8a54 --- /dev/null +++ b/cm4abtf/README.md @@ -0,0 +1 @@ +Development CM repository: https://github.com/mlcommons/cm4abtf diff --git a/cm4mlops/README.md b/cm4mlops/README.md new file mode 100644 index 0000000000..afc3746840 --- /dev/null +++ b/cm4mlops/README.md @@ -0,0 +1,5 @@ +Development repositories: +* https://github.com/mlcommons/cm4mlops + +Deprecated: +* https://github.com/mlcommons/ck-mlops diff --git a/cm4mlperf/README.md b/cm4mlperf/README.md new file mode 100644 index 0000000000..15c19d819c --- /dev/null +++ b/cm4mlperf/README.md @@ -0,0 +1,4 @@ +Development repositories: +* https://github.com/mlcommons/cm4mlperf-results +* https://github.com/mlcommons/cm4mlperf-inference +* https://github.com/mlcommons/cm4mlops/tree/mlperf-inference diff --git a/cmx/README.md b/cmx/README.md new file mode 100644 index 0000000000..f74c85a069 --- /dev/null +++ b/cmx/README.md @@ -0,0 +1,17 @@ +# Collective Mind v3 aka CMX + +CMX represents the next generation of the CM automation framework. +Developed by Grigori Fursin, it builds upon valuable feedback +gathered from CM users. + + +## Documentation + +* [Installation (Linux, Windows, MacOS)](install.md) + +TBD + + +## Contacts + +* Contact the CM/CMX author and maintainer for more details: [Grigori Fursin](https://cKnowledge.org/gfursin) diff --git a/cmx/install.md b/cmx/install.md new file mode 100644 index 0000000000..35db645f03 --- /dev/null +++ b/cmx/install.md @@ -0,0 +1,126 @@ +[ [Back to index](README.md) ] + +# CM Installation + +CM framework requires minimal dependencies to run on any platform: `python 3.7+, pip, venv, git, git-lfs, wget, curl`. + +By default, CM will pull Git repositories and cache installations and downloaded files in your `$HOME/CM` directory on Linux and MacOS +or `%userprofile%\CM` directory on Windows. +You can change it to any another directory using the `CM_REPOS` environment variable, for example `export CM_REPOS=/scratch/CM`. + +*Feel free to use the [online installation GUI](https://access.cknowledge.org/playground/?action=install-cmx)*. + + +## Ubuntu, Debian + +*We have successfully tested CMX with the following system dependencies on Ubuntu 20.x, 22.x , 24.x:* + +```bash +sudo apt update +sudo apt install python3 python3-pip python3-venv git git-lfs wget curl + +python3 -m venv cm +source cm/bin/activate + +pip install cmind +``` + +Note that you may need to restart your shell to update PATH to the "cmx" binary. + +Alternatively you can run + +```bash +source $HOME/.profile +``` + +You can now check that all system dependencies are installed using the following command: +```bash +cmx init +``` +You can also check that CMX core works using the following command: +```bash +cmx test core +``` + +## Red Hat + +*We have successfully tested CM on Red Hat 9 and CentOS 8* + +```bash +sudo dnf update + +sudo dnf install python3 python-pip git git-lfs wget curl + +python3 -m pip install cmind --user + +``` + + +## MacOS + +*Note that CM currently does not work with Python installed from the Apple Store. + Please install Python via brew as described below.* + +If `brew` package manager is not installed, please install it as follows (see details [here](https://brew.sh/)): +```bash +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" +``` + +Don't forget to add brew to PATH environment as described in the end. + +Then install python, pip, git and wget: + +```bash +brew install python3 git git-lfs wget curl + +python3 -m pip install cmind +``` + +*Sometimes python does not add `cm` and `cmr` binaries to the `PATH` environment variable. + You may need to find these files and add their path to `PATH` variable. + We plan to simplify this installation in the future.* + + +## Windows + +* Configure Windows 10+ to [support long paths](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later) from command line as admin: + + + + ```bash + reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /v LongPathsEnabled /t REG_DWORD /d 1 /f + ``` + + + +* Download and install Git from [git-for-windows.github.io](https://git-for-windows.github.io). + * Configure Git to accept long file names: `git config --system core.longpaths true` +* Download and install Python 3+ from [www.python.org/downloads/windows](https://www.python.org/downloads/windows). + * Don't forget to select option to add Python binaries to PATH environment! + * Configure Windows to accept long fie names during Python installation! + +* Install CM via PIP: + +```bash +python -m pip install cmind +``` + +*Note that we [have reports](https://github.com/mlcommons/ck/issues/844) + that CM does not work when Python was first installed from the Microsoft Store. + If CM fails to run, you can find a fix [here](https://stackoverflow.com/questions/57485491/python-python3-executes-in-command-prompt-but-does-not-run-correctly)*. + + +*We plan to provide a self-sustained package in the future to simplify CM installation on Windows.* + + + +# CM CLI testing + +If the installation is successful, you can run the CM CLI as follows: + +```bash +gfursin@cmind:~$ cmx + +cmx {action} {automation} {artifact(s)} {flags} @input.yaml @input.json +``` + diff --git a/dev/common-paper/README.md b/cmx4mlops/README.md similarity index 100% rename from dev/common-paper/README.md rename to cmx4mlops/README.md diff --git a/cmx4mlops/cmr.yaml b/cmx4mlops/cmr.yaml new file mode 100644 index 0000000000..13493b9e90 --- /dev/null +++ b/cmx4mlops/cmr.yaml @@ -0,0 +1,12 @@ +alias: cmx4mlops +uid: 428611a6db02407f + +git: true + +version: "0.5.1" + +author: "Grigori Fursin" + +install_python_requirements: false + +min_cm_version: "3.4.4" diff --git a/dev/reproducibility/README.md b/dev/reproducibility/README.md deleted file mode 100644 index a0990367ef..0000000000 --- a/dev/reproducibility/README.md +++ /dev/null @@ -1 +0,0 @@ -TBD diff --git a/docs/cm/README.md b/docs/cm/README.md new file mode 100644 index 0000000000..e9b96edd06 --- /dev/null +++ b/docs/cm/README.md @@ -0,0 +1,88 @@ +[![PyPI version](https://badge.fury.io/py/cmind.svg)](https://pepy.tech/project/cmind) +[![Python Version](https://img.shields.io/badge/python-3+-blue.svg)](https://github.com/mlcommons/ck/tree/master/cm/cmind) +[![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md) +[![Downloads](https://static.pepy.tech/badge/cmind)](https://pepy.tech/project/cmind) + +[![arXiv](https://img.shields.io/badge/arXiv-2406.16791-b31b1b.svg)](https://arxiv.org/abs/2406.16791) +[![CM test](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml) +[![CM script automation features test](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml) + +### About + +Collective Mind (CM) is a small, modular, cross-platform and decentralized workflow automation framework +with a human-friendly interface to make it easier to build, run, benchmark and optimize applications +across diverse models, data sets, software and hardware. + +CM is a part of [Collective Knowledge (CK)](https://github.com/mlcommons/ck) - +an educational community project to learn how to run emerging workloads +in the most efficient and cost-effective way across diverse +and continuously changing systems. + +CM includes a collection of portable, extensible and technology-agnostic automation recipes +with a common API and CLI (aka CM scripts) to unify and automate different steps +required to compose, run, benchmark and optimize complex ML/AI applications +on any platform with any software and hardware. + +CM scripts extend the concept of `cmake` with simple Python automations, native scripts +and JSON/YAML meta descriptions. They require Python 3.7+ with minimal dependencies and are +[continuously extended by the community and MLCommons members](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) +to run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux +and any other operating system, in a cloud or inside automatically generated containers +while keeping backward compatibility. + +CM scripts were originally developed based on the following requirements from the +[MLCommons members](https://mlcommons.org) +to help them automatically compose and optimize complex MLPerf benchmarks, applications and systems +across diverse and continuously changing models, data sets, software and hardware +from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors: +* must work out of the box with the default options and without the need to edit some paths, environment variables and configuration files; +* must be non-intrusive, easy to debug and must reuse existing + user scripts and automation tools (such as cmake, make, ML workflows, + python poetry and containers) rather than substituting them; +* must have a very simple and human-friendly command line with a Python API and minimal dependencies; +* must require minimal or zero learning curve by using plain Python, native scripts, environment variables + and simple JSON/YAML descriptions instead of inventing new workflow languages; +* must have the same interface to run all automations natively, in a cloud or inside containers. + +### Maintainers + +* [Grigori Fursin](https://cKnowledge.org/gfursin) + +### Resources + +* CM v2.x (2022-cur) (stable): [installation on Linux, Windows, MacOS](https://access.cknowledge.org/playground/?action=install) ; + [docs](https://docs.mlcommons.org/ck) ; [popular commands](https://github.com/mlcommons/ck/tree/master/cm/docs/demos/some-cm-commands.md) ; + [getting started guide](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md) +* CM v3.x aka CMX (2024-cur) (stable): [docs](https://github.com/orgs/mlcommons/projects/46) +* MLPerf inference benchmark automated via CM + * [Run MLPerf for submissions](https://docs.mlcommons.org/inference) + * [Run MLPerf at the Student Cluster Competition'24](https://docs.mlcommons.org/inference/benchmarks/text_to_image/reproducibility/scc24) +* Examples of modular containers and GitHub actions with CM commands: + * [GitHub action with CM commands to test MLPerf inference benchmark](https://github.com/mlcommons/inference/blob/master/.github/workflows/test-bert.yml) + * [Dockerfile to run MLPerf inference benchmark via CM](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-mlperf-inference/dockerfiles/bert-99.9/ubuntu_22.04_python_onnxruntime_cpu.Dockerfile) + +### License + +[Apache 2.0](LICENSE.md) + +### Citing CM project + +If you found CM automations useful, please cite this article: +[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. + +You can learn more about the motivation behind these projects from the following presentations: + +* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] +* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] +* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] + +### Acknowledgments + +The Collective Mind (CM) automation framework was originally +developed by [Grigori Fursin](https://cKnowledge.org/gfursin), +as a part of the [Collective Knowledge educational initiative](https://cKnowledge.org), +sponsored by [cTuning.org](https://cTuning.org) and [cKnowledge.org](https://cKnowledge.org), +and contributed to MLCommons for the benefit of all. +This open-source technology, including CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, and more, +is a collaborative community-driven project made possible by our +[amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! diff --git a/docs/history.md b/docs/history.md index b92250b128..39b353d616 100644 --- a/docs/history.md +++ b/docs/history.md @@ -24,8 +24,11 @@ and systems (CM4MLOps scripts and workflows)](https://github.com/mlcommons/cm4ml and helped establish the [MLCommons Task Force on Automation and Reproducibility](taskforce.md) co-led with Arjun Suresh. -We continue extending CM to support different MLCommons projects to modularize and unify benchmarking -of ML/AI systems as a collaborative engineering effort based on [user feedback](../CONTRIBUTING.md). - You can learn more about the CM concept and motivation from the [keynote at ACM REP'23](https://doi.org/10.5281/zenodo.8105339) and this [white paper](https://arxiv.org/abs/2406.16791). + +# CMX (the next generation of CM and CM4MLOps) + +Grigori started prototyping CMX during summer 2024 to simplify CM interfaces based on user feedback. +It is available as a part of the standard [CM package at PYPI](https://pypi.org/project/cmind). +Please reach out to learn more about our plans. diff --git a/docs/index.md b/docs/index.md index a972b6da3f..32d46ee883 120000 --- a/docs/index.md +++ b/docs/index.md @@ -1 +1 @@ -../cm/README.md \ No newline at end of file +../README.md \ No newline at end of file diff --git a/dev/meetings/20240731.md b/docs/meetings/20240731.md similarity index 100% rename from dev/meetings/20240731.md rename to docs/meetings/20240731.md diff --git a/dev/meetings/20240808.md b/docs/meetings/20240808.md similarity index 100% rename from dev/meetings/20240808.md rename to docs/meetings/20240808.md diff --git a/docs/tests.md b/docs/tests.md new file mode 100644 index 0000000000..53de4fb893 --- /dev/null +++ b/docs/tests.md @@ -0,0 +1 @@ +# TESTS diff --git a/mkdocs.yml b/mkdocs.yml index 5fe16a7ee0..0b2dbe75bd 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,4 +1,4 @@ -site_name: Collective Mind Documentation +site_name: Collective Knowledge Documentation repo_url: https://github.com/mlcommons/ck theme: name: material @@ -19,13 +19,15 @@ theme: - navigation.top - toc.follow nav: - - Install: - - install/index.md - - Documentation: - - CM CLI: specs/cm-cli.md + - HOME: index.md + - CM: + - Summary: cm/README.md - Getting Started: getting-started.md - - Release Notes: - - What's New: news.md + - CM installation: install/index.md + - CM CLI: specs/cm-cli.md + - Releases: + - CM/CMX releases: https://github.com/mlcommons/ck/releases + - CK/CM/CMX news: news.md markdown_extensions: - pymdownx.tasklist: diff --git a/requirements.txt b/requirements.txt index 4752077a68..f0e34ec5e0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -cmind>=2.0.1 +cmind>=3.3.3 pyyaml requests setuptools