{"id":13910828,"url":"https://github.com/mlcommons/ck","last_synced_at":"2025-07-18T10:31:47.532Z","repository":{"id":22882399,"uuid":"26230485","full_name":"mlcommons/ck","owner":"mlcommons","description":"Collective Knowledge (CK), Collective Mind (CM/CMX) and MLPerf automations: community-driven projects to facilitate collaborative and reproducible research and to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf methodology and benchmarks","archived":false,"fork":false,"pushed_at":"2025-06-19T17:37:00.000Z","size":36843,"stargazers_count":628,"open_issues_count":9,"forks_count":118,"subscribers_count":49,"default_branch":"master","last_synced_at":"2025-07-02T02:39:33.619Z","etag":null,"topics":["automation","best-practices","ck","cknowledge","cm","cmind","cmx","collaboration","ctuning","education","metadata","mlops","mlperf","mlperf-automations","mlperf-inference","modularity","optimization","portability","reusability","workflows"],"latest_commit_sha":null,"homepage":"https://access.cKnowledge.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlcommons.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.CM.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"citation.bib","codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2014-11-05T17:14:43.000Z","updated_at":"2025-06-29T17:13:24.000Z","dependencies_parsed_at":"2023-01-14T13:30:39.352Z","dependency_job_id":"44be28a2-0b17-4ca4-b57a-7ad0ea8583cf","html_url":"https://github.com/mlcommons/ck","commit_stats":{"total_commits":3737,"total_committers":32,"mean_commits":116.78125,"dds":0.5806796895905807,"last_synced_commit":"ffcf31d18b14f55b8f20b7a9d078ebd61023ca7e"},"previous_names":["ctuning/ck"],"tags_count":140,"template":false,"template_full_name":null,"purl":"pkg:github/mlcommons/ck","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fck","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fck/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fck/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fck/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlcommons","download_url":"https://codeload.github.com/mlcommons/ck/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fck/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265742238,"owners_count":23820813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","best-practices","ck","cknowledge","cm","cmind","cmx","collaboration","ctuning","education","metadata","mlops","mlperf","mlperf-automations","mlperf-inference","modularity","optimization","portability","reusability","workflows"],"created_at":"2024-08-07T00:01:47.491Z","updated_at":"2025-07-18T10:31:47.521Z","avatar_url":"https://github.com/mlcommons.png","language":"Python","readme":"[![PyPI version](https://badge.fury.io/py/cmind.svg)](https://pepy.tech/project/cmind)\n[![Python Version](https://img.shields.io/badge/python-3+-blue.svg)](https://github.com/mlcommons/ck/tree/master/cm/cmind)\n[![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md)\n[![Downloads](https://static.pepy.tech/badge/cmind)](https://pepy.tech/project/cmind)\n[![arXiv](https://img.shields.io/badge/arXiv-2406.16791-b31b1b.svg)](https://arxiv.org/abs/2406.16791)\n\n[![CMX image classification test](https://github.com/mlcommons/ck/actions/workflows/test-cmx-image-classification-onnx.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cmx-image-classification-onnx.yml)\n[![CMX MLPerf inference resnet-50 test](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-resnet50.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-resnet50.yml)\n[![CMX MLPerf inference r-GAT test](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-rgat.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-rgat.yml)\n[![CMX MLPerf inference BERT deepsparse test](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-bert-deepsparse-tf-onnxruntime-pytorch.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cmx-mlperf-inference-bert-deepsparse-tf-onnxruntime-pytorch.yml)\n\n## Collective Knowledge project (CK)\n\n[Collective Knowledge (CK)](https://cKnowledge.org) \nis a community-driven project dedicated to supporting open science, enhancing reproducible research, \nand fostering collaborative learning on how to run AI, ML, and other emerging workloads in the most efficient and cost-effective way\nacross diverse models, data sets, software and hardware:\n[ [white paper](https://arxiv.org/abs/2406.16791) ].\n\nIt includes the following sub-projects.\n\n### Collective Mind project (MLCommons CM)\n\nThe [Collective Mind automation framework (CM)](https://github.com/mlcommons/ck/tree/master/cm)\nwas developed to support open science and facilitate\ncollaborative, reproducible, and reusable research, development, \nand experimentation based on [FAIR principles](https://en.wikipedia.org/wiki/FAIR_data).\n\nIt helps users non-intrusively convert their software projects \ninto file-based repositories of portable and reusable artifacts \n(code, data, models, scripts) with extensible metadata\nand reusable automations, a unified command-line interface, \nand a simple Python API.\n\nSuch artifacts can be easily chained together into portable \nand technology-agnostic automation workflows, enabling users to \nrerun, reproduce, and reuse complex experimental setups across diverse and rapidly evolving models, datasets,\nsoftware, and hardware. \n\nFor example, CM helps to modularize, automate and customize MLPerf benchmarks.\n\n#### Legacy CM API and CLI (2021-2024)\n\nSee the [project page](https://github.com/mlcommons/ck/blob/master/cm/README.CM.md) for more details.\n\nLegacy and simplified CM and MLPerf automations were donated to MLCommons by Grigori Fursin, the cTuning foundation and OctoML.\nThey are now supported by the MLCommons Infra WG (MLCFlow, MLC scripts, mlcr ...).\n\n#### New CM API and CLI (CMX, 2025+)\n\n[Collective Mind eXtension or Common Metadata eXchange (CMX)](https://github.com/mlcommons/ck/tree/master/cmx) \nis the next evolution of the [Collective Mind automation framework (MLCommons CM)](https://github.com/mlcommons/ck/tree/master/cm) \ndesigned to enhance simplicity, flexibility, and extensibility of automations \nbased on user feedback. It is backwards compatible with CM, released along with CM \nin the [cmind package](https://pypi.org/project/cmind/) and can serve as drop-in replacement \nfor CM and legacy MLPerf automations while providing a simpler and more robust interface.\n\nSee the [project page](https://github.com/mlcommons/ck/tree/master/cmx) \nand [CMX4MLOps automations](https://github.com/mlcommons/ck/tree/master/cmx4mlops) for more details.\n\n### MLOps and MLPerf automations\n\nWe have developed a collection of portable, extensible and technology-agnostic automation recipes\nwith a common CLI and Python API (CM scripts) to unify and automate \nall the manual steps required to compose, run, benchmark and optimize complex ML/AI applications \non diverse platforms with any software and hardware. \n\nThe two key automations are *script* and *cache*:\nsee [online catalog at CK playground](https://access.cknowledge.org/playground/?action=scripts),\n[online MLCommons catalog](https://docs.mlcommons.org/cm4mlops/scripts).\n\nCM scripts extend the concept of `cmake` with simple Python automations, native scripts\nand JSON/YAML meta descriptions. They require Python 3.8+ with minimal dependencies and are \n[continuously extended by the community and MLCommons members](https://github.com/mlcommons/ck/blob/master/CONTRIBUTORS.md)\nto run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux\nand any other operating system, in a cloud or inside automatically generated containers\nwhile keeping backward compatibility.\n\nSee the [online MLPerf documentation](https://docs.mlcommons.org/inference) \nat MLCommons to run MLPerf inference benchmarks across diverse systems using CMX.\nJust install `pip install cmx4mlperf` and substitute the following commands and flags:\n* `cm` -\u003e `cmx`\n* `mlc` -\u003e `cmlc`\n* `mlcr` -\u003e `cmlcr`\n* `-v` -\u003e `--v`\n\n### Collective Knowledge Playground\n\n[Collective Knowledge Playground](https://access.cKnowledge.org) - \na unified and open-source platform designed to [index all CM/CMX automations](https://access.cknowledge.org/playground/?action=scripts) \nsimilar to PYPI and assist users in preparing CM/CMX commands to:\n\n* aggregate, process, visualize, and compare [MLPerf benchmarking results](https://access.cknowledge.org/playground/?action=experiments) for AI and ML systems\n* [run MLPerf benchmarks](https://access.cknowledge.org/playground/?action=howtorun)\n* organize [open and reproducible optimization challenges and tournaments](https://access.cknowledge.org/playground/?action=challenges). \n\n### Artifact Evaluation and Reproducibility Initiatives\n\n[Artifact Evaluation automation](https://cTuning.org/ae) - a community-driven initiative \nleveraging CK, CM and CMX to automate artifact evaluation \nand support reproducibility efforts at ML and systems conferences.\n\n\n## Legacy projects \n\n* [CM-MLOps](https://github.com/mlcommons/ck/tree/master/cm-mlops) (2021)\n* [CM4MLOps](https://github.com/mlcommons/cm4mlops) (2022-2024)\n* [CK automation framework v1 and v2](https://github.com/mlcommons/ck/tree/master/ck)\n\n\n## License\n\n[Apache 2.0](LICENSE.md)\n\n## Copyright\n\nCopyright (c) 2021-2025 MLCommons\n\nGrigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone.\n\nCopyright (c) 2014-2021 cTuning foundation\n\n## Author\n\n* [Grigori Fursin](https://cKnowledge.org/gfursin)\n\n## Maintainers\n\n* Legacy CM, CM4MLOps and MLPerf automations: [MLCommons infra WG](https://mlcommons.org)\n* CMX (the next generation of CM since 2025): [Grigori Fursin](https://cKnowledge.org/gfursin)\n\n## Concepts\n\nTo learn more about the motivation behind this project, please explore the following articles and presentations:\n\n* HPCA'25 article \"MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI\": [ [Arxiv](https://arxiv.org/abs/2410.12032) ], [ [tutorial to reproduce results using CM/CMX](https://github.com/aryatschand/MLPerf-Power-HPCA-2025/blob/main/measurement_tutorial.md) ]\n* NeuralMagic's vLLM MLPerf inference 4.1 submission automated by CM: [ [README] ](https://github.com/mlcommons/inference_results_v4.1/blob/main/open/NeuralMagic/measurements/4xH100-SXM-80GB_vLLM_FP8-reference-cpu-pytorch-v2.3.1-default_config/llama2-70b-99/server/README.md)\n* SDXL MLPerf inference 4.1 submission automated by CM: [ [README] ](https://github.com/mlcommons/inference_results_v4.1/tree/main/open/CTuning/code/stable-diffusion-xl)\n* \"Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments\": [ [ArXiv](https://arxiv.org/abs/2406.16791) ]\n* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] \n* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]\n* Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ]\n\n### Acknowledgments\n\nThis open-source project was created by [Grigori Fursin](https://cKnowledge.org/gfursin)\nand sponsored by cTuning.org, OctoAI and HiPEAC.\nGrigori donated this project to MLCommons to modularize and automate MLPerf benchmarks,\nbenefit the community, and foster its development as a collaborative, community-driven effort.\n\nWe thank [MLCommons](https://mlcommons.org), [FlexAI](https://flex.ai) \nand [cTuning](https://cTuning.org) for supporting this project,\nas well as our dedicated [volunteers and collaborators](https://github.com/mlcommons/ck/blob/master/CONTRIBUTORS.md)\nfor their feedback and contributions!\n\nIf you found the CM, CMX and MLPerf automations helpful, kindly reference this article:\n[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ].\n\nYou are welcome to contact the [author](https://cKnowledge.org/gfursin) to discuss long-term plans and potential collaboration.\n","funding_links":[],"categories":["Python","Framework"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlcommons%2Fck","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlcommons%2Fck","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlcommons%2Fck/lists"}