{"id":13936752,"url":"https://github.com/ctuning/ck-tensorflow","last_synced_at":"2026-01-22T19:22:38.037Z","repository":{"id":86000387,"uuid":"65807155","full_name":"ctuning/ck-tensorflow","owner":"ctuning","description":"Collective Knowledge components for TensorFlow (code, data sets, models, packages, workflows):","archived":false,"fork":false,"pushed_at":"2021-07-10T12:36:16.000Z","size":8676,"stargazers_count":93,"open_issues_count":9,"forks_count":26,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-11-27T04:31:34.514Z","etag":null,"topics":["apis","automation","collective-knowledge","knowledge-management","metadata","package-management","pipelines","workflow-automation"],"latest_commit_sha":null,"homepage":"http://cKnowledge.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ctuning.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-08-16T09:33:40.000Z","updated_at":"2024-01-04T16:06:55.000Z","dependencies_parsed_at":"2023-03-13T08:09:00.115Z","dependency_job_id":null,"html_url":"https://github.com/ctuning/ck-tensorflow","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/ctuning/ck-tensorflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctuning%2Fck-tensorflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctuning%2Fck-tensorflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctuning%2Fck-tensorflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctuning%2Fck-tensorflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ctuning","download_url":"https://codeload.github.com/ctuning/ck-tensorflow/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ctuning%2Fck-tensorflow/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266031238,"owners_count":23866716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apis","automation","collective-knowledge","knowledge-management","metadata","package-management","pipelines","workflow-automation"],"created_at":"2024-08-07T23:02:57.869Z","updated_at":"2026-01-22T19:22:38.032Z","avatar_url":"https://github.com/ctuning.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Collective Knowledge components for TensorFlow\n\n**All CK components can be found at [cKnowledge.io](https://cKnowledge.io) and in [one GitHub repository](https://github.com/ctuning/ck-mlops)!**\n\n*This project is hosted by the [cTuning foundation](https://cTuning.org).*\n\n[![compatibility](https://github.com/ctuning/ck-guide-images/blob/master/ck-compatible.svg)](https://github.com/ctuning/ck)\n[![automation](https://github.com/ctuning/ck-guide-images/blob/master/ck-artifact-automated-and-reusable.svg)](http://cTuning.org/ae)\n[![workflow](https://github.com/ctuning/ck-guide-images/blob/master/ck-workflow.svg)](http://cKnowledge.org)\n\n[![DOI](https://zenodo.org/badge/65807155.svg)](https://zenodo.org/badge/latestdoi/65807155)\n[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)\n\nLinux/MacOS: [![Travis Build Status](https://travis-ci.org/ctuning/ck-tensorflow.svg?branch=master)](https://travis-ci.org/ctuning/ck-tensorflow)\nWindows: [![AppVeyor Build status](https://ci.appveyor.com/api/projects/status/github/ctuning/ck-tensorflow?branch=master\u0026svg=true)](https://ci.appveyor.com/project/ens-lg4/ck-tensorflow)\n\n# Introduction\n\nCK-TensorFlow repository provides automation components in the [CK format](https://github.com/ctuning/ck) \nfor tedious and repetitive tasks such as detecting and installing different TensorFlow versions, models and data sets \nacross diverse platforms and running AI/ML workflows in a unified way.\n\nNote that if some third-party automation fails or misses some functionality \n(software detection, package installation, bechmarking and autotuning workflow, etc),\nthe CK concept is to continuously and collaboratively improve such reusable components! \nPlease provide your feedback and report bugs via GitHub issues\nor get in touch with the community using this [public CK mailing list](https://groups.google.com/forum/#!forum/collective-knowledge)!\n\n\n# Installation\n\n## Prerequisites for Ubuntu\n\n* Python 2.x:\n```\n$ sudo apt-get install python-dev python-pip python-setuptools python-opencv git\n```\n\nor\n\n* Python 3.x:\n\n```\n$ sudo apt-get install python3-dev python3-pip python3-setuptools\n```\n\nNote that CK will automatically install the following dependencies into CK TF virtual space:\n```protobuf easydict joblib image wheel numpy scipy absl-py```\n\n## Optional dependencies depending on your use cases:\n\n* CUDA/cuDNN if you have [CUDA-enabled GPU](https://developer.nvidia.com/cuda-gpus)\n\n* Android NDK if you want to compile and run TF for Android devices\n\n## CK installation\n\nFollow these [instructions](https://github.com/ctuning/ck#installation) to install CK.\n\n## Installation of ck-tensorflow repository\n\n```\n$ ck pull repo:ck-tensorflow\n```\n\n\n\n# Basic usage\n\n## Example of a unified TensorFlow installation on Ubuntu or Windows via CK (pre-build versions)\n\n```\n$ ck install package:lib-tensorflow-1.8.0-cpu\n and/or (CK enables easy co-existance of different versions of tools\n$ ck install package:lib-tensorflow-1.8.0-cuda\n```\n\nCheck that TF is installed locally and registered in the CK:\n```\n$ ck show env --tags=lib,tensorflow\n```\n\nUse CK virtual environment to test it (similar to Python virtual env but for any binary package installed via CK):\n```\n$ ck virtual env --tags=lib,tensorflow\n```\n\nInstall other TF versions available in the CK:\n```\n$ ck install package --tags=lib,tensorflow\n```\n\n## Test unified image classification workflow via CK using above TF\n\n```\n$ ck run program:tensorflow --cmd_key=classify\n```\n\nNote, that you will be asked to select a jpeg image from available CK data sets.\nWe added standard demo images (cat.jpg, catgrey.jpg, fish-bike.jpg, computer_mouse.jpg)\nto the ['ctuning-datasets-min' repository](https://github.com/ctuning/ctuning-datasets-min).\nYou can list them via\n```\n $ ck pull repo:ctuning-datasets-min\n $ ck search dataset --tags=dnn\n```\n\n## Customize builds for different platforms\n\nYou can find more details about customized TensorFlow builds via CK for Android, Linux, Windows, \nRaspberry Pi, odroid, etc [here](https://github.com/ctuning/ck-tensorflow/wiki/Installation).\n\n\n\n\n# Benchmarking\n```\n $ ck run program:tensorflow (--env.BATCH_SIZE=10) (--env.NUM_BATCHES=5)\n```\n\nSelect one of the `test_cpu` and `test_cuda` commands;  select an available version of TensorFlow, \nif prompted (more than one choice); select an available benchmark, if prompted (more than one choice),\nand select TensorFlow model.\n\n## Crowd-benchmarking\n\nIt is now possible to participate in crowd-benchmarking of TensorFlow\n(early prototype):\n```\n$ ck crowdbench tensorflow --user={your email or ID to acknowledge contributions} (--env.BATCH_SIZE=128 --env.NUM_BATCHES=100)\n```\n\nYou can see continuously aggregated results in the \n[public Collective Knowledge repository](http://cknowledge.org/repo)\nunder 'crowd-benchmark TensorFlow library' scenario.\n\nNote, that this is an on-going, heavily evolving and long-term project\nto enable collaborative and systematic benchmarking\nand tuning of realistic workloads across diverse hardware \n([ARM TechCon'16 talk and demo](https://github.com/ctuning/ck/wiki/Demo-ARM-TechCon'16), \n[DATE'16](http://tinyurl.com/zyupd5v), [CPC'15](http://arxiv.org/abs/1506.06256)).\nWe also plan to add crowd-benchmarking and crowd-tuning of Caffe, TensorFlow \nand other DNN frameworks to our \n[Android application](http://cKnowledge.org/android-apps.html) \nsoon - please, stay tuned!\n\n## Unified, multi-dimensional and multi-objective autotuning\n\nIt is now possible to take advantage of our [universal multi-objective CK autotuner](https://github.com/ctuning/ck/wiki/Autotuning)\nto optimize TensorFlow. As a first simple example, we added batch size tuning via CK. You can invoke it as follows:\n\n```\n$ ck autotune tensorflow\n```\n\nAll results will be recorded in the local CK repository and \nyou will be given command lines to plot graphs or replay experiments such as:\n```\n$ ck plot graph:{experiment UID}\n$ ck replay experiment:{experiment UID} --point={specific optimization point}\n```\n\n## Collaborative and unified DNN optimization\n\nWe are now working to extend above autotuner and crowdsource optimization \nof the whole SW/HW/model/data set stack ([paper 1](https://scholar.google.com/citations?view_op=view_citation\u0026hl=en\u0026user=IwcnpkwAAAAJ\u0026citation_for_view=IwcnpkwAAAAJ:maZDTaKrznsC), \n[paper 2](https://arxiv.org/abs/1506.06256)).\n\nWe would like to thank the community for their interest and feedback about \nthis collaborative AI optimization approach powered by CK \nat [ARM TechCon'16](https://github.com/ctuning/ck/wiki/Demo-ARM-TechCon'16)\nand the Embedded Vision Summit'17 - so please stay tuned ;) !\n\n[![logo](http://cKnowledge.org/images/dividiti_arm_stand.jpg)](https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability)\n\n## Using other DNN via unified CK API\n\nCK allows us to unify AI interfaces while collaboratively optimizing underneath engines.\nFor example, we added similar support to install, use and evaluate [Caffe/Caffe2](https://github.com/ctuning/ck-caffe2),\n[CK-PyTorch](https://github.com/ctuning/ck-pytorch) and [MXNet](https://github.com/ctuning/ck-mxnet) via CK:\n\n```\n$ ck pull repo:ck-caffe2\n$ ck pull repo --url=https://github.com/dividiti/ck-caffe\n$ ck pull repo:ck-mxnet\n\n$ ck install package:lib-caffe-bvlc-master-cpu-universal --env.CAFFE_BUILD_PYTHON=ON\n$ ck install package:lib-caffe2-master-eigen-cpu-universal --env.CAFFE_BUILD_PYTHON=ON\n$ ck install package --tags=mxnet\n\n$ ck run program:caffe --cmd_key=classify\n$ ck run program:caffe2 --cmd_key=classify\n$ ck run program:mxnet --cmd_key=classify\n\n$ ck crowdbench caffe --env.BATCH_SIZE=5\n$ ck crowdbench caffe2 --env.BATCH_SIZE=5 --user=i_want_to_ack_my_contribution\n\n$ ck autotune caffe\n$ ck autotune caffe2\n```\n\n## Realistic/representative training sets\n\nWe provided an option in all our AI crowd-tuning tools to let the community report \nand share mispredictions (images, correct label and wrong misprediction) \nto gradually and collaboratively build realistic data/training sets:\n* [Public repository (see \"mispredictions and unexpected behavior)](http://cknowledge.org/repo/web.php?action=index\u0026module_uoa=wfe\u0026native_action=show\u0026native_module_uoa=program.optimization)\n* [Misclassified images via CK-based AI web-service](http://cknowledge.org/repo/web.php?action=index\u0026module_uoa=wfe\u0026native_action=show\u0026native_module_uoa=program.optimization)\n\n## Online demo of a unified CK-AI API \n\n* [Simple demo](http://cknowledge.org/repo/web.php?template=ck-ai-basic) to classify images with\ncontinuous optimization of DNN engines underneath, sharing of mispredictions and creation of a community training set;\nand to predict compiler optimizations based on program features.\n\n# Open R\u0026D challenges\n\nWe use crowd-benchmarking and crowd-tuning of such realistic workloads across diverse hardware for \n[open academic and industrial R\u0026D challenges](https://github.com/ctuning/ck/wiki/Research-and-development-challenges.mediawiki) - \njoin this community effort!\n\n# Publications\n\n* [CK publications](https://github.com/ctuning/ck/wiki/Publications)\n\n# Troublesooting\n\n* SqueezeDet demo currently work well with Python 3.5 and package:squeezedetmodel-squeezedet, so install it first:\n```\n$ ck install package:squeezedetmodel-squeezedet\n$ ck run program:squeezedet\n```\n\n# Feedback\n\nGet in touch with ck-tensorflow developers via CK mailing list: http://groups.google.com/group/collective-knowledge !\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctuning%2Fck-tensorflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fctuning%2Fck-tensorflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fctuning%2Fck-tensorflow/lists"}