{"id":13677028,"url":"https://github.com/m3dev/kannon","last_synced_at":"2026-01-16T06:35:44.397Z","repository":{"id":142580491,"uuid":"612086366","full_name":"m3dev/kannon","owner":"m3dev","description":"Kannon is a wrapper for the gokart library that allows gokart tasks to be easily executed in a distributed and parallel manner on multiple kubernetes jobs.","archived":false,"fork":false,"pushed_at":"2025-01-17T00:49:50.000Z","size":415,"stargazers_count":26,"open_issues_count":5,"forks_count":3,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-12-22T11:53:18.268Z","etag":null,"topics":["gokart","kubernetes","machine-learning","mlops","task-pipeline"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m3dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-10T07:01:48.000Z","updated_at":"2025-11-21T06:54:48.000Z","dependencies_parsed_at":"2024-01-14T15:01:01.369Z","dependency_job_id":"ac1b2d33-775e-4798-8793-1eaaa108ae40","html_url":"https://github.com/m3dev/kannon","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/m3dev/kannon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m3dev%2Fkannon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m3dev%2Fkannon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m3dev%2Fkannon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m3dev%2Fkannon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m3dev","download_url":"https://codeload.github.com/m3dev/kannon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m3dev%2Fkannon/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28477881,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T06:30:42.265Z","status":"ssl_error","status_checked_at":"2026-01-16T06:30:16.248Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gokart","kubernetes","machine-learning","mlops","task-pipeline"],"created_at":"2024-08-02T13:00:35.892Z","updated_at":"2026-01-16T06:35:44.386Z","avatar_url":"https://github.com/m3dev.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# kannon\n\nKannon is a wrapper for the [gokart](https://github.com/m3dev/gokart) library that allows gokart tasks to be easily executed in a distributed and parallel manner on multiple [kubernetes](https://kubernetes.io/) jobs.\n\n# Install\nKannon can be installed via `pip`.\n\n```bash\npip install kannon\n```\n\n# [Quick Starter](./example/README.md)\nAn easy and self-contained tutorial can be found in [here](./example/README.md)!\n\n# Usage\nIt is required for users to prepare following two scripts and copy them into a docker container:\n- A script to start task pipeline on master job.\n- A script for child jobs to run assigned tasks.\n\nEasy and self-contained quick starter will be available soon!\n\n## Give custom permissions to the service account\nIf you have a service account `my-batch` within a namespace `mynamespace`, the following permissions on `jobs` and `jobs/status` should be added.\n\n```yml\n# role.yaml\napiVersion: rbac.authorization.k8s.io/v1\nkind: Role\nmetadata:\n  namespace: mynamespace\n  name: job-admin\nrules:\n  - apiGroups: [\"batch\"]\n    resources: [\"jobs\"]\n    verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n  - apiGroups: [\"batch\"]\n    resources: [\"jobs/status\"]\n    verbs: [\"get\", \"list\", \"watch\"]\n\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBinding\nmetadata:\n  name: job-admin-binding\n  namespace: mynamespace\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: Role\n  name: job-admin\nsubjects:\n  - namespace: mynamespace\n    kind: ServiceAccount\n    name: my-batch\n```\n\n## How to define tasks\nThe way to define tasks is almost the same as for gokart.\nAll you have to do is replace the parent class of the task class you want to distribute with `kannon.TaskOnBullet` from `gokart.TaskOnKart`.\n\nTo get detailed explanations on how to define a `gokart.TaskOnKart`, please refer to [gokart docs](https://gokart.readthedocs.io/en/latest/task_on_kart.html).\n\nHere is an example to define the following task pipeline with gokart and kannon.\n\n\u003cdiv align=\"center\"\u003e\n\n![](./image/readme_example_tasks.jpeg)\n\n\u003c/div\u003e\n\nTwo parts (Task B0 + Task C0) and (Task B1 + Task C1) can be run in a distributed and parallel manner.\n\n```python\nimport gokart\nimport luigi\nimport kannon\n\n\nclass TaskA(gokart.TaskOnKart):\n    param = luigi.Parameter()\n\n    def run(self):\n        self.dump(\"A\")\n\n# TODO: Change gokart.TaskOnKart -\u003e kannon.TaskOnBullet\nclass TaskB(kannon.TaskOnBullet):\n    param = luigi.Parameter()\n    parent = gokart.TaskInstanceParameter()\n    \n    def requires(self):\n        return {\"parent\": self.parent}\n\n    def run(self):\n        self.dump(\"B\")\n\n# TODO: Change gokart.TaskOnKart -\u003e kannon.TaskOnBullet\nclass TaskC(kannon.TaskOnBullet):\n    param = luigi.Parameter()\n    parent = gokart.TaskInstanceParameter()\n    \n    def requires(self):\n        return {\"parent\": self.parent}\n\n    def run(self):\n        self.dump(\"C\")\n\n\nclass TaskD(gokart.TaskOnKart):\n    param = luigi.Parameter()\n    parent_0 = gokart.TaskInstanceParameter()\n    parent_1 = gokart.TaskInstanceParameter()\n    parent_2 = gokart.TaskInstanceParameter()\n    \n    def requires(self):\n        return {\"parent_0\": self.parent_0, \"parent_1\": self.parent_1, \"parent_2\": self.parent_2}\n\n    def run(self):\n        self.dump(\"D\")\n\ntask_a = TaskA(param=\"a\")\n# b0 and b1 are executed in parallel\ntask_b0 = TaskB(param=\"b0\", parent=task_a)\ntask_b1 = TaskB(param=\"b1\", parent=task_a)\n# c0 and c1 are executed in parallel\ntask_c0 = TaskC(param=\"c0\", parent=task_b0)\ntask_c1 = TaskC(param=\"c1\", parent=task_b1)\ntask_d = TaskD(param=\"d\", parent_0=task_c0, parent_1=task_c1, parent_2=task_a)\n```\n\n\n## A script to run job on master job\nSteps:\n1. Import module where `gokart.TaskOnKart` and `kannon.TaskOnBullet` classes are defined.\n2. Load luigi and k8s configs.\n3. Create a task instance.\n4. Create a template job for child jobs.\n5. Run `Kannon.build`.\n\n```python\n\"\"\" This script runs on master job. \"\"\"\nimport logging\n\nimport gokart\nimport luigi\nfrom kubernetes import config, client\nimport fire\n\n# TODO: Import task definition here!\nimport example_tasks\nfrom kannon import Kannon\n\nlogging.basicConfig(level=logging.INFO)\n\n\ndef main(\n    container_name: str,\n    image_name: str,\n):\n    # TODO: Load luigi config here!\n    luigi.configuration.LuigiConfigParser.add_config_path(\"./conf/base.ini\")\n\n    # TODO: Load kube config here!\n    config.load_incluster_config()\n    v1 = client.BatchV1Api()\n    # TODO: Create task instance here!\n    task_root = [CREATE TASK INSTANCE HERE]\n    \n    # TODO: create a template for child job from yaml file or using k8s API.\n    template_job = utils.create_from_yaml(v1, PATH_TO_CHILD_JOB_YAML)\n    # TODO: Run Kannon.build!\n    Kannon(\n        api_instance=v1,\n        template_job=template_job,\n        job_prefix=\"template-child-job\",\n        path_child_script=\"./run_child.py\",\n    ).build(task_root)\n\n\nif __name__ == \"__main__\":\n    Fire.fire(main)\n```\n\nNote that `job.spec.template.containers[i].command` and `job.metadata.name` are replaced within `Kannon.build`. \n\n## A script for child jobs to run assigned tasks\nFor now, it is required for users to prepare the following script. In the future release, it will not be required.\n\nSteps:\n1. Import module where `gokart.TaskOnKart` and `kannon.TaskOnBullet` classes are defined.\n2. Load luigi config.\n3. Parse a serialized task instance.\n4. Run `gokart.build`.\n\n```python\n\"\"\" This script requires to be defined by user. \"\"\"\nimport gokart\nimport luigi\nimport logging\n\n# TODO: Import task definitions here!\nimport example_tasks\n\nimport fire\n\nlogging.basicConfig(level=logging.INFO)\n\n\ndef main(serialized_task: str):\n    # TODO: Load luigi config here!\n    luigi.configuration.LuigiConfigParser.add_config_path(\"./conf/base.ini\")\n    \n    # TODO: Parse a serialized gokart.TaskOnKart here!\n    task: gokart.TaskOnKart = gokart.TaskInstanceParameter().parse(serialized_task)\n    # TODO: Run gokart.build!\n    gokart.build(task)\n\n\nif __name__ == \"__main__\":\n    fire.Fire(main)\n```\n\n# Thanks\n\nKannon is a wrapper for gokart. Thanks to gokart and dependent projects!\n\n- [gokart](https://github.com/m3dev/gokart)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm3dev%2Fkannon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm3dev%2Fkannon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm3dev%2Fkannon/lists"}