{"id":19935027,"url":"https://github.com/optimizely/kubeluigi","last_synced_at":"2025-05-03T12:30:55.867Z","repository":{"id":39973380,"uuid":"423510861","full_name":"optimizely/kubeluigi","owner":"optimizely","description":"Run Luigi tasks as Jobs on K8s - An updated Kubernetes contrib ","archived":false,"fork":false,"pushed_at":"2024-07-01T09:25:53.000Z","size":97,"stargazers_count":9,"open_issues_count":3,"forks_count":4,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-24T05:36:57.406Z","etag":null,"topics":["kubernetes","luigi"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/optimizely.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-01T15:10:41.000Z","updated_at":"2024-12-30T22:26:27.000Z","dependencies_parsed_at":"2024-06-28T21:21:48.701Z","dependency_job_id":"d40a77d0-b5fb-4b36-9456-f18d72a66093","html_url":"https://github.com/optimizely/kubeluigi","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/optimizely%2Fkubeluigi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/optimizely%2Fkubeluigi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/optimizely%2Fkubeluigi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/optimizely%2Fkubeluigi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/optimizely","download_url":"https://codeload.github.com/optimizely/kubeluigi/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252190628,"owners_count":21708914,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kubernetes","luigi"],"created_at":"2024-11-12T23:18:30.489Z","updated_at":"2025-05-03T12:30:55.338Z","avatar_url":"https://github.com/optimizely.png","language":"Python","readme":"# Kubeluigi\n\nKubeluigi lets you run Luigi tasks as Pods in a Kubernetes Cluster.  Kubeluigi is an update on Luigi's default `KubernetesTask`.\n\nImprovements over default luigi's contrib:\n\n- Currently default K8s task on Luigi is outdated, it does not handle some edge cases but most importantly it is not a priority in Luigi, this makes it slow to get changes merged.\n- Kubeluigi uses a non-deprecated kubernetes client library\n- Kubeluigi handle edge error cases not handled by the stdlib KubernetesTask\n- For most cases with only minor changes you should be able to unplug Luigi's default KubernetesJobTask and use Kubeluigi instead.\n- Realtime logging from Kubernetes Tasks\n- Logsfrom Kubernetes Tasks include a prefix which makes it easier to track the association of Job, Task, and Pod.\n\n## Installation\n`pip install kubeluigi`\n\n\n## Moving from Luigi's default contrib\n\nJust replace your imports of `KubernetesJobTask`: \n\n```diff\n- from luigi.contrib.kubernetes import KubernetesJobTask\n+ from kubeluigi import KubernetesJobTask\n```\n\n\n\n## Example\n\n```python\nfrom kubeluigi import KubernetesJobTask\n\nclass Task(KubernetesJobTask):\n\n    container_name = \"ubuntu:18.04\"\n\n    @property\n    def limits(self):\n        r = {\"requests\": {\n                    \"memory\": \"50M\",\n                    \"cpu\": \"1m\"\n                }}\n        return r\n\n    @property\n    def cmd(self):\n        return \"echo something \u0026\u0026 sleep 2 \u0026\u0026 echo after \u0026\u0026 sleep 5 \u0026\u0026 echo again\"\n\n    @property\n    def namespace(self):\n        return \"moussaka\"\n\n    @property\n    def labels(self):\n        return {\"my_label\": \"my_label_1\"}\n\n    def spec_schema(self):\n        print(self.container_name)\n        print(self.container_tag)\n        return {\n            \"containers\": [\n                {\n                    \"name\": self.name,\n                    \"image\": self.container_name + self.container_tag,\n                    \"args\": [\"/bin/bash\", \"-c\", self.cmd],\n                    \"imagePullPolicy\": \"Always\",\n                    \"resources\": self.limits,\n                    \"env\": [\n                        {\n                          \"name\": \"my_env\",\n                          \"value\": \"env\"\n                         }\n                    ]\n                }\n            ],\n        }\n\n    @property\n    def name(self):\n        return 'dummytask'\n\n```\n\n### Volumes\n\nYou can use volumes in the pods that run your tasks:\n\nSimple Ephemeral volume example:\n\n```python\nwith_ephemeral_volume200 = EphemeralVolume(size_in_gb=200)\nclass MyTask:\n  def __init__(self...):\n    ...\n    # you can find this volume in your containers under `/mnt/data/`\n    self.volumes =  [with_ephemeral_volume200]\n```\n\nBy leveraging volumes with cloud storage you can read and write data as if it existed locally. For example by mounting CSI drives your tasks can read inputs and write outputs to `/mnt/my_s3_bucket/`, this avoids complicated setups in which tasks have to know cloud specifics to read inputs and outputs\n\nWe provide a base class for Azure blob storage, this pressuposes you installed azure blob CSI driver in your AKS cluster.\n\n```python\nwith_azure_blob_volume = AzureBlobStorageVolume(storage_account=AZ_STORAGE_ACCOUNT,\n                                                storage_container=AZ_CONTAINER)\nclass MyTask:\n  def __init__(self...):\n    ...\n    # you can find this volume in your containers under `/mnt/{AZ_STORAGE_ACCOUNT}/{AZ_CONTAINER}`\n    # you can use this convention to have your containers inputs and outputs params\n    # read data from this mount point\n    self.volumes =  [with_azure_blob_volume]\n                                                \n```\n\n\n## Logs\n\nKubeluigi's task logs include Job, Task, and Pod identifiers: \n\n```\nINFO:luigi-interface:JOB: dummytask-20211028031913-a5eb1d7e634b43c8 POD: dummytask-20211028031913-a5eb1d7e634b43c8-9cnmt: some echo message\n```\n\n\n## Development\n\n- local setup: \n\n- install local package\n`pip install -e .`\n\n# testing\n`pip install nox`\n`nox`\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foptimizely%2Fkubeluigi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foptimizely%2Fkubeluigi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foptimizely%2Fkubeluigi/lists"}