{"id":23402996,"url":"https://github.com/kenza-ai/kenza","last_synced_at":"2025-10-14T04:28:12.292Z","repository":{"id":39104512,"uuid":"266864611","full_name":"Kenza-AI/kenza","owner":"Kenza-AI","description":"Open-Source Machine Learning Platform","archived":false,"fork":false,"pushed_at":"2023-01-05T11:31:51.000Z","size":5896,"stargazers_count":6,"open_issues_count":19,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-11T20:45:20.917Z","etag":null,"topics":["aws","aws-sagemaker","cd4ml","continuous-training","machine-learning","machine-learning-platform","ml-platform","open-source-ml-platform","sagemaker"],"latest_commit_sha":null,"homepage":"https://kenza-ai.github.io/kenza/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kenza-AI.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null}},"created_at":"2020-05-25T19:35:11.000Z","updated_at":"2024-10-08T14:37:47.000Z","dependencies_parsed_at":"2023-02-04T03:17:52.638Z","dependency_job_id":null,"html_url":"https://github.com/Kenza-AI/kenza","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Kenza-AI/kenza","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kenza-AI%2Fkenza","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kenza-AI%2Fkenza/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kenza-AI%2Fkenza/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kenza-AI%2Fkenza/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kenza-AI","download_url":"https://codeload.github.com/Kenza-AI/kenza/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kenza-AI%2Fkenza/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017946,"owners_count":26086212,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-sagemaker","cd4ml","continuous-training","machine-learning","machine-learning-platform","ml-platform","open-source-ml-platform","sagemaker"],"created_at":"2024-12-22T12:38:23.248Z","updated_at":"2025-10-14T04:28:12.264Z","avatar_url":"https://github.com/Kenza-AI.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Kenza\n\n*Kenza* is an open-source Machine Learning Platform.\n\nMore specifically, it is an open source cloud-native (moving from `Docker Swarm` to `Kubernetes` in 2020) system for Machine Learning Continuous Integration and Delivery (CD4ML) you can run in one command. It leverages containers and the cloud to provide basic mechanisms for training, tuning and deploying Machine Learning models.\n\n**What ML Engines does it support?**\n\n- AWS SageMaker\n- More to be added soon\n\n**What does it provide to Data Scientists?**\n\n- A web UI where you can track and compare your training and hyperparameter tuning jobs, and model deployments\n\n**How does a Data Scientist use it?**\n\n- It's very simple! You need to implement a `train` and `predict` functionality in Python, and define a YAML file. Example:\n\n        sagify:\n        \n        train:\n          input_s3_dir: s3://kenza-ds-demo/iris-data/iris.data\n          output_s3_dir: s3://kenza-training-models\n          hyperparameters_file: hyperparameters.json\n          ec2_type: ml.m5.large\n          volume_size: 50\n          timeout: 86400\n          metrics: Accuracy\n          \n          deploy:\n            instances_count: 1\n            ec2_type: ml.t2.medium\n            endpoint: some-endpoint\n\n**Why a Machine Learning team should care?**\n\n- Focus on Machine Learning, not ML Ops\n- Continuous and Reliable training,  hyperparameter tuning and deployment\n- Version control of ML models\n- Shorter time to put a model in production\n- No need to spend months to change your current ML codebase to meet the needs of this ML platform\n- Integrates easily with existing software engineering best practices\n- Less resources invested on ML Infrastructure Engineers\n\n**Why a Machine Learning Infrastructure team should care?**\n\n- Clean open-source ML Platform\n- Customize it as you wish \n- Shorter time to deliver \n- Easy to integrate with existing Engineering processes\n- Support of many ML Engines (at the moment only AWS SageMaker)\n\n**The best part is that you can continue using your favorite libraries!**\n\n**Go to https://kenza-ai.github.io/kenza/**, install Kenza and follow the steps in *Getting started for Data Scientists and ML Infrastructure Engineers*\n\n## Installation\n\nDownload the binary from the latest GitHub release:\n\n```sh\n# Linux\ncurl -L https://github.com/kenza-ai/kenza/releases/download/v0.0.1-alpha/kenza_0.0.1-alpha_linux_amd64 -o kenza\n```\n\n```sh\n# macOS\ncurl -L https://github.com/kenza-ai/kenza/releases/download/v0.0.1-alpha/kenza_0.0.1-alpha_darwin_amd64 -o kenza\n```\n\nMove it under a PATH directory, we prefer `/usr/local/bin`:\n```sh\nchmod +x kenza\nsudo mv ./kenza /usr/local/bin/kenza\n```\n\nEnsure you are on the expected version:\n\n```sh\nkenza info\n```\n\nYou should see output similar to the following:\n\n```sh\nKenza info\n\nVersion: v0.0.32\nBuilt:   2019-12-09T18:57:05Z\nCommit:  099415b5087d919d086b383da73afe1b99bf5k0a\n```\n\n## Getting Started\n\n#### Starting Kenza\nTo start (or restart *Kenza*) run:\n\n```sh\nkenza start\n```\n\n\u003e **Note:** The first run might take longer than subsequent runs due to the *Docker* images downloading for the first time.\n\n\u003e **Important:** The directory from which the `kenza` commands are run from is significant. `kenza start` creates a `kenza` directory in the directory the command was run from. If you run the command again in a different directory, a new `kenza` directory will be created there, essentially a separate `kenza` installation. \n\nAfter Kenza has started, it will attempt to navigate you to `http://localhost/#/signup` to create an account and get you started.\n\n#### Checking current service status\n\nYou can check the status of *Kenza* and its services with:\n\n```sh\nkenza status\n```\n\n\u003e If the output feels familiar, it's because *Kenza* is deployed as a *Docker stack*. Running `docker stack ps kenza` would generate the same output.\n\n\n### Scaling down/up\n\nKenza runs \"one job per worker\"; workers are ephemeral in nature and only handle one job before shutting down. To run more than one jobs in parallel, simply add more workers:\n\n```sh\nkenza scale worker=5\n```\n\n\u003e **Note:** Kenza workers do not need nearly as many resources as one may think (due to the nature of ML jobs) because the actual training takes place on the cloud. Kenza workers only clone the repos, prepare the job commands to be run and report on the status of the jobs as they progress through their lifetime.\n\n### Cleaning up\n\nYou can stop Kenza _without any data loss_ with:\n\n```sh\nkenza stop\n```\n\n\n### Updating Kenza\n\nTo update to the latest available version, run:\n\n```sh\nkenza update\n```\n\u003e **Note:** Currently, this only updates the Kenza executable, future work will stop Kenza, apply all necessary changes and restart the system to ensure all services are brought up to their latest versions, migrations are performed etc. For now, please run `kenza stop` before updating.\n\n## Running Kenza on the Cloud\n\n### Provisioning resources\n\n*Kenza* leverages [containers](https://docs.docker.com/machine/overview/) (currently orchestrated with _Docker Swarm_, moving to _Kubernetes_ in 2020) to run on the cloud. Before starting *Kenza*, the required resources (manager server(s) / instances, security groups etc) need to be provisioned first.\n\nEnsure your local *AWS* access levels (the profile or role you will be using when running `kenza provision` commands) meet the [*IAM* policy requirements for deploying a *Docker Machine*](https://github.com/docker/machine/issues/1655#issuecomment-409407523).\n\nTo provision a machine with the [default values](https://docs.docker.com/machine/drivers/aws/#options) on *AWS*, run:\n\n```sh\nkenza provision --driver amazonec2 --amazonec2-iam-instance-profile your-sagemaker-aware-intance-profile kenza-machine-1\n```\n\nAny other options you pass will be honored; all options are passed as-is to the corresponding `docker-machine` command. One would pass additional options to use a pre-existing _VPC_ or _Security Group_ to limit access to the instance to a specific office IP range for example. The full list of options available can be found [here](https://docs.docker.com/machine/drivers/aws/#options).\n\nYou can use any name for the `Docker Machine` (_kenza-machine-1_ in the example above) but the only `driver` supported for now is `\"amazonec2\"`.\n\n\u003e **Note**: It is highly recommended that the role assigned to the Kenza manager instance follows the [Principle of Least Privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) and only provides access to the services and resources that will actually be needed. To identify the exact permissions needed for your use cases use [this AWS reference](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html)\nspecific to _SageMaker_. If unsure, *AWS* has been aggressively adding [tools](https://aws.amazon.com/blogs/security/tag/access-advisor/) to make control of roles' more manageable. There are also open-source _Least Privilege Policy_ generators like _Saleforce's_ [Policy Centry](https://github.com/salesforce/policy_sentry/) you can use to ensure permissions are only as elevated as needed.\n\n\nRun `docker machine ls` to verify the machine you just created is available.\n\nYou can also check the [EC2 Dashboard](https://console.aws.amazon.com/ec2/home) on your *AWS* account for the various resources created (e.g. an instance and a key pair matching the \"name\" parameter provided earlier to the `provision` command, the \"docker-machine\" security group and others).\n\nTo deploy *Kenza* on the newly created resources, we first need to ensure the `Docker Machine` we just created is [*active*](https://docs.docker.com/machine/reference/active/). To do this, run (substituting if needed `kenza-machine-1` with the name you provided to the `provision` command):\n\n```sh\neval $(kenza env kenza-machine-1)\n```\n\nVerify `Docker` is now actually \"forwarding all calls\" to the remote machine:\n\n```sh\ndocker-machine active\n```\n\nWith the machine set up, all *Kenza* commands will now be run against the newly deployed infrastructure, not your local machine.\n\nTo start _Kenza_ on EC2, simply run (substituting if needed `kenza-machine-1` with the name you provided to the `provision` command):\n\n```sh\nkenza start --name kenza-machine-1 --github-secret webhooks-secret --apikey a-randomly-generated-key\n```\n\nAfter _Kenza_ starts, it will open your default browser to the URL / Public IP of the machine where the _Kenza_ web app can be reached.\n\nOnce launched, you can [associate your instance with a static IP or a domain name](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-ec2-instance.html).\n\n## Troubleshooting\n\n#### Getting detailed service execution details\n\nYou can observe detailed log output for a service with:\n\n```sh\nkenza logs service_name\n```\n\nYou can stop an individual service with:\n\n```sh\nkenza stop service_name\n```\n\nValid service names:\n- db\n- api\n- web\n- worker\n- pubsub\n- progress\n- scheduler\n\n#### Restarting Kenza\n\nRestarting _Kenza_ or _Docker_ can sometimes help when `Docker Swarm` seems to be \"stuck\".\n\nFor any other issue, please [raise an issue](https://github.com/Kenza-AI/kenza/issues/new).\n\n## Component Overview\n\nKenza is composed of the following components:\n\n- **API** - Service called by all other services, including the cli, to read / mutate Kenza related data (projects, jobs, schedules etc). \n\n\u003e Note for contributors: API is the only service with direct access / dependency to the Kenza data store(s). All other services *MUST* go through the API.\n\n- **Web** - *React.js* web application, the *Kenza UI*.\n\n- **Worker** - Worker nodes, the container tasks actually running the jobs. Workers are ephemeral and strictly process one job and one job only before shutting themselves down.\n\n- **Progress** - Listens for job updates published by the worker nodes and propagates them to the *API*.\n\n- **Scheduler** - Listens for job arrivals (on-demand, webhooks and scheduled jobs) and schedules them accordingly to be picked up by workers for processing.\n\n- **PubSub** - *RabbitMQ* exchanges and queues, used for async comms among services.\n\n- **DB** - The *kenza* data store (currently *Postgres*). It can be a *Postgres* container (default option, provided by *Kenza* as a container) or an external resource e.g. an *AWS RDS*, *Heroku* or on-prem installation.\n\n- **CLI** - The *Kenza* command line utility. Think *kubectl, systemctl*.\n\n*Kenza* currently supports *Docker Swarm* environments. Support for *Kubernetes* is being added in 2020.\n\n\n## Kenza UI\n\nThe Kenza web application is a [ReactJS](https://github.com/facebook/react) / [Redux](https://react-redux.js.org) Single Page Application (SPA). You can use the standard tooling e.g. React Tools ([Chrome](https://chrome.google.com/webstore/detail/react-developer-tools/fmkadmapgofadopljbjfkapdkoienihi?hl=en), [Firefox](https://addons.mozilla.org/en-US/firefox/addon/react-devtools/)) to troubleshoot  / report issues with specific browsers.\n\n## Note on tests (or lack thereof)\n_Kenza_ was originally built as a typical cloud based pipeline on _AWS_; tests will be being moved as they are getting adapted to the container-based world, probably starting with the ones that are the least impacted by the move e.g. the UI / web app.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkenza-ai%2Fkenza","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkenza-ai%2Fkenza","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkenza-ai%2Fkenza/lists"}