{"id":15024087,"url":"https://github.com/sap/machine-learning-lab","last_synced_at":"2025-04-07T15:10:28.230Z","repository":{"id":37857000,"uuid":"290998762","full_name":"SAP/machine-learning-lab","owner":"SAP","description":"ML Lab enables teams to be more productive in delivering machine learning solutions for their products and datasets.","archived":false,"fork":false,"pushed_at":"2025-03-24T14:23:08.000Z","size":36965,"stargazers_count":70,"open_issues_count":65,"forks_count":27,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-04-05T13:13:36.720Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SAP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null}},"created_at":"2020-08-28T08:53:12.000Z","updated_at":"2025-03-21T03:28:03.000Z","dependencies_parsed_at":"2023-02-19T03:45:42.101Z","dependency_job_id":null,"html_url":"https://github.com/SAP/machine-learning-lab","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP%2Fmachine-learning-lab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP%2Fmachine-learning-lab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP%2Fmachine-learning-lab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP%2Fmachine-learning-lab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SAP","download_url":"https://codeload.github.com/SAP/machine-learning-lab/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247675607,"owners_count":20977378,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T19:59:47.465Z","updated_at":"2025-04-07T15:10:28.211Z","avatar_url":"https://github.com/SAP.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n    \u003ca href=\"https://github.com/sap/machine-learning-lab\" title=\"ML Lab Home\"\u003e\n    \u003cimg width=80% alt=\"\" src=\"./docs/images/lab-header.png\"\u003e \u003c/a\u003e\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cstrong\u003eEnd-to-end collaborative development platform to build and run machine learning solutions.\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/sap/machine-learning-lab/commits/\" title=\"Last Commit\"\u003e\u003cimg src=\"https://img.shields.io/github/last-commit/sap/machine-learning-lab/py-ml-lab\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/sap/machine-learning-lab/blob/master/LICENSE\" title=\"ML Lab License\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-Apache%202.0-green.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://api.reuse.software/info/github.com/sap/machine-learning-lab\" title=\"REUSE status\"\u003e\u003cimg src=\"https://api.reuse.software/badge/github.com/sap/machine-learning-lab\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#getting-started\"\u003eArchitecture\u003c/a\u003e •\n  \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e •\n  \u003ca href=\"#development\"\u003eDevelopment\u003c/a\u003e •\n\u003c/p\u003e\n\nML Lab is a centralized hub for development teams to seamlessly build, deploy, and operate machine learning solutions at scale. It is designed to cover the end-to-end machine learning lifecycle from data processing and experimentation to model training and deployment. It combines the libraries, languages, and tools data scientists love, with the infrastructure, services and workflows they need to deliver machine learning solutions into production.\n\n\u003e❗❗❗  \n\u003e This branch contains a new experimental version of ML Lab rewritten in python. The original java version can be found in the branch ml-lab-java.\n\n## Highlights\n\n- 🔐 Secure multi-user development platform for machine learning solutions.\n- 🛠 Workspace with integrated tooling (Jupyter, VS Code, SSH, VNC, Hardware Monitoring, ...)\n- 🗃️ Upload, manage, version, and share datasets \u0026 models.\n- 🎛 Deploy and operate machine learning solutions for productive usage.\n- 🐳 Deployable on a single-server via Docker or a server-cluster via Kubernetes.\n\n## Architecture\nMachine Learning Lab (ML Lab) builds on the [contaxy](https://github.com/ml-tooling/contaxy) server which provides a generic API for managing users and projects, storing files, deploying services and database access.\nUsing the extension mechanism of contaxy, ML Lab builds machine learning specific functionality on top.\nThese extensions are called ML Lab components and each have their own backend, frontend and docker image.\nThe ML Lab backend image leverages the contaxy library to combine all ML Lab components and adds a React frontend application.\nThe different parts of an ML Lab installation are shown in this diagram:\n![Lab Architecture](./docs/images/lab-architecture.png)\n\n\n## Getting Started\n\n### Local Docker Installation\n\nThe latest ML Lab docker image are pushed in the [GitHub container registry](https://github.com/orgs/SAP/packages?repo_name=machine-learning-lab).\nML Lab can be run locally using docker compose which will start the ML Lab backend, the Postgres and Minio Database and the ML Lab Components (Workspace Manager, etc.)\n```\ncd deployment/mllab-docker\ndocker pull ghcr.io/sap/machine-learning-lab/lab-backend\ndocker tag ghcr.io/sap/machine-learning-lab/lab-backend lab-backend\ndocker pull ghcr.io/sap/machine-learning-lab/lab-workspace-manager\ndocker tag ghcr.io/sap/machine-learning-lab/lab-workspace-manager lab-workspace-manager\ndocker pull mltooling/ml-workspace-minimal\ndocker-compose up\n# Visit http://localhost:30010/ and login with username and password \"admin\"\n# Run docker-compose up -d to start the application in the background\n```\n**Important**: The configurations in the docker-compose.yaml are not meant to be used for production\nas the JWT secret is the default one and the ports of all services are published, instead of only the ML Lab backend service.\nFor a list of all configurable environment variables, have a look at the [contaxy config file](https://github.com/ml-tooling/contaxy/blob/main/backend/src/contaxy/config.py#L32).\nAll fields of the `Settings` class represent an environment variable that can be set.\n\n### Kubernetes Installation\n\nTo install ML Lab on Kubernetes, the [helm chart](./deployment/mllab-kubernetes) can be used.\nAgain, the provided default values are not production ready and can be changed in the [values.yaml](./deployment/mllab-kubernetes/values.yaml) file.\n\n\n## Development\n\n### The build pipeline\nEach folder in this repository that contains a buildable part of ML Lab contains a build.py python script.\nThis script allows to lint (--check), build (--make) and test (--test) the contents of that folder: \n```\npython build.py --check --make --test\n```\nThe build.py script in the root folder calls the build.py scripts of all sub folders and thereby builds the entire ML Lab project.\nIt is possible to exclude specific folders from the build pipeline:\n```\npython build.py --skip-path webapp --skip-path components/lab-workspace-manager\n```\nThe build.py script always performs a full rebuild which leads to a long build time.\nThis is great for a reproducible build in the CI/CD pipeline build but slows down development.\nTherefore, it makes sense to use the options below during developing ML Lab.\n\n### Build ML Lab docker image manually\nThis paragraph describes the steps required for building the ML Lab backend image. \nThese steps are basically the same as the ones executed automatically by the build.py script.\nHowever, during development only some steps need to be rerun depending on what files are changed.\n\nBuild the webapp bundle (only required when changing the webapp):\n```\ncd webapp\nyarn install        # Only rerun if dependencies changed \nyarn run build\ncd ..\n```\n\nBuild the ML Lab backend image:\n``\ndocker build -t lab-backend .\n``\n\nThe same commands can be used for building any of the [components](components).\n\n### Developing the ML Lab Web Application\nThe ML Lab web app is build with JavaScript and React. It communicates with the ML Lab backend (via the contaxy API) and provides the overall UI structure of ML Lab. \nThe ML Lab components are integrated into this UI via iframes.\nDuring the build process, the web app is compiled into a minified JavaScript bundle and is then later served by the ML Lab backend.\nFor development, the web app can be started locally and connect to any other local or remote ML Lab instance.\nFor more information, see the [webapp folder](./webapp).\n\n### Developing Contaxy Core Features\nSometimes it is necessary to add new features to the contaxy library directly instead of building a contaxy extension (ML Lab component).\nDetailed information on how to develop contaxy can be found in the [contaxy repository](https://github.com/ml-tooling/contaxy).\nTo make the development of contaxy features in conjunction with ML Lab development easier, the contaxy repo is included as a git submodule.\nRun the following command to download the contaxy source into the [contaxy folder](./contaxy)\n```\ngit submodule update --init\n```\nTo use this local contaxy code in the ML Lab backend image instead of the version [released on PyPi](https://pypi.org/project/contaxy/), uncomment the [Dockerfile](./Dockerfile) lines 95-98.\nThe following `.env` file can be used to connect a locally started contaxy server with the Postgres and Minio DB started by the docker compose script in this repository:\n```\nPOSTGRES_CONNECTION_URI=postgresql://postgres:postgres-password-change-me@127.0.0.1:30011/postgres\nS3_ENDPOINT=127.0.0.1:30013\nS3_ACCESS_KEY=minioadmin\nS3_SECRET_KEY=minio-password-change-me\nBACKEND_CORS_ORIGINS=http://localhost:3000\nSYSTEM_NAMESPACE=pylab\n```\n\n### Developing an ML Lab Component\nAn ML Lab component is its own separate application with a frontend and backend packaged into a docker image.\nIt is started alongside the ML Lab Backend container which discovers the component via labels set on the container.\nThe component can extend the ML Lab API and its UI is integrated in the main ML Lab UI with an iframe.\nFor an example of such an ML Lab component, refer to the [ML Lab Workspace Manager](./components/lab-workspace-manager). It contains further information on how the extension is structured and how to develop it locally.\nIf you want to develop a new ML Lab component, you can copy the [template component](./components/template) and follow the instruction in that folder.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsap%2Fmachine-learning-lab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsap%2Fmachine-learning-lab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsap%2Fmachine-learning-lab/lists"}