{"id":26851234,"url":"https://github.com/deeeelin/ssis-dispatcher","last_synced_at":"2025-09-01T08:11:40.267Z","repository":{"id":286937692,"uuid":"958565284","full_name":"deeeelin/SSIS-Dispatcher","owner":"deeeelin","description":"A Kubernetes serving manager for machine learning inference system enabled with NVIDIA MIG/MPS GPU-Sharing support","archived":false,"fork":false,"pushed_at":"2025-07-02T16:04:54.000Z","size":122,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-02T16:38:15.680Z","etag":null,"topics":["docker","go","golang","k8s","knative","knative-serving","kubernetes","mlops","mps","multi-instance-gpu","nvidia","nvidia-gpu"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deeeelin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-01T12:05:22.000Z","updated_at":"2025-07-02T16:04:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"040428d4-84e2-4e18-bf8a-31b6df34acfe","html_url":"https://github.com/deeeelin/SSIS-Dispatcher","commit_stats":null,"previous_names":["deeeelin/ssis-dispatcher"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/deeeelin/SSIS-Dispatcher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeeelin%2FSSIS-Dispatcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeeelin%2FSSIS-Dispatcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeeelin%2FSSIS-Dispatcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeeelin%2FSSIS-Dispatcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deeeelin","download_url":"https://codeload.github.com/deeeelin/SSIS-Dispatcher/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeeelin%2FSSIS-Dispatcher/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273093648,"owners_count":25044438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","go","golang","k8s","knative","knative-serving","kubernetes","mlops","mps","multi-instance-gpu","nvidia","nvidia-gpu"],"created_at":"2025-03-30T22:19:09.357Z","updated_at":"2025-09-01T08:11:40.259Z","avatar_url":"https://github.com/deeeelin.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SSIS-Dispatcher\n\n## About\nThe SSIS-Dispatcher project is a subproject branched from the SSIS(Scalable Serving Inference System for Language Models with NVIDIA MIG) project. It is a served as a serving manager component in the system. SSIS-Dispatcher is capable of receiving model inference requests and luanching inference pod under [Knative](https://knative.dev/docs/) framework while leveraging GPU sharing features supported my Nvidia [Multi-Instance GPU(MIG)](https://www.nvidia.com/en-us/technologies/multi-instance-gpu/) or [Multi-Process Service (MPS)](https://docs.nvidia.com/deploy/mps/index.html), which allows finegrained unitlization of GPU resources, enhancing system efficiency.\n* Check out the [K-SSIS Repository](https://github.com/mike911209/K-SISS), for additional autoscaler or performance monitor support.\n\n## Getting Start\n### Prerequisite\n* Requires a kubernetes cluster with version \u003e 1.28\n* This demo project default runs all knative service, pods  on `nthulab` namespace\n* You should have MIG or MPS kubernetes resource registered on your cluster\n    * For MIG environment setup, reference the [GPU operator documentation](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html)\n    * For MPS setup, recommended [Nebuly GPU device plugin](https://github.com/nebuly-ai/k8s-device-plugin)\n\n### 1. Setup Knative and Kourier Ingress/ Load Balancer\n\n* Run `make setup_knative`\n* `k get po -n kourier-system`, check if kourier gateway is running\n* `k get svc -n kourier-system`, check if kourier svc and kourier-internal service is established\n* You can use `curl \u003ckourier service external ip\u003e` to test kourier external gateway or run a pod on cluster that runs `curl http://kourier-internal.kourier-system.svc.cluster.local` to check the in-cluster gateway is operating\n* Use `kn service list` and find the url for the dispatcher, ex: `http://dispatcher.nthulab.192.168.1.10.sslip.io`\n\n### 2. Build Your Own Dispatcher Image (Optional)\n\n* If you want to build your own dispatcher image, Run `make build`\n\n### 3. Deploy dispatcher\n\n* Run `make deploy` to deploy your own dispatcher image, run `kubectl apply -f https://raw.githubusercontent.com/deeeelin/SSIS-Dispatcher/main-deployment/configuration.yaml` to deploy prebuilt image from main branch\n\n### 4. Configure Dispatcher and Restart pod\n\n* Run `kubectl edit configmap dispatcher-config\n* Edit data section to set service namespace, inference image and GPU resource names that applies to your system environment\n    * The MIG resource defined in node may have the example resource name format below:\n    ```\n    nvidia.com/mig-1g.5gb\n    nvidia.com/mig-2g.10gb\n    nvidia.com/mig-3g.20gb\n    nvidia.com/mig-4g.20gb\n    nvidia.com/mig-7g.40gb\n    ```\n    * The nebuly MPS resource defined in node may have the example the resource name format below:\n    ```\n    nvidia.com/gpu-1gb\n    nvidia.com/gpu-2gb\n    nvidia.com/gpu-3gb\n    nvidia.com/gpu-4gb\n    ...\n    nvidia.com/gpu-30gb\n    nvidia.com/gpu-31gb\n    nvidia.com/gpu-32gb\n    ```\n\n* Restart the dispatcher pod to reload configurations (by deleting it)\n\n### 5. Forward kourier in-cluster gateway \n\n* Assume the cluster external ip is unavailable, we make our test using in-cluster ip, which is likely available in most cases\n\n* Open another terminal window , then : `make forward`\n\n### 6. Send test API request to Dispatcher\n* Export your HuggingFace token : `export HF_TOKEN=\"\u003cYour token\u003e\"`\n* Change Directory to `/test` and install required python package through `pip install -r requirements.txt`\n* Run `python test.py` to send sample request to Dispatcher\n\n### (OPTIONAL) Send Customize request to Dispatcher\n* Make sure you done all steps above.\n* You can set custom request through modifying `/test/payload.json`\t:\n```\n{\n    \"token\": \"What is Deep Learning?\",\n    \"par\": {\n        \"max_new_tokens\": \"20\"\n    },\n    \"env\": {\n        \"MODEL_ID\": \"openai-community/gpt2\",\n        \"HF_TOKEN\": \"\"\n    }\n}\n``` \n* Reference for parameters (par): https://huggingface.co/docs/transformers/main_classes/text_generation\n* Reference for environment variables (env) : https://huggingface.co/docs/text-generation-inference/main/en/reference/launcher\n\n## Uninstall Project\n* Delete all service running\n* Run `make clean` to remove dispatcher\n* Run `make remove_knative` to remove knative\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeeelin%2Fssis-dispatcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeeelin%2Fssis-dispatcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeeelin%2Fssis-dispatcher/lists"}