{"id":28168657,"url":"https://github.com/nearai/private-ml-sdk","last_synced_at":"2025-05-15T15:14:00.151Z","repository":{"id":273640963,"uuid":"914460214","full_name":"nearai/private-ml-sdk","owner":"nearai","description":"Run LLMs and agents on TEEs leveraging NVIDIA GPU TEE and Intel TDX technologies.","archived":false,"fork":false,"pushed_at":"2025-03-18T09:12:19.000Z","size":1475,"stargazers_count":39,"open_issues_count":5,"forks_count":9,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-03-18T10:25:25.026Z","etag":null,"topics":["cvm","tdx","tee"],"latest_commit_sha":null,"homepage":"https://docs.near.ai/private-ml-sdk","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nearai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-09T16:33:12.000Z","updated_at":"2025-03-15T18:14:09.000Z","dependencies_parsed_at":"2025-03-18T10:24:29.350Z","dependency_job_id":"1b8fc183-74dd-4a84-9d21-de0fbc1d77ac","html_url":"https://github.com/nearai/private-ml-sdk","commit_stats":null,"previous_names":["nearai/private-ml-sdk"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nearai%2Fprivate-ml-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nearai%2Fprivate-ml-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nearai%2Fprivate-ml-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nearai%2Fprivate-ml-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nearai","download_url":"https://codeload.github.com/nearai/private-ml-sdk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254364248,"owners_count":22058880,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvm","tdx","tee"],"created_at":"2025-05-15T15:13:59.507Z","updated_at":"2025-05-15T15:14:00.144Z","avatar_url":"https://github.com/nearai.png","language":"Python","funding_links":[],"categories":["Related Repositories"],"sub_categories":["Monthly Budget Targets"],"readme":"# Private ML SDK\n\nA secure and verifiable solution for running Large Language Models (LLMs) in Trusted Execution Environments (TEEs), leveraging NVIDIA GPU TEE and Intel TDX technologies.\n\n![Architecture Overview](./assets/image/gpu-tee.webp)\n\n## Overview\n\nPrivate ML SDK provides a secure environment for running LLM workloads with guaranteed privacy and security, preventing unauthorized access to both the model and user data during inference operations. The solution leverages NVIDIA's TEE GPU technology (H100/H200/B100) and Intel CPUs with TDX support to ensure that AI model execution and data processing remain fully protected within secure enclaves.\n\nKey features:\n- Tamper-proof data processing\n- Secure execution environment\n- Open source and reproducible builds\n- Verifiable execution results\n- Nearly native speed performance (up to 99% efficiency)\n\n## Architecture\n\nThe system consists of several core components:\n\n- **Secure Compute Environment**: TEE-based execution environment\n- **Remote Attestation**: Verification of the TEE environment\n- **Secure Communication**: End-to-end encryption between users and LLM\n- **Key Management Service (KMS)**: Key management service to manage keys for encryption and decryption\n\n## Getting Started\n\n### Build the TDX guest image\n\nPrerequisites:\n- Install Docker:\n  ```bash\n  curl -fsSL https://get.docker.com -o get-docker.sh\n  sudo sh get-docker.sh\n  ```\n- Add the current user to the docker group:\n  ```bash\n  sudo usermod -aG docker $USER\n  newgrp docker  # Apply group changes without logout\n  ```\n- Verify Docker installation:\n  ```bash\n  docker --version\n  docker run hello-world\n  ```\n\nClone the repository and build the TDX guest image:\n\n```\ngit clone https://github.com/nearai/private-ml-sdk --recursive\ncd private-ml-sdk/\n./build.sh\n```\n\nIf everything goes well, you should see the images files in `private-ml-sdk/images/`.\n\nThere are two image directories:\n- `dstack-nvidia-0.3.0/`: the production image without developer tools.\n- `dstack-nvidia-dev-0.3.0/`: the development image with developer tools, such as `sshd`, `strace`.\n\n### Run the Local KMS\n\nBefore launching the CVM, ensure that the Local KMS is operational, as it provides the essential keys required for the proper initialization of the CVM. These keys are derived from the local TEE hardware environment.\n\nThe Local KMS service can be launched by following commands:\n\n```bash\ncd private-ml-sdk/meta-dstack-nvidia/dstack/key-provider-build/\n./run.sh\n```\n\n### Run the TDX guest image\n\nThis requires a TDX host machine with the TDX driver installed and Nvidia GPU what support GPU TEE installed.\n\n```\n# Add the scripts/bin directory to the PATH environment variable\npushd private-ml-sdk/meta-dstack-nvidia/scripts/bin\nPATH=$PATH:`pwd`\npopd\n\n# List the Available GPUs\ndstack lsgpu\n\n# Output like the following:\n# Available GPU IDs:\n# ID      Description\n# 18:00.0 3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB] (rev a1)\n# 2a:00.0 3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB] (rev a1)\n# 3a:00.0 3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB] (rev a1)\n# 5d:00.0 3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB] (rev a1)\n# 9a:00.0 3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB] (rev a1)\n# ab:00.0 3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB] (rev a1)\n# ba:00.0 3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB] (rev a1)\n# db:00.0 3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB] (rev a1)\n\n# Choose one or more GPU IDs and run the following command to create a CVM instance\ndstack new app.yaml -o my-gpu-cvm \\\n    --local-key-provider \\\n    --gpu 18:00.0 \\\n    --image images/dstack-nvidia-dev-0.3.3 \\\n    -c 2 -m 4G -d 100G \\\n    --port tcp:127.0.0.1:10022:22 \\\n    --port tcp:0.0.0.0:8888:8888\n\n# Run the CVM:\nsudo -E dstack run my-gpu-cvm\n```\n\nAn example of the `app.yaml` file is as follows:\n\n```yaml\n# app.yaml\nservices:\n  jupyter:\n    image: kvin/cuda-notebook\n    privileged: true\n    ports:\n      - \"8888:8888\"\n    volumes:\n      - /var/run/tappd.sock:/var/run/tappd.sock\n    deploy:\n      resources:\n        reservations:\n          devices:\n            - driver: nvidia\n              count: all\n              capabilities: [gpu]\n    runtime: nvidia\n```\n\n### Verifying the TDX CVM Service Status\n\n1. **SSH into the CVM**\n   Note: SSH access is only available when using the development image `images/dstack-nvidia-dev-0.3.3`.\n\n   Execute the following command to establish an SSH connection:\n   ```bash\n   ssh -p 10022 root@localhost\n   ```\n   *Note: Port 10022 is mapped to the CVM's port 22 during the creation of the CVM.*\n\n   After logging in, check the status of the Docker Compose services by running:\n   ```bash\n   docker ps -a\n   ```\n\n2. **Verify the Jupyter Service Status**\n   To confirm that the Docker Compose services are running correctly, access the Jupyter service through your web browser:\n\n   - For local access, navigate to:\n     [http://localhost:8888](http://localhost:8888)\n     *(Port 8888 is mapped to the CVM's port 8888 during the creation of the CVM.)*\n\n   - For remote access, use the following URL:\n     [http://\u003cpublic-ip\u003e:8888](http://\u003cpublic-ip\u003e:8888)\n     *(Replace `\u003cpublic-ip\u003e` with the actual public IP address of your CVM.)*\n\n\n### Updating the Environment Variables of the TDX CVM\n\nTo update the environment variables for the TDX CVM, execute the following command:\n\n```bash\ncp env-file \u003cyour-cvm-path\u003e/shared/env-file\n```\n\nThe `env-file` is a text file that contains the necessary environment variables for the Docker Compose services. An example of the contents of the `env-file` is as follows:\n\n```\n# env-file\nREDPILL_API_KEY=sk-1234567890\nREDPILL_MODEL=phala/llama-3.3-70b-instruct\n```\n\nAfter copying the `env-file`, restart the CVM. The environment variables specified in the `env-file` will be accessible within the Docker Compose service YAML. During the boot process, this `env-file` is copied to the `/tapp/env-file` directory within the CVM.\n\nFor instance, in your `docker-compose.yaml`, you can reference the `env-file` as shown below:\n\n```yaml\n# docker-compose.yaml, using the Jupyter service as an example\nservices:\n  jupyter:\n    env_file:\n      - /tapp/env-file\n```\n\nThis approach ensures that your environment variables are properly configured and accessible to your services.\n\n### Getting TDX quote inside the container\n\n1. Install the SDK package:\n```bash\npip install dstack-sdk\n```\n\n2. Get TDX quote using Python:\n```python\nfrom dstack_sdk import TappdClient\n\n# Initialize the client\nclient = TappdClient()\n\n# Get quote for a message\nresult = client.tdx_quote('test')\nprint(result.quote)\n```\n\n## Performance\n\nBased on benchmarks running LLMs in NVIDIA H100 and H200:\n- Efficiency approaches 99% as input size grows\n- Minimal overhead for larger models (e.g., Phi3-14B-128k and Llama3.1-70B)\n- Performance scales well with increased input sizes and model complexities\n- I/O overhead becomes negligible in high-computation scenarios\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## References\n\n- [NVIDIA Confidential Computing](https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/)\n- [Intel TDX Documentation](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnearai%2Fprivate-ml-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnearai%2Fprivate-ml-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnearai%2Fprivate-ml-sdk/lists"}