{"id":28458788,"url":"https://github.com/openmined/syft_experimental","last_synced_at":"2025-07-02T05:30:56.678Z","repository":{"id":103100254,"uuid":"280111508","full_name":"OpenMined/syft_experimental","owner":"OpenMined","description":"Deliberate experimental Rust implementation of Syft ","archived":false,"fork":false,"pushed_at":"2021-05-20T02:53:38.000Z","size":69,"stargazers_count":11,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"dev","last_synced_at":"2025-06-07T00:40:06.826Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenMined.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":"openmined","open_collective":"openmined"}},"created_at":"2020-07-16T09:27:14.000Z","updated_at":"2024-05-11T01:53:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"96fa10e9-5e46-48c1-92fd-b7d084661f3f","html_url":"https://github.com/OpenMined/syft_experimental","commit_stats":{"total_commits":13,"total_committers":2,"mean_commits":6.5,"dds":0.3076923076923077,"last_synced_commit":"6bcfd587fe4c939ccc3e8feb9a66ce9d150774b0"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OpenMined/syft_experimental","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2Fsyft_experimental","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2Fsyft_experimental/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2Fsyft_experimental/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2Fsyft_experimental/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenMined","download_url":"https://codeload.github.com/OpenMined/syft_experimental/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenMined%2Fsyft_experimental/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263081076,"owners_count":23410807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-07T00:39:48.413Z","updated_at":"2025-07-02T05:30:56.666Z","avatar_url":"https://github.com/OpenMined.png","language":"Rust","funding_links":["https://github.com/sponsors/openmined","https://opencollective.com/openmined"],"categories":[],"sub_categories":[],"readme":"![Core](https://github.com/OpenMined/syft_experimental/workflows/Core/badge.svg)\n![Python](https://github.com/OpenMined/syft_experimental/workflows/Python/badge.svg)\n\n# Syft Experiment\n\nThis repo is a coordinated effort to build a Rust alternative for PySyft, with native\nhost language bindings.\n\nThis README.md is formatted with remark:\nmrmlnc.vscode-remark\n\n## Monorepo Structure\n\nThe folder structure looks like this:\n\n```\n├── platforms\n│   └── python    \u003c- Syft Python Host Code\n├── protos        \u003c- Shared Proto definitions\n└── syft          \u003c- Syft Core Rust Code\n    ├── src\n    └── target\n```\n\n## Setup\n\n- python 3.7+ - https://www.python.org/\n- rustup - https://rustup.rs/\n- bloomrpc - https://github.com/uw-labs/bloomrpc\n- protoc - https://github.com/protocolbuffers/protobuf\n- vscode - https://github.com/microsoft/vscode\n\n### Linux\n\n### MacOS\n\nPython\n\n```\n$ brew install python\n```\n\nrustup\n\n```\n$ brew install rustup\n$ rustup-init\n```\n\nbloomrpc\n\n```\n$ brew cask install bloomrpc\n```\n\nprotoc\n\n```\n$ brew install protobuf\n```\n\n### Windows\n\n## Rust Toolchain\n\nWe are currently using nightly due to some rust dependencies.\n\n```\n$ rustup toolchain install nightly\n$ rustup default nightly\n```\n\n### Formatting and Linting\n\nRust comes with an opinionated formatter and linter so we will mandate that these\nare used.\n\nInstall Rust Format:\n\n```\n$ rustup component add rustfmt\n```\n\nInstall Rust Language Server:\n\n```\n$ rustup component add rls\n```\n\nInstall Rust Linting:\n\n```\n$ rustup component add clippy\n```\n\n### VSCode Configuration\n\nWhile VSCode is not required it is highly recommended.\n\nInstall Rust VSCode Extension:\nhttps://marketplace.visualstudio.com/items?itemName=rust-lang.rust\n\n```\n$ code --install-extension rust-lang.rust\n```\n\nInstall Even Better TOML Extension:\nhttps://marketplace.visualstudio.com/items?itemName=tamasfe.even-better-toml\n\n```\n$ code --install-extension tamasfe.even-better-toml\n```\n\nAdd to settings:\n\n```\n{\n  \"evenBetterToml.formatter.reorderKeys\": false,\n  \"evenBetterToml.formatter.alignEntries\": true\n}\n```\n\n## Python\n\n### Setup\n\nMake sure you have `python3.7+`\n\nWe use a virtual environment to isolate the syft core python wheel development and\nbuild process.\n\nWe include support for Pipenv, Conda and pip with virtualenv.\n\n### Formatting and Linting\n\nTo keep code clean and bug free we mandate all code inside syft core python, uses an\nagreed upon set of linting and formatting standards.\n\n- black - https://github.com/psf/black\n- isort - https://github.com/timothycrosley/isort\n- mypy - http://mypy-lang.org/\n\n```\n$ pip install black isort mypy\n```\n\n### VSCode Configuration\n\nAdd these to your settings.json, making sure to update the paths as necessary to your\nplatform.\n\n```\n{\n  \"python.linting.enabled\": true,\n  \"python.linting.mypyEnabled\": true,\n  \"python.formatting.provider\": \"black\",\n  \"python.linting.mypyPath\": \"/usr/local/bin/mypy\",\n  \"python.formatting.blackPath\": \"/usr/local/bin/black\"\n}\n```\n\n### Python Package Managers\n\n#### Pipenv\n\nUpgrade pip:\n\n```\n$ pip install --upgrade pip\n```\n\nInstall pipenv:\n\n```\n$ pip install pipenv\n```\n\nEnter virtualenv:\n\n```\n$ cd platforms/python\n$ pipenv shell\n```\n\nInstall packages:\n\n```\n$ pipenv install --dev --skip-lock\n```\n\n#### Conda\n\nCreate your conda environment, navigate to the /platforms/python directory:\n\n```\n$ conda create --name syft --file requirements.txt\n```\n\n#### pip and virtualenv\n\nCreate a virtualenv in the /platforms/python folder and install the packages inside\nrequirements.txt\n\n### Virtualenv\n\nMake sure to enable your virtualenv from Pipenv, conda or other virtualenv system when\ndoing any commands relating to maturin or python.\n\nIf you are using pipenv:\n\n```\n$ cd platforms/python\n$ pipenv shell\n```\n\nIf you are using conda:\n\n```\n$ cd platforms/python\n$ conda activate syft\n```\n\n## Python Development\n\nYou can compile and install the python library from the virtualenv in one command:\n\n```\n$ maturin develop\n```\n\n## Python Tests\n\nWe are using pytest which is listed in the Pipfile / requirements.txt.\n\nRun tests from the platforms/python directory inside your virtualenv:\n\n```\n$ cd platforms/python\n$ maturin develop; pytest\n```\n\n## Mixed Python \u0026 Rust Module Imports\n\nThe rust crate pyo3 allows us to mix compiled Rust code as a CPython module and vanilla\npython code in the same wheel. The vanilla python code must go into a folder named the\nsame as the module and must contain at least a single `__init__.py` file in that folder.\nThat is why you will see this inside /platforms/python:\n\n```\n├── src           \u003c--- Rust Code\n│   └── ffi\n├── syft          \u003c--- Python Code\n│   ├── message\n│   ├── node\n│   └── protos\n├── target\n│   ├── debug\n│   ├── rls\n│   └── wheels\n└── tests         \u003c--- Python Tests\n```\n\nTo allow for a nice consistent import interface there is some code inside the vanilla\npython source which acts to convert the awkward issues with CPython module names\nand the way the Rust pyo3 submodules are defined.\n\nWithout this fix, importing a Rust module defined as syft.message might look like:\n\n```\nfrom syft.syft import message\n```\n\nImporting further nested items results in an error:\n\n```\nfrom syft.syft.message import run_class_method_message\n\u003e\u003e\u003e ModuleNotFoundError: No module named 'syft.syft.message';\n```\n\nHowever by re-exporting the modules using a matching directory structure and\n`__init__.py` files we can provide a clean interface:\n\n```\n# import from rust in platforms/python/src\nfrom syft.message import run_class_method_message\n# import from vanilla python in platforms/python/syft\n```\n\nhttps://github.com/PyO3/maturin/issues/326\n\n## Build Python Wheel\n\nDuring this step:\n\n- The syft core rust library is built\n- The synthesized python interface to the protos are compiled with protoc\n- The python platform ffi code in platforms/python/src is compiled for your system arch\n- The vanilla python code inside platforms/python/syft is added\n- Code inside the `__init__.py` files allows for a consistent module import syntax\n- A wheel is created with both these mixed source files\n\nBuild wheel and install wheel:\n\n```\n$ maturin build -i python\n$ pip install `find -L ./target/wheels -name \"*.whl\"`\n```\n\n# Hello World Demo\n\n## Start Worker from Python\n\n```\n$ pipenv shell\n$ maturin develop\n$ python -i examples/worker.py\n```\n\nYou should see:\n\n```\nStarting node on [::1]:50051\nTokio thread started\nCapability registered: hello\nCapability registered: sum\nCapability registered: sum_np\n\u003e\u003e\u003e\n```\n\n## Start Client from Python\n\n```\n$ pipenv shell\n$ maturin develop\n$ python -i examples/client.py\n```\n\nYou should see:\n\n```\nTokio thread started\nCapabilities returned: [\"sum\", \"hello\", \"sum_np\"]\nNode at: http://[::1]:50051 has capabilities: ['sum', 'hello', 'sum_np']\nHello: Client 1\n6\n6\n\u003e\u003e\u003e\n```\n\nTry issuing a command like:\n\n```\n\u003e\u003e\u003e  execute_capability(target_addr, \"sum\", [i for i in range(0, 100)])\n4950\n\u003e\u003e\u003e\n```\n\n## Jupyter Notebook\n\nYou can run the Hello World demo with jupyter by opening the two notebooks.\n\n```\n$ pipenv shell\n$ maturin develop\n$ jupyter notebook\n```\n\nMake sure to initialize the worker and register capability functions before sending requests from the client.\n\n## Zero Config Port Forwarding\n\nIf you want to test this between computers on different networks over the internet try ngrok.\n\nhttps://ngrok.com/\n\n### Linux\n\n### MacOS\n\nInstall ngrok:\n\n```\n$ brew cask install ngrok\n```\n\nStart your worker and pick a port then run ngrok like so:\n\n```\n$ ngrok tcp 50051\n```\n\nYou should see output like:\n\n```\nForwarding                    tcp://0.tcp.ngrok.io:12345 -\u003e localhost:50051\n```\n\nIn your client use the address like so:\n\n```\ntarget_addr = \"tcp://0.tcp.ngrok.io:12345\"\nexecute_capability(target_addr, \"sum\", [i for i in range(0, 10)])\n45\n\u003e\u003e\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenmined%2Fsyft_experimental","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenmined%2Fsyft_experimental","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenmined%2Fsyft_experimental/lists"}