{"id":22054092,"url":"https://github.com/cmpadden/dagster-pipes-rust","last_synced_at":"2025-10-11T13:30:27.839Z","repository":{"id":265114771,"uuid":"895180975","full_name":"cmpadden/dagster-pipes-rust","owner":"cmpadden","description":"Dagster pipes implementation in Rust","archived":false,"fork":false,"pushed_at":"2024-12-06T20:05:04.000Z","size":47,"stargazers_count":6,"open_issues_count":5,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-06T21:01:35.091Z","etag":null,"topics":["dagster","data","integrations","orchestration","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cmpadden.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-27T17:50:29.000Z","updated_at":"2024-12-06T20:14:44.000Z","dependencies_parsed_at":"2024-12-01T07:00:58.021Z","dependency_job_id":null,"html_url":"https://github.com/cmpadden/dagster-pipes-rust","commit_stats":null,"previous_names":["cmpadden/dagster-pipes-rust"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmpadden%2Fdagster-pipes-rust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmpadden%2Fdagster-pipes-rust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmpadden%2Fdagster-pipes-rust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmpadden%2Fdagster-pipes-rust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cmpadden","download_url":"https://codeload.github.com/cmpadden/dagster-pipes-rust/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":236096170,"owners_count":19094178,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dagster","data","integrations","orchestration","rust"],"created_at":"2024-11-30T15:19:33.283Z","updated_at":"2025-10-11T13:30:22.547Z","avatar_url":"https://github.com/cmpadden.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e [!NOTE]\n\u003e This project has been moved to the official Dagster `community-integrations` repo. The maintained version can be found [here](https://github.com/dagster-io/community-integrations/tree/c7238204fcd539532d67abb3b8dda3bf30a24c07/libraries/pipes/implementations/rust).\n\n---\n\n# dagster-pipes-rust\n\nGet full observability into your Rust workloads when orchestrating through Dagster. With this light weight interface, you can retrieve data directly from the Dagster context, report asset materializations, report asset checks, provide structured logging, end more.\n\n[![Crates.io](https://img.shields.io/crates/v/dagster_pipes_rust.svg)](https://crates.io/crates/dagster_pipes_rust)\n\n## Usage\n\n### Installation\n\n```sh\ncargo add dagster_pipes_rust\n```\n\n### Example\n\nAn example project can be found in [./example-dagster-pipes-rust-project](./example-dagster-pipes-rust-project).\n\nIn this project there exists a `rust_processing_jobs` binary, which demonstrates how to use the Dagster context to report materializations to Dagster through the `context.report_asset_materialization` method.\n\n```rust\nuse dagster_pipes_rust::open_dagster_pipes;\nuse serde_json::json;\n\nfn main() {\n    let mut context = open_dagster_pipes();\n    let metadata = json!({\"row_count\": {\"raw_value\": 100, \"type\": \"int\"}});\n    context.report_asset_materialization(\"example_rust_subprocess_asset\", metadata);\n}\n```\n\n\u003cimg width=\"1355\" alt=\"image\" src=\"https://github.com/user-attachments/assets/ddc8c261-3e96-4e82-ad4c-723dd6b3dece\"\u003e\n\nIt also demonstrates how to run the Rust binary in a subprocess from Dagster. Note, that it's also possible to launch processes in external compute environments like Kubernetes.\n\n```python\nimport shutil\n\nimport dagster as dg\n\n\n@dg.asset(\n    group_name=\"pipes\",\n    kinds={\"rust\"},\n)\ndef example_rust_subprocess_asset(\n    context: dg.AssetExecutionContext, pipes_subprocess_client: dg.PipesSubprocessClient\n) -\u003e dg.MaterializeResult:\n    \"\"\"Demonstrates running Rust binary in a subprocess.\"\"\"\n    cmd = [shutil.which(\"cargo\"), \"run\"]\n    cwd = dg.file_relative_path(__file__, \"../rust_processing_jobs\")\n    return pipes_subprocess_client.run(\n        command=cmd,\n        cwd=cwd,\n        context=context,\n    ).get_materialize_result()\n\n\ndefs = dg.Definitions(\n    assets=[example_rust_subprocess_asset],\n    resources={\n        \"pipes_subprocess_client\": dg.PipesSubprocessClient(\n            context_injector=dg.PipesEnvContextInjector(),\n        )\n    },\n)\n```\n\n\n## Contributing\n\n### Pipes Schema\n\nWe use [jsonschema](https://json-schema.org/) to define the pipes protocol and [quicktype](https://quicktype.io/) to generate the Rust structs. Currently, the json schemas live in `jsonschema/pipes` but they should be hosted/defined in a centralized repository in the future.\n\nTo generate the Rust structs, make sure to install quicktype with `npm install -g quicktype`. Then run:\n\n```bash\n./quicktype.sh\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmpadden%2Fdagster-pipes-rust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcmpadden%2Fdagster-pipes-rust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmpadden%2Fdagster-pipes-rust/lists"}