{"id":13671923,"url":"https://github.com/substantic/rain","last_synced_at":"2025-04-27T18:31:56.629Z","repository":{"id":29341859,"uuid":"121410309","full_name":"substantic/rain","owner":"substantic","description":"Framework for large distributed pipelines","archived":false,"fork":false,"pushed_at":"2023-03-04T01:27:07.000Z","size":12186,"stargazers_count":753,"open_issues_count":37,"forks_count":54,"subscribers_count":42,"default_branch":"master","last_synced_at":"2025-04-12T21:02:20.756Z","etag":null,"topics":["distributed-computing","pipelines","python-api","rust","workflows"],"latest_commit_sha":null,"homepage":"https://substantic.github.io/rain/docs/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/substantic.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-02-13T17:04:46.000Z","updated_at":"2025-04-11T15:06:08.000Z","dependencies_parsed_at":"2023-02-14T14:01:25.237Z","dependency_job_id":"5c9b7360-da1e-48ff-88df-bafc6492f281","html_url":"https://github.com/substantic/rain","commit_stats":{"total_commits":712,"total_committers":11,"mean_commits":64.72727272727273,"dds":0.401685393258427,"last_synced_commit":"477948554150760164c6fe48eac27bcf06c7933b"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/substantic%2Frain","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/substantic%2Frain/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/substantic%2Frain/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/substantic%2Frain/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/substantic","download_url":"https://codeload.github.com/substantic/rain/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251187352,"owners_count":21549621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","pipelines","python-api","rust","workflows"],"created_at":"2024-08-02T09:01:22.145Z","updated_at":"2025-04-27T18:31:54.311Z","avatar_url":"https://github.com/substantic.png","language":"Rust","readme":"# Rain\n\n[![](https://img.shields.io/github/license/substantic/rain.svg)](https://github.com/substantic/rain/blob/master/LICENSE)\n[![PyPI](https://img.shields.io/pypi/v/rain-python.svg)](https://pypi.org/project/rain-python/)\n[![Crates.io](https://img.shields.io/crates/v/rain_core.svg)](https://www.crates.io/crates/rain_core)\n[![Build Status](https://travis-ci.org/substantic/rain.svg?branch=master)](https://travis-ci.org/substantic/rain)\n[![Gitter](https://badges.gitter.im/substantic/rain.svg)](https://gitter.im/substantic/rain?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n\u003cimg align=\"right\" width=\"35%\" src=\"docs/imgs/logo.svg?sanitize=true\"\u003e\n\n**Rain** is an open-source distributed computational framework for processing\nof large-scale task-based pipelines.\n\nRain aims to lower the entry barrier to the world of distributed computing. Our\nintention is to provide a light yet robust distributed framework that features\nan intuitive Python API, straightforward installation and deployment with\ninsightful monitoring on top.\n\n\u003e Despite that this is an early release of Rain, it is a fully functional\n\u003e project that can be used out-of-the box. Being aware that there is still\n\u003e a lot that can be improved and added, we are looking for external\n\u003e users and collaborators to help to move this work forward.\n\u003e Talk to us online at Gitter or via email and let us know what your\n\u003e projects and use-cases need, submit bugs or feature\n\u003e requests at GitHub or even contribute with pull requests.\n\n## Features\n\n- **Dataflow programming.** Computation in Rain is defined as a flow graph of\n  tasks. Tasks may be built-in functions, Python/C++/Rust code, or an external\n  applications, short and light or long-running and heavy. The system is\n  designed to integrate any code into a pipeline, respecting its resource\n  requirements, and to handle very large task graphs (hundreds thousands tasks).\n\n- **Easy to use.** Rain was designed to be easy to deployed anywhere, ranging\n  from a single node deployments to large-scale distributed systems and clouds\n  ranging thousands of cores.\n\n- **Rust core, Python/C++/Rust API.** Rain is written in Rust for safety and\n  efficiency and has a high-level Python API to Rain core infrastructure, and\n  even supports Python tasks out-of-the-box. Rain also provides libraries for\n  writing own tasks in C++ and Rust.\n\n- **Monitoring.** Rain is designed to support both online and postmortem\n  monitoring.\n\n  ![Dashboard screencast](docs/imgs/rain-dashboard.gif)\n\n## Documentation\n\n[Overview](http://substantic.github.io/rain/docs/overview.html) \u0026bull; [Quickstart](http://substantic.github.io/rain/docs/quickstart.html) \u0026bull; [User guide](http://substantic.github.io/rain/docs/user.html) \u0026bull; [Python API](http://substantic.github.io/rain/docs/python_api.html) \u0026bull; [Examples](http://substantic.github.io/rain/docs/examples.html)\n\n## Quick start\n\n- Download binary\n\n```\n$ wget https://github.com/substantic/rain/releases/download/v0.4.0/rain-v0.4.0-linux-x64.tar.xz\n$ tar xvf rain-v0.4.0-linux-x64.tar.xz\n```\n\n- Install Python API\n\n```\n$ pip3 install rain-python\n```\n\n- Start server \u0026 a single local governor\n\n```\n$ ./rain-v0.4.0-linux-x64/rain start --simple\n```\n\n- Rain \"Hello world\" in Python\n\n```python\nfrom rain.client import Client, tasks, blob\n\nclient = Client(\"localhost\", 7210)\n\nwith client.new_session() as session:\n    task = tasks.Concat((blob(\"Hello \"), blob(\"world!\")))\n    task.output.keep()\n    session.submit()\n    result = task.output.fetch().get_bytes()\n    print(result)\n```\n\n### Installation via cargo\n\nIf you have installed Rust, you can install and start Rain as follows:\n\n```\n$ cargo install rain_server\n\n$ pip3 install rain-python\n\n$ rain start --simple\n```\n\n[Read the docs](http://substantic.github.io/rain/docs/examples.html) for more examples.\n","funding_links":[],"categories":["Rust"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubstantic%2Frain","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsubstantic%2Frain","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubstantic%2Frain/lists"}