{"id":13562504,"url":"https://github.com/argoproj-labs/old-argo-dataflow","last_synced_at":"2025-05-16T12:05:06.627Z","repository":{"id":37218283,"uuid":"343874574","full_name":"argoproj-labs/old-argo-dataflow","owner":"argoproj-labs","description":"Dataflow is a Kubernetes-native platform for executing large parallel data-processing pipelines.","archived":false,"fork":false,"pushed_at":"2025-04-02T02:43:51.000Z","size":5034,"stargazers_count":269,"open_issues_count":3,"forks_count":29,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-02T05:58:17.529Z","etag":null,"topics":["data","jetstream","kafka","kubernetes","pipeline"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/argoproj-labs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-02T18:34:05.000Z","updated_at":"2024-09-30T18:11:37.000Z","dependencies_parsed_at":"2024-01-14T03:46:23.958Z","dependency_job_id":"51ec6a64-5c77-40a2-a726-ba166ca945db","html_url":"https://github.com/argoproj-labs/old-argo-dataflow","commit_stats":{"total_commits":1394,"total_committers":20,"mean_commits":69.7,"dds":"0.24677187948350077","last_synced_commit":"79bad8a3496bab5f5af2561dd5346be8f5a27561"},"previous_names":["argoproj-labs/argo-dataflow"],"tags_count":141,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argoproj-labs%2Fold-argo-dataflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argoproj-labs%2Fold-argo-dataflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argoproj-labs%2Fold-argo-dataflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/argoproj-labs%2Fold-argo-dataflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/argoproj-labs","download_url":"https://codeload.github.com/argoproj-labs/old-argo-dataflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247994119,"owners_count":21030050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","jetstream","kafka","kubernetes","pipeline"],"created_at":"2024-08-01T13:01:09.326Z","updated_at":"2025-04-09T07:04:44.105Z","avatar_url":"https://github.com/argoproj-labs.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Dataflow\n\n## NOTICE\nArgo Dataflow has been reimplemented in the scope of a broader project focussed on real-time data processing and analytics.\nPlease checkout the new [numaflow project](https://github.com/numaproj/numaflow).\n\n## Summary\n\nDataflow is a Kubernetes-native platform for executing large parallel data-processing pipelines.\n\nEach pipeline is specified as a Kubernetes custom resource which consists of one or more steps which source and sink\nmessages from data sources such Kafka, NATS Streaming, or HTTP services.\n\nEach step runs zero or more pods, and can scale horizontally using HPA or based on queue length using built-in scaling\nrules. Steps can be scaled-to-zero, in which case they periodically briefly scale-to-one to measure queue length so they\ncan scale a back up.\n\nLearn more about [features](docs/FEATURES.md).\n\n[![Introduction to Dataflow](https://img.youtube.com/vi/afZT3aJ__jI/0.jpg)](https://youtu.be/afZT3aJ__jI)\n\n## Use Cases\n\n* Real-time \"click\" analytics\n* Anomaly detection\n* Fraud detection\n* Operational (including IoT) analytics\n\n## Screenshot\n\n![Screenshot](docs/assets/screenshot.png)\n\n## Example\n\n```bash\npip install git+https://github.com/argoproj-labs/argo-dataflow#subdirectory=dsls/python\n```\n\n```python\nfrom argo_dataflow import cron, pipeline\n\nif __name__ == '__main__':\n    (pipeline('hello')\n     .namespace('argo-dataflow-system')\n     .step(\n        (cron('*/3 * * * * *')\n         .cat()\n         .log())\n    )\n     .run())\n```\n\n## Documentation\n\nRead in order:\n\nBeginner:\n\n* [Quick start](docs/QUICK_START.md)\n* [Concepts](docs/CONCEPTS.md)\n* [Sources](docs/SOURCES.md)\n* [Processors](docs/PROCESSORS.md)\n* [Sinks](docs/SINKS.md)\n* [Examples](docs/EXAMPLES.md)\n\nIntermediate:\n\n* [Handlers](docs/CODE.md)\n* [Git usage](docs/GIT.md)\n* [Expression syntax](docs/EXPRESSIONS.md)\n* [Garbage collection](docs/GC.md)\n* [Scaling](docs/SCALING.md)\n* [Command line](docs/CLI.md)\n* [Kubectl](docs/KUBECTL.md)\n* [Events interop](docs/EVENTS_INTEROP.md)\n* [Workflow interop](docs/WORKFLOW_INTEROP.md)\n* [Meta-data](docs/META.md)\n* [Idempotence](docs/IDEMPOTENCE.md)\n\nAdvanced\n\n* [Configuration](docs/CONFIGURATION.md)\n* [Features](docs/FEATURES.md)\n* [Limitations](docs/LIMITATIONS.md)\n* [Reliability](docs/RELIABILITY.md)\n* [Metrics](docs/METRICS.md)\n* [Image contract](docs/IMAGE_CONTRACT.md)\n* [Jaeger tracing](docs/JAEGER.md)\n* [Reading material](docs/READING.md)\n* [Security](docs/SECURITY.md)\n* [Dataflow vs X](docs/DATAFLOW_VS_X.md)\n* [Contributing](docs/CONTRIBUTING.md)\n\n### Architecture Diagram\n\n[![Architecture](docs/assets/architecture.png)](https://docs.google.com/drawings/d/1Dk7mgZ3jKpBg_DQ3c8og04ULoKpGTGUt52pBE-Vet2o/edit)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fargoproj-labs%2Fold-argo-dataflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fargoproj-labs%2Fold-argo-dataflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fargoproj-labs%2Fold-argo-dataflow/lists"}