{"id":14108676,"url":"https://github.com/redpanda-data/benthos","last_synced_at":"2025-06-22T11:09:48.753Z","repository":{"id":241809013,"uuid":"803466352","full_name":"redpanda-data/benthos","owner":"redpanda-data","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-17T08:45:31.000Z","size":25119,"stargazers_count":362,"open_issues_count":52,"forks_count":85,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-06-21T00:36:37.993Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redpanda-data.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-20T19:22:37.000Z","updated_at":"2025-06-19T09:47:35.000Z","dependencies_parsed_at":"2024-06-12T13:44:45.327Z","dependency_job_id":"e083d3de-b15d-44b8-9613-2ec21e6f71bb","html_url":"https://github.com/redpanda-data/benthos","commit_stats":null,"previous_names":["redpanda-data/benthos"],"tags_count":496,"template":false,"template_full_name":null,"purl":"pkg:github/redpanda-data/benthos","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redpanda-data%2Fbenthos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redpanda-data%2Fbenthos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redpanda-data%2Fbenthos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redpanda-data%2Fbenthos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redpanda-data","download_url":"https://codeload.github.com/redpanda-data/benthos/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redpanda-data%2Fbenthos/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261282312,"owners_count":23134940,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-14T10:01:34.348Z","updated_at":"2025-06-22T11:09:43.740Z","avatar_url":"https://github.com/redpanda-data.png","language":"Go","funding_links":[],"categories":["Go","others"],"sub_categories":[],"readme":"![Benthos](icon.png \"Benthos\")\n\n[![godoc for redpanda-data/benthos][godoc-badge]][godoc-url]\n[![Build Status][actions-badge]][actions-url]\n\n\u003e Note: if you are looking for the original benthos repo with connectors, it moved here: https://github.com/redpanda-data/connect\n\n\nBenthos is a framework for creating declarative stream processors where a pipeline of one or more sources, an arbitrary series of processing stages, and one or more sinks can be configured in a single config:\n\n```yaml\ninput:\n  gcp_pubsub:\n    project: foo\n    subscription: bar\n\npipeline:\n  processors:\n    - mapping: |\n        root.message = this\n        root.meta.link_count = this.links.length()\n        root.user.age = this.user.age.number()\n\noutput:\n  redis_streams:\n    url: tcp://TODO:6379\n    stream: baz\n    max_in_flight: 20\n```\n\n### Delivery Guarantees\n\nDelivery guarantees [can be a dodgy subject](https://youtu.be/QmpBOCvY8mY). Benthos processes and acknowledges messages using an in-process transaction model with no need for any disk persisted state, so when connecting to at-least-once sources and sinks it's able to guarantee at-least-once delivery even in the event of crashes, disk corruption, or other unexpected server faults.\n\nThis behaviour is the default and free of caveats, which also makes deploying and scaling Benthos much simpler.\n\n## Lint\n\nBenthos uses [golangci-lint][golangci-lint] for linting, which you can install with:\n\n```shell\ncurl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin\n```\n\nAnd then run it with `make lint`.\n\n[godoc-badge]: https://pkg.go.dev/badge/github.com/redpanda-data/benthos/v4/public\n[godoc-url]: https://pkg.go.dev/github.com/redpanda-data/benthos/v4/public\n[actions-badge]: https://github.com/redpanda-data/benthos/actions/workflows/test.yml/badge.svg\n[actions-url]: https://github.com/redpanda-data/benthos/actions/workflows/test.yml\n\n[golangci-lint]: https://golangci-lint.run/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredpanda-data%2Fbenthos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredpanda-data%2Fbenthos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredpanda-data%2Fbenthos/lists"}