{"id":34147711,"url":"https://github.com/usedatabrew/tango","last_synced_at":"2026-03-11T02:02:35.708Z","repository":{"id":210292539,"uuid":"726129915","full_name":"usedatabrew/tango","owner":"usedatabrew","description":"Open Source pipeline lib with backpressure","archived":false,"fork":false,"pushed_at":"2023-12-11T21:18:27.000Z","size":23,"stargazers_count":11,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2024-05-13T13:09:06.756Z","etag":null,"topics":["backpressure","etl","golang-library","pipeline","streaming-data"],"latest_commit_sha":null,"homepage":"https://databrew.tech","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/usedatabrew.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-12-01T15:43:18.000Z","updated_at":"2024-05-07T19:27:56.000Z","dependencies_parsed_at":"2023-12-11T23:06:14.544Z","dependency_job_id":null,"html_url":"https://github.com/usedatabrew/tango","commit_stats":null,"previous_names":["usedatabrew/tango"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/usedatabrew/tango","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usedatabrew%2Ftango","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usedatabrew%2Ftango/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usedatabrew%2Ftango/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usedatabrew%2Ftango/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/usedatabrew","download_url":"https://codeload.github.com/usedatabrew/tango/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usedatabrew%2Ftango/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30367800,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"online","status_checked_at":"2026-03-11T02:00:07.027Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backpressure","etl","golang-library","pipeline","streaming-data"],"created_at":"2025-12-15T04:29:29.915Z","updated_at":"2026-03-11T02:02:35.699Z","avatar_url":"https://github.com/usedatabrew.png","language":"Go","readme":"## \u003cp align=\"center\"\u003eTango - Open Source Golang Pipeline with Backpressure (WIP)\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./image/logo.png\" alt=\"\"\u003e\n\u003c/p\u003e\n\nTango can help you implement a multi-stage pipeline that will apply backpressure on the data producer channel (will pause\nreading)\nin case the producer is faster than the stage processors. It may be a great choice if you want to create an ETL pipeline with\ncustom processing\n\n## Concept\nWhile building the pipeline, you must consider the case then one of the ETL stages may be slower than the rest.\nE.g you may want to execute custom data enrichment with SQL queries, that means you will be reading data from the source (like Kafka)\nway faster than processing them. That why we created Tango.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://miro.medium.com/v2/resize:fit:640/format:webp/1*B6KBntP9kiFBPN7NXlGruQ.gif\" alt=\"\"\u003e\n\u003c/p\u003e\n\n## Usage example\n\n```go\npackage main\n\nimport (\n\t\"github.com/usedatabrew/tango\"\n\t\"time\"\n)\n\nfunc main() {\n\tinstance := tango.NewTango()\n\n\tstages := []tango.Stage{\n\t\t{\n\t\t\tChannel: make(chan interface{}),\n\t\t\tFunction: func(msg interface{}) (interface{}, error) {\n\t\t\t\treturn msg, nil\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tChannel: make(chan interface{}),\n\t\t\tFunction: func(msg interface{}) (interface{}, error) {\n\t\t\t\tif msg.(int)%5 == 0 {\n\t\t\t\t\ttime.Sleep(time.Millisecond * 10)\n\t\t\t\t}\n\t\t\t\treturn msg, nil\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tChannel: make(chan interface{}),\n\t\t\tFunction: func(msg interface{}) (interface{}, error) {\n\t\t\t\treturn msg, nil\n\t\t\t},\n\t\t},\n\t}\n\t\n\tinstance.SetStages(stages)\n\n\tproducerChannel := make(chan interface{})\n\n\tgo func() {\n\t\tfor i := 0; i \u003c= 10000000; i++ {\n\t\t\tproducerChannel \u003c- i\n\t\t\ttime.Sleep(time.Millisecond * 200)\n\t\t}\n\t}()\n\n\t// Tango will consume messages from the producer channel and \n\t// pass them through the stages.\n\tinstance.SetProducerChannel(producerChannel)\n\t\n\tif err := instance.Start(); err != nil {\n\t\tpanic(err)\n\t}\n}\n```\n\n### Buffered channels\n\nYou can pass buffered channels for each stage, that will help you mitigate spikes in when a lot of updates going to the pipeline\n\n```go\nstage := tango.Stage{\n    Channel: make(chan interface{}, 100),\n    Function: func(msg interface{}) (interface{}, error) {\n        return msg, nil\n    },\n},\n```\n\n### Accomplished callback\nSometimes you need to perform some extra work after the last stage is done, like messages was written to the sink\nYou can do this by passing callback function to  `OnProcessed` method of Tango instance\n\n```go\ntangoInstance.OnProcessed(func(i interface{}, err error) {\n    if err != nil {\n        fmt.Errorf(\"Messages happened in the last stage %v\", err)\n    }\n    fmt.Println(\"Message processed\", i)\n})\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fusedatabrew%2Ftango","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fusedatabrew%2Ftango","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fusedatabrew%2Ftango/lists"}