{"id":19541346,"url":"https://github.com/wingify/agentredrabbit","last_synced_at":"2025-06-15T08:38:54.247Z","repository":{"id":9480370,"uuid":"11367665","full_name":"wingify/agentredrabbit","owner":"wingify","description":"Transport agent that moves data from Redis to RabbitMQ","archived":false,"fork":false,"pushed_at":"2019-04-30T17:53:37.000Z","size":68,"stargazers_count":60,"open_issues_count":0,"forks_count":8,"subscribers_count":49,"default_branch":"master","last_synced_at":"2025-04-26T17:46:46.994Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wingify.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-07-12T12:00:11.000Z","updated_at":"2025-03-17T08:36:06.000Z","dependencies_parsed_at":"2022-08-25T21:41:00.773Z","dependency_job_id":null,"html_url":"https://github.com/wingify/agentredrabbit","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/wingify/agentredrabbit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wingify%2Fagentredrabbit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wingify%2Fagentredrabbit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wingify%2Fagentredrabbit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wingify%2Fagentredrabbit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wingify","download_url":"https://codeload.github.com/wingify/agentredrabbit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wingify%2Fagentredrabbit/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259945736,"owners_count":22935877,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T03:09:59.169Z","updated_at":"2025-06-15T08:38:54.226Z","avatar_url":"https://github.com/wingify.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# agentredrabbit: Redis to RabbitMQ transport agent\n\nagentredrabbit is a transport agent written in Python which moves data from\nRedis to RabbitMQ. It uses multithreaded workers to do the job and supports\nfailsafe graceful shutdown and recovery.\n\n![agentredrabbit](/docs/flow.png \"agentredrabbit\")\n\n- `agentredrabbit` on startup loads failed messages from a dump file, if any\n- Thread workers synchronize using a global lock when working with failsafe queue\n- It polls to check any messages in a Redis list by name `queue:queue1` and pops\n  messages in max. chunk size of 100 and tries to publish to RabbitMQ broker\n- If message publishing fails, it saves the chunk to an internal failsafe message\n  queue (a heap queue), stops consuming from Redis and retries publishing to\n  RabbitMQ\n- On shutdown, a global event object is used and it dumps any messages from its\n  failsafe queue to a dump file\n- On connectivity failures, it tries to reconnect to either Redis or RabbitMQ\n\n## Naming conventions and assumptions\n\n- AMQP heartbeat is assumed at 100s.\n- AMQP exchange routing key string is of format: hostname.queuename.log\n- For every RabbitMQ queue named `q` that is to be transported, the corresponding\n  Redis list name is assumed as `queue:q`\n- Any service that is populating data in Redis should push in the corresponding\n  list(s) using RPUSH, `agentredrabbit` uses LPUSH, LPOP, LTRIM, LRANGE commands\n  internally\n\n## Building\n\nDependencies:\n\n- pika\n- redis (with hiredis for fast parsing)\n- nose, mock (for tests)\n- Local outbound SMTP server for sending email notifications\n\nThese are taken care of by the installation procedure.\n\n### Doc generation\n\nTo generate documentation of the code:\n\n    $ make docs\n\nThis requires `epydoc` and will generate documentation in docs/agentredrabbit.\n\n### Tests\n\nTo run unit tests:\n\n    $ make runtests\n\n### Installation\n\nTo install agentredrabbit, simply:\n\n    $ make build\n    $ make install\n\n## Usage\n\nAfter installation, get help docs:\n\n    $ agentredrabbit --help\n\nRun with a configuration:\n\n    $ agentredrabbit --config /path/to/conf.cfg\n\nExample of running as a background process manually:\n\n    $ agentredrabbit -c /path/to/cfg \u003e agent.log 2\u003e agent.log \u0026\n\nUsing init.d script for process management after `make install`:\n\n    $ /etc/init.d/agentredrabbit {start|stop|restart}\n\n## Author\n\nRohit \"[bhaisaab](http://bhaisaab.org)\" Yadav, rohit.yadav@wingify.com\n\n## Contributing\n\nSend a pull request on the GitHub repository: `https://github.com/wingify/agentredrabbit`\n\nYou may contact the author and Wingify's engineering team on engineering@wingify.com\n\n## Acknowledgements and Attributions\n\n[Example code of Async publisher using `pika`](https://pika.readthedocs.org/en/latest/examples/asynchronous_publisher_example.html)\n\n## Copyright and License\n\nLicensed under the opensource permissive MIT license.\n\nCopyright 2013 Rohit \"[bhaisaab](http://bhaisaab.org)\" Yadav, [Wingify](http://engineering.wingify.com/opensource)\n\nPermission is hereby granted, free of charge, to any person obtaining\na copy of this software and associated documentation files (the\n\"Software\"), to deal in the Software without restriction, including\nwithout limitation the rights to use, copy, modify, merge, publish,\ndistribute, sublicense, and/or sell copies of the Software, and to\npermit persons to whom the Software is furnished to do so, subject to\nthe following conditions:\n\nThe above copyright notice and this permission notice shall be\nincluded in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\nEXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\nMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\nNONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\nLIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION\nWITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwingify%2Fagentredrabbit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwingify%2Fagentredrabbit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwingify%2Fagentredrabbit/lists"}