{"id":13815217,"url":"https://github.com/rednafi/hook-slinger","last_synced_at":"2025-04-06T00:08:20.756Z","repository":{"id":36980087,"uuid":"387929396","full_name":"rednafi/hook-slinger","owner":"rednafi","description":"A generic service to send, retry, and manage webhooks","archived":false,"fork":false,"pushed_at":"2025-03-01T03:51:16.000Z","size":2997,"stargazers_count":114,"open_issues_count":5,"forks_count":11,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-29T23:08:23.903Z","etag":null,"topics":["docker","docker-compose","fastapi","http2","httpx","message-broker","message-queue","python","redis","request","rich","rq","saas","webhook"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rednafi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-20T22:37:17.000Z","updated_at":"2025-03-18T22:28:34.000Z","dependencies_parsed_at":"2023-01-17T11:30:11.713Z","dependency_job_id":"5927803a-e5b6-4f67-92ad-15d19433398d","html_url":"https://github.com/rednafi/hook-slinger","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rednafi%2Fhook-slinger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rednafi%2Fhook-slinger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rednafi%2Fhook-slinger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rednafi%2Fhook-slinger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rednafi","download_url":"https://codeload.github.com/rednafi/hook-slinger/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247415967,"owners_count":20935387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","fastapi","http2","httpx","message-broker","message-queue","python","redis","request","rich","rq","saas","webhook"],"created_at":"2024-08-04T04:03:08.801Z","updated_at":"2025-04-06T00:08:20.729Z","avatar_url":"https://github.com/rednafi.png","language":"Python","funding_links":[],"categories":["Python","saas"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\n![logo][logo]\n\n\u003cstrong\u003e\u003e\u003e \u003ci\u003eA generic service to send, retry, and manage webhooks.\u003c/i\u003e \u003c\u003c\u003c/strong\u003e\n\n\u003c/div\u003e\n\n## Description\n\n### What?\n\nHook Slinger acts as a simple service that lets you send, retry, and manage\nevent-triggered POST requests, aka webhooks. It provides a fully self-contained docker\nimage that is easy to orchestrate, manage, and scale.\n\n### Why?\n\nTechnically, a webhook is a mere POST request—triggered by a system—when a particular\nevent occurs. The following diagram shows how a simple POST request takes the webhook\nnomenclature when invoked by an event trigger.\n\n![Webhook Concept][webhook-concept]\n\nHowever, there are a few factors that make it tricky to manage the life cycle of a\nwebhook, such as:\n\n* Dealing with server failures on both the sending and the receiving end.\n* Managing HTTP timeouts.\n* Retrying the requests gracefully without overloading the recipients.\n* Avoiding retry loop on the sending side.\n* Monitoring and providing scope for manual interventions.\n* Scaling them quickly; either vertically or horizontally.\n* Decoupling webhook management logic from your primary application logic.\n\nProperly dealing with these concerns can be cumbersome; especially when sending webhooks\nis just another small part of your application and you just want it to work without you\nhaving to deal with all the hairy details every time. Hook Slinger aims to alleviate\nthis pain point.\n\n### How?\n\nHook Slinger exposes a single endpoint where you can post your webhook payload,\ndestination URL, auth details, and it'll make the POST request for you asynchronously in\nthe background. Under the hood, the service uses:\n\n* [FastAPI][fastapi] to provide a [Uvicorn][uvicorn] driven [ASGI][asgi] server.\n\n* [Redis][redis] and [RQ][rq] for implementing message queues that provide the\nasynchrony and robust failure handling mechanism.\n\n* [Rqmonitor][rqmonitor] to provide a dashboard for monitoring the status of the\nwebhooks and manually retrying the failed jobs.\n\n* [Rich][rich] to make the container logs colorful and more human friendly.\n\nThe simplified app architecture looks something this:\n\n![Topology][topology]\n\nIn the above image, the webhook payload is first sent to the `app` and the `app`\nleverages the `worker` instance to make the POST request. Redis DB is used for fast\nbookkeeping and async message queue implementation. The `monitor` instance provides a\nGUI to monitor and manage the webhooks. Multiple `worker` instances can be spawned to\nachieve linear horizontal scale-up.\n\n## Installation\n\n* Make sure you've got the latest version of [Docker][docker] and\n[Docker Compose V2][docker-compose] installed in your system.\n\n* Clone the repository and head over to the root directory.\n\n* To start the orchestra, run:\n\n    ```\n    make start-servers\n    ```\n\n    This will:\n\n    * Start an `app` server that can be accessed from port `5000`.\n\n    * Start an Alpine-based Redis server that exposes port `6380`.\n\n    * Start a single `worker` that will carry out the actual tasks.\n\n    * Start a `rqmonitor` instance that opens port `8899`.\n\n* To shut down everything, run:\n\n    ```\n    make stop-servers\n    ```\n\n*TODO: Generalize it more before making it installable with a `docker pull` command.*\n\n## Usage\n\n### Exploring the interactive API docs\n\nTo try out the entire workflow interactively, head over to the following URL on your\nbrowser:\n\n```\nhttp://localhost:5000/docs\n```\n\nYou should see a panel like this:\n\n![API Docs][api-docs]\n\nThis app implements a rudimentary token-based authentication system where you're\nexpected to send an API token by adding `Authorization: Token \u003ctoken_value\u003e` field to\nyour request header. To do that here, click the `POST /hook_slinger/` ribbon and that\nwill reveal the API description like this:\n\n![API Description][api-description]\n\nCopy the default token value from the description corpus, then click the green button on\nthe top right that says **Authorize**, and paste the value in the prompt box. Click\nthe **Authorize** button again and that'll conclude the login step. In your production\napplication, you should implement a robust authentication system or at least change this\ndefault token.\n\nTo send a webhook, you'll need a URL where you'll be able to make the POST request. For\nthis demonstration, let's pick this [webhook site][webhook-site-url] service to\nmonitor the received webhooks. It gives you a unique URL against which you'll be able to\nmake the post requests and monitor them in a dashboard like this:\n\n\n![Webhook Site][webhook-site]\n\nOn the API docs page, click the **Try it out** button near the **request body** section:\n\n![API Request][api-request]\n\nThis should reveal a panel like the following one where you can make your request:\n\n![API Request][api-request-2]\n\nNotice that the section is prefilled with an example request payload. You can use this\nexact payload to make a request. Go ahead and click the execute button. If you scroll\ndown a little, you'll notice the HTTP response:\n\n\n![API Response][api-response]\n\nNow, if you head over to the [webhook site][webhook-site-url-detail] URL, you should be\nable to see your API payload:\n\n\n![API Response][api-response-2]\n\nTo monitor the webhook tasks, head over to the following URL:\n\n```\nhttp://localhost:8899/\n```\n\nYou should be presented with a GUI like this:\n\n![RQ Monitor][rq-monitor]\n\nIf you click **Workers** on the left panel, you'll be presented with a panel where you\ncan monitor all the workers:\n\n![RQ Monitor][rq-monitor-2]\n\n\nThe **Jobs** panel lists all the tasks, and from there you'll be able to requeue a\nfailed job. By default, Hook Slinger retries a failed job 3 times with 5 seconds linear\nbackoff. However, this can be configured using environment variables in the `.env` file.\n\n![RQ Monitor][rq-monitor-3]\n\n\n### Sending a webhook via cURL\n\nRun the following command on your terminal; this assumes that you haven't changed the\nauth token (you should):\n\n```sh\ncurl -X 'POST' \\\n  'http://localhost:5000/hook_slinger/' \\\n  -H 'accept: application/json' \\\n  -H 'Authorization: Token $5$1O/inyTZhNvFt.GW$Zfckz9OL.lm2wh3IewTm8YJ914wjz5txFnXG5XW.wb4' \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n  \"to_url\": \"https://webhook.site/b30da7ce-c3cc-47e2-b2ae-68747b3d7789\",\n  \"to_auth\": \"\",\n  \"tag\": \"Dhaka\",\n  \"group\": \"Bangladesh\",\n  \"payload\": {\n    \"greetings\": \"Hello, world!\"\n  }\n}' | python -m json.tool\n```\n\nYou should expect the following output:\n\n```json\n{\n    \"status\": \"queued\",\n    \"ok\": true,\n    \"message\": \"Webhook registration successful.\",\n    \"job_id\": \"Bangladesh_Dhaka_a07ca786-0b7a-4029-bac0-9a7c6eb68a98\",\n    \"queued_at\": \"2021-11-06T16:54:54.728999\"\n}\n```\n\n### Sending a webhook via Python\n\nFor this purpose, you can use an HTTP library like [httpx][httpx].\n\nMake the request with the following script:\n\n```python\nimport asyncio\nfrom http import HTTPStatus\nfrom pprint import pprint\n\nimport httpx\n\n\nasync def send_webhook() -\u003e None:\n    wh_payload = {\n        \"to_url\": \"https://webhook.site/b30da7ce-c3cc-47e2-b2ae-68747b3d7789\",\n        \"to_auth\": \"\",\n        \"tag\": \"Dhaka\",\n        \"group\": \"Bangladesh\",\n        \"payload\": {\"greetings\": \"Hello, world!\"},\n    }\n\n    async with httpx.AsyncClient(http2=True) as session:\n        headers = {\n            \"Content-Type\": \"application/json\",\n            \"Authorization\": (\n                \"Token $5$1O/inyTZhNvFt.GW$Zfckz9OL.lm2wh3IewTm8YJ914wjz5txFnXG5XW.wb4\"\n            ),\n        }\n\n        response = await session.post(\n            \"http://localhost:5000/hook_slinger\",\n            headers=headers,\n            json=wh_payload,\n            follow_redirects=True,\n        )\n\n        # Hook Slinger returns http code 202, accepted, for a successful request.\n        assert response.status_code == HTTPStatus.ACCEPTED\n        result = response.json()\n        pprint(result)\n\n\nif __name__ == \"__main__\":\n    asyncio.run(send_webhook())\n```\n\nThis should return a similar response as before:\n\n```\n{\n    'job_id': 'Bangladesh_Dhaka_139fc35a-d2a5-4d01-a6af-e980c52f55bc',\n    'message': 'Webhook registration successful.',\n    'ok': True,\n    'queued_at': '2021-07-23T20:15:04.389690',\n    'status': 'queued'\n}\n```\n\n### Exploring the container logs\n\nHook Slinger overloads the Python root logger to give you a colorized and user-friendly\nlogging experience. To explore the logging messages of the application server, run:\n\n```\nmake app-logs\n```\n\nNotice the colorful logs cascading down from the app server:\n\n![App Logs][app-logs]\n\nNow, to explore the worker instance logs, in a separate terminal, run:\n\n```\nmake worker-logs\n```\n\nYou should see something like this:\n\n![Worker Logs][worker-logs]\n\n\n### Scaling up the service\n\nHook Slinger offers easy horizontal scale-up, powered by the `docker-compose --scale`\ncommand. In this case, scaling up means, spawning new workers in separate containers.\nLet's spawn 3 worker containers this time. To do so, first shut down the orchestra by\nrunning:\n\n```\nmake stop-servers\n```\n\nNow, run:\n\n```\nmake worker-scale n=3\n```\n\nThis will start the **App server**, **Redis DB**, **RQmonitor**, and 3 **Worker**\ninstances. Spawning multiple worker instances are a great way to achieve job concurrency\nwith the least amount of hassle.\n\n### Troubleshooting\n\nOn the Rqmonitor dashboard, if you see that your webhooks aren't reaching the\ndestination, make sure that the destination URL in the webhook payload can accept the\nPOST requests sent by the workers. Your webhook payload looks like this:\n\n```\n{\n    \"to_url\": \"https://webhook.site/f864d28d-9162-4ad5-9205-458e2b561c07\",\n    \"to_auth\": \"\",\n    \"tag\": \"Dhaka\",\n    \"group\": \"Bangladesh\",\n    \"payload\": {\"greetings\": \"Hello, world!\"},\n}\n\n```\n\nHere, `to_url` must be able to receive the payloads and return HTTP code 201.\n\n## Philosophy \u0026 limitations\n\nHooks Slinger is designed to be simple, transparent, upgradable, and easily extensible\nto cater to your specific needs. It's not built around AMQP compliant message queues\nwith all the niceties and complexities that come with them—this is intentional.\n\nAlso, if you scrutinize the end-to-end workflow, you'll notice that it requires making\nHTTP requests from the sending service to the Hook Slinger. This inevitably adds another\npoint of failure. However, from the sending service's POV, it's sending the HTTP\nrequests to a single service, and the target service is responsible for fanning out the\nwebhooks to the destinations. The developers are expected to have control over both\nservices, which theoretically should mitigate the failures. The goal is to transfer\nsome of the code complexity around managing webhooks from the sending service over to\nthe Hook Slinger. Also, I'm playing around with some of the alternatives to using HTTP\nPOST requests to send the payloads from the sending end to the Hook Slinger. Suggestions\nare always appreciated.\n\n\n\u003cdiv align=\"center\"\u003e\n\u003ci\u003e ✨ 🍰 ✨ \u003c/i\u003e\n\u003c/div\u003e\n\n[logo]: https://user-images.githubusercontent.com/30027932/126405827-8b859b4c-89cd-40c8-a7d3-fe6e9fc64770.png\n[forthebadge]: https://forthebadge.com\n[black-magic-badge]: https://forthebadge.com/images/badges/powered-by-black-magic.svg\n[build-with-love-badge]: https://forthebadge.com/images/badges/built-with-love.svg\n[made-with-python-badge]: https://forthebadge.com/images/badges/made-with-python.svg\n[webhook-concept]: ./art/webhook_concept.png\n[fastapi]: https://fastapi.tiangolo.com/\n[uvicorn]: https://www.uvicorn.org/\n[asgi]: https://asgi.readthedocs.io/en/latest/#\n[redis]: https://redis.io/\n[rq]: https://python-rq.org/docs/jobs/\n[rqmonitor]: https://github.com/pranavgupta1234/rqmonitor\n[rich]: https://github.com/willmcgugan/rich\n[topology]: ./art/topology.png\n[docker]: https://www.docker.com/\n[docker-compose]: https://docs.docker.com/compose/cli-command/\n[api-docs]: ./art/api_docs.png\n[api-description]: ./art/api_descr.png\n[webhook-site]: ./art/webhook_site.png\n[webhook-site-url]: https://webhook.site/\n[webhook-site-url-detail]: https://webhook.site/#!/f864d28d-9162-4ad5-9205-458e2b561c07\n[api-request]: ./art/api_request.png\n[api-request-2]: ./art/api_request_2.png\n[api-response]: ./art/api_response.png\n[api-response-2]: ./art/api_response_2.png\n[rq-monitor]: ./art/rq_monitor.png\n[rq-monitor-2]: ./art/rq_monitor_2.png\n[rq-monitor-3]: ./art/rq_monitor_3.png\n[app-logs]: ./art/app_logs.png\n[worker-logs]: ./art/worker_logs.png\n[httpx]: https://www.python-httpx.org\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frednafi%2Fhook-slinger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frednafi%2Fhook-slinger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frednafi%2Fhook-slinger/lists"}