{"id":18711872,"url":"https://github.com/algolia/discourse-webhook-collector","last_synced_at":"2025-06-19T15:44:25.246Z","repository":{"id":66050818,"uuid":"93891474","full_name":"algolia/discourse-webhook-collector","owner":"algolia","description":"Listen to Discourse events and send to other APIs using webtask.io","archived":false,"fork":false,"pushed_at":"2023-03-03T09:20:46.000Z","size":49,"stargazers_count":22,"open_issues_count":4,"forks_count":6,"subscribers_count":73,"default_branch":"master","last_synced_at":"2025-04-25T11:53:16.808Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/algolia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-09T19:41:37.000Z","updated_at":"2024-07-15T13:49:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"5e51e95b-7883-49a5-81b2-6afd943691d0","html_url":"https://github.com/algolia/discourse-webhook-collector","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/algolia/discourse-webhook-collector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/algolia%2Fdiscourse-webhook-collector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/algolia%2Fdiscourse-webhook-collector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/algolia%2Fdiscourse-webhook-collector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/algolia%2Fdiscourse-webhook-collector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/algolia","download_url":"https://codeload.github.com/algolia/discourse-webhook-collector/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/algolia%2Fdiscourse-webhook-collector/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260781825,"owners_count":23062310,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T12:40:42.763Z","updated_at":"2025-06-19T15:44:20.218Z","avatar_url":"https://github.com/algolia.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# discourse-webhook-collector\n\n[Discourse](https://discourse.org) has [webhooks](https://meta.discourse.org/t/setting-up-webhooks/49045). These are very helpful for connecting Discourse to other things so you can [scale your community support](https://devrel.net/developer-experience/scale-community-support-apis).\n\nThis repository contains a set of functions that catch Discourse webhooks, transform or enrich the JSON payload, and then call other downstream APIs and SaaS services.\n\n![discourse-webhook-collector architecture](https://cl.ly/2Q0o3w3T0i3S/Screenshot%202017-06-11%2013.35.56.png)\n\nSupport is currently included for [Slack](https://api.slack.com/), [Keen IO](https://keen.io/docs) and [HelpScout](http://developer.helpscout.net/help-desk-api/), but you can deploy as few or as many as you like. Connecting more services would not be hard - PR's are welcome!\n\nThese webhooks are designed to run on the [webtask.io](https://webtask.io/docs/101) function-as-a-service platform from [Auth0](https://auth0.com). Webtask functions are written in JavaScript and can take advantage of packages on NPM. If you don't have previous webtask experience, I would recommend [taking a tutorial](https://auth0.com/blog/building-serverless-apps-with-webtask/) before working with this repository.\n\n## Support\n\nThere are a few moving parts here. If you run into any trouble getting up and running, the Algolia team is happy to lend a hand. Just send an email to [community@algolia.com](mailto:community@algolia.com).\n\n## Prerequisites\n\nYou will need admin access to running Discourse instance and [wt-cli](https://github.com/auth0/wt-cli) with an active webtask profile. Depending on which webhooks you want to use, you will need access to projects and API keys for the various supported services.\n\nCommands like `wt ls` and `wt create` should be working in your console before you begin.\n\n## Getting Started\n\nThe `SendToConsole` task simply logs a few Discourse-specific HTTP headers and the JSON body to the console. It's best to make sure you can run this successfully before connecting other downstream APIs.\n\nTo do that, first clone this repo:\n\n```\ngit clone git@github.com:algolia/discourse-webhook-collector.git\n```\n\nInstall dependencies with yarn:\n\n```\nyarn\n```\n\nCreate an empty `.secrets.development` file, as you do not need to specify any API keys just to log to the console.\n\nDeploy a function named `SendToConsoleDev` to webtask using the task shortcut in package.json.\n\n```\nyarn run dev-send-to-console\n```\n\nThe shortcut is just a wrapper for this wt-cli command:\n\n```\n# you don't need to run this\nwt create --name SendToConsoleDev --bundle --no-parse --secrets-file .secrets.development --watch SendToConsole.js\n```\n\nHere's a breakdown of the command:\n\n- `--no-parse` is used to work around webtask HTTP payload size limits\n- `--bundle` allows us to put code in multiple files to make development cleaner\n- `--name` is the name of the webtask. It's very helpful two have two versions of each function deployed at the same time - one for live development and one for production. The development version webtask names are always suffixed with \"Dev\" and use `.secrets.development` instead of `.secrets.production` so you can specify different non-production downstream targets, like a Slack channel or Keen IO project that's used just for testing.\n- `--watch` keeps the process attached to the webtask, so you can see logs when it's invoked and so that any code changes you make are automatically uploaded (this is especially why you shouldn't use it for production).\n\nNow that our `SendToConsoleDev` webtask has been created, let's test it with cURL. In a new shell, export a `url` variable that points to your webtask profile domain:\n\n```\nexport WEBTASK_URL=https://\u003cyour-subdomain\u003e.\u003cregion-subdomain\u003e.webtask.io\n```\n\nNow, run a command shortcut that will send a cURL request to your new webtask using a JSON file in the `test` directory of this repository:\n\n```\nyarn run test-send-to-console\n```\n\nWhich is just a shortcut for:\n\n```\n# you don't need to run this\ncurl -X POST $WEBTASK_URL/SendToConsoleDev --data '@./test/DiscourseTopicEvent.json' --header 'Content-Type: application/json' --header 'x-discourse-event-type: topic' --header 'x-discourse-event: topic_created'\n```\n\nYou should see this output in the log of your webtask:\n\n```\nSendToConsole - Received webhook\nSendToConsole - Discourse event: topic_created\nSendToConsole - Discourse event type: topic\nSendToConsole - JSON payload:\n{\n \"topic\": {\n  \"id\": 1608\n }\n}\nSendToConsole Success\n```\n\nIf you do, that means that your webtask setup is working properly. Hooray!\n\n## Connecting Discourse\n\nNow you're ready to add webhooks to your Discourse instance. In your Discourse admin UI, navigate to API, then webhooks, then click the button for \"New webhook\".\n\nLet's create a webhook that points to our `SendToConsoleDev` webtask. The Payload URL is the key field to populate, substituting the right values for your webtask domain:\n\n![Discourse create webhook UI](https://cl.ly/1b3h371R2k06/Screenshot%202017-06-11%2012.27.57.png)\n\n Choose \"Send me everything\" from the event types section or select just the subset of events that you'd like to test with. Check the \"Active\" checkbox and then click \"Create\". Click \"Go to events\" and then click the \"Ping\" button. You should see the following in your webtask log:\n\n```\nSendToConsole - Received webhook\nSendToConsole - Discourse event: ping\nSendToConsole - Discourse event type: ping\nSendToConsole - JSON payload:\n{\n \"ping\": \"OK\"\n}\nSendToConsole Success\n```\n\nIf you see this in the logs, your Discourse can successfully send events to your webtasks. Yay! Depending on what event types you chose, you will start seeing output when topics, posts and user events happen.\n\n## Connecting APIs\n\nJust logging to the console isn't very interesting. Calling APIs would definitely be more interesting.\n\n### Prerequisite - Discourse API access\n\nDiscourse webhook JSON payloads do not always contain all of the information that we want and contents vary greatly based on the event type. That makes it difficult to keep event schema consistent for forwarding to analytics services like Keen IO.\n\nThe solution that discourse-webhook-collector proposes is not to patch Discourse or use a plugin (at least not yet) but to use the Discourse API to fetch the missing JSON. Do this, discourse-webhook-collector needs to know the location of your Discourse instance and have an API key with admin access.\n\nTo find or generate a Discourse API key, navigate to \"API\" and then \"API\" in the submenu that appears.\n\nPut the API key and the domain of your Discourse in your `.secrets.development` and `.secrets.production` files, like so:\n\n```\nDISCOURSE_URL=https://\u003cmy-discourse-domain\u003e\nDISCOURSE_API_KEY=\u003cmy-discourse-api-key\u003e\n```\n\n### Slack\n\nBefore beginning, create an [incoming webhook](https://api.slack.com/incoming-webhooks) using the Slack API pointing to the channel where you want Discourse activity to go to. Copy the webhook URL and add it to the secrets files. (You may want to use different values for `.secrets.development` and `.secrets.production` if you'd like to separate your testing from live messages.)\n\n```\nSEND_TO_SLACK_WEBHOOK_URL=\u003cmy-slack-webhook-url\u003e\n```\n\nNext, deploy the development version of the `SendToSlack` webtask:\n\n```\nyarn run dev-send-to-slack\n```\n\nNext, add another webhook to your Discourse. Instead of pointing to the `SendToConsoleDev` path in the Payload URL field, use `SendToSlackDev` instead.\n\nNow, perform an action on your Discourse like creating a new topic. A new message should post to the Slack channel you configured in your incoming webhook.\n\nTo deploy a production version (the only difference being that it uses the `.secrets.production` file instead), run the deploy command:\n\n```\nyarn run deploy-send-to-slack\n```\n\nAfter doing that, add a new webhook in your Discourse that points to `SendToSlack` in the payload URL field. You can leave the development webhook active or make it inactive, that's up to you. The main idea is to keep the production path separate so you can develop and test new webhook functionality without impacting production.\n\n### Keen IO\n\nThe instructions are essentially the same as for Slack. These are the values you'll need to put in the secrets files:\n\n```\nKEEN_PROJECT_ID=\u003cmy-project-id\u003e\nKEEN_WRITE_API_KEY=\u003cmy-write-api-key\u003e\n```\n\nDouble-check to make sure you're using an API key that has write access.\n\nThe commands to create the development and production versions of the webtask are as follows:\n\n```\nyarn run dev-send-to-keen\nyarn run deploy-send-to-keen\n```\n\nThe corresponding webtask names to put in your Discourse Payload URL are `SendToKeenDev` and `SendToKeen`.\n\nIf you want a head start on creating Keen IO dashboards for your Discourse activity, have a look at [vue-keen-dashboards](https://github.com/algolia/vue-keen-dashboards). Includes authentication and deploys easily to Netlify.\n\n### HelpScout\n\nThese are the secrets values that are required:\n\n```\nHELPSCOUT_MAILBOX_ID=\u003cmy-mailbox-id\u003e\nHELPSCOUT_API_KEY=\u003cmy-api-key\u003e\n```\n\nSee the [HelpScout API documentation](http://developer.helpscout.net/help-desk-api/) for information on how to provision API keys and get mailbox IDs.\n\nThese are the commands to create the webtasks:\n\n```\nyarn run dev-send-to-helpscout\nyarn run deploy-send-to-helpscout\n```\n\nThe corresponding webtask names to put in your Discourse Payload URL are `SendToHelpScoutDev` and `SendToHelpScout`.\n\n### Contributing\n\nAll contributions are welcome. If there is another API or service that you are using with Discourse, feel free to add it here using the current conventions and submit a PR. If you have a question, please open an issue. Thanks!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falgolia%2Fdiscourse-webhook-collector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falgolia%2Fdiscourse-webhook-collector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falgolia%2Fdiscourse-webhook-collector/lists"}