{"id":23479413,"url":"https://github.com/thethingsnetwork/lorawan-stack-application-cookbook","last_synced_at":"2025-10-06T15:16:44.658Z","repository":{"id":39994639,"uuid":"439332403","full_name":"TheThingsNetwork/lorawan-stack-application-cookbook","owner":"TheThingsNetwork","description":"A guide for building an application or integration for The Things Stack. Work in progress. 🚧","archived":false,"fork":false,"pushed_at":"2022-05-19T13:35:02.000Z","size":48,"stargazers_count":8,"open_issues_count":7,"forks_count":1,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-13T19:11:26.681Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TheThingsNetwork.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-17T12:55:54.000Z","updated_at":"2022-10-18T23:55:19.000Z","dependencies_parsed_at":"2022-09-15T22:00:17.714Z","dependency_job_id":null,"html_url":"https://github.com/TheThingsNetwork/lorawan-stack-application-cookbook","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TheThingsNetwork/lorawan-stack-application-cookbook","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheThingsNetwork%2Florawan-stack-application-cookbook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheThingsNetwork%2Florawan-stack-application-cookbook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheThingsNetwork%2Florawan-stack-application-cookbook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheThingsNetwork%2Florawan-stack-application-cookbook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TheThingsNetwork","download_url":"https://codeload.github.com/TheThingsNetwork/lorawan-stack-application-cookbook/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheThingsNetwork%2Florawan-stack-application-cookbook/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278631019,"owners_count":26018874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-24T19:29:15.915Z","updated_at":"2025-10-06T15:16:44.641Z","avatar_url":"https://github.com/TheThingsNetwork.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The Things Stack Application Cookbook\n\nThis is a guide for building an application or integration for The Things Stack.\n\n\u003e 🚧 This guide is work in progress.\n\nThis guide follows a number of best practices that will help you build scalable, reliable and yet simple applications:\n\n- **Use Webhooks** to receive uplink traffic from The Things Stack. Webhooks require less resources than having to keep a connection to The Things Stack open at all times, or constantly polling The Things Stack for updates.\n- **Decouple ingest from processing** with pub/sub queues. This allows you to add/remove additional processing pipelines without interrupting your other processing pipelines. When updating a processing pipeline, messages will be queued until your update processing pipeline is up and running.\n\nThe architecture of the application we'll build looks as follows:\n\n![overview](./img/overview.mmd.svg)\n\n- The Things Stack will send Webhooks to our server.\n- A reverse proxy ([NGINX](https://nginx.org/)) forwards these Webhooks to an \"ingest\" pipeline ([Benthos](https://www.benthos.dev/)).\n- The \"ingest\" pipeline checks the API key in the request, drops invalid Webhooks and publishes valid Webhooks on a Pub/Sub system ([Redis](https://redis.io/), [AMQP (RabbitMQ)](https://www.rabbitmq.com/) or [MQTT (Mosquitto)](https://mosquitto.org/)).\n- The \"process\" pipeline (also [Benthos](https://www.benthos.dev/)) subscribes to the Pub/Sub system, de-duplicates messages (we don't want to process retries), and writes the messages to \"outputs\" (in the first version of this guide, just to stdout).\n\n## Prerequisites\n\n- A server with a public IP address.\n  - It is **highly recommended** to use a hosted server. Unless you really know what you're doing, you probably shouldn't use a self-hosted server at home, behind a NAT or with a dynamic IP.\n  - This guide was written while using a **DEV1-S** instance from [Scaleway](https://www.scaleway.com/) running **Ubuntu 20.04**.\n- A domain name with a DNS record pointing at that public IP address.\n  - This guide will use `placeholder.example.com` as a placeholder, and `integration.example.com` as \"your domain\".\n\n## Basic Server Setup\n\nConnect to the server over SSH:\n\n```bash\n% ssh username@integration.example.com\n```\n\nMake sure the system is up to date:\n\n```bash\n% sudo apt-get update\n\n% sudo apt-get dist-upgrade\n\n% sudo snap install core\n\n% sudo snap refresh core\n```\n\n\u003e ℹ️ You may want to reboot your server at this point.\n\nNext, install some packages that we'll need to install the rest of the server:\n\n```bash\n% sudo apt-get install ca-certificates curl git gnupg lsb-release\n```\n\n## Downloading the Cookbook on the Server\n\nIt will be useful to have this guide and example files on your server, so let's download the entire cookbook.\n\n```\n% git clone https://github.com/htdvisser/lorawan-stack-application-cookbook cookbook\n```\n\n## Installing NGINX\n\n```bash\n% curl -fsSL https://nginx.org/keys/nginx_signing.key \\\n    | sudo gpg --dearmor -o /usr/share/keyrings/nginx-archive-keyring.gpg\n\n% echo \"deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \\\nhttp://nginx.org/packages/ubuntu `lsb_release -cs` nginx\" \\\n    | sudo tee /etc/apt/sources.list.d/nginx.list \u003e /dev/null\n\n% sudo apt-get update\n\n% sudo apt-get install nginx\n\n% sudo systemctl enable nginx --now\n```\n\n## Installing Certbot\n\n```bash\n% sudo snap install --classic certbot\n```\n\n## Configuring NGINX and Certbot\n\nWe'll set up NGINX as a reverse proxy, with HTTPS certificates from Let's Encrypt that we'll request with Certbot.\n\n\u003e ℹ️ _tl;dr_ if you are already familiar with NGINX:\n\u003e - Proxy `https://integration.example.com/post/*` to `http://127.0.0.1:4195`\n\nCopy the NGINX configuration from the cookbook directory to the NGINX config directory:\n\n```bash\n% sudo cp cookbook/nginx/nginx.conf /etc/nginx/nginx.conf\n```\n\nCopy the HTTP configuration from the cookbook directory to the NGINX config directory:\n\n```bash\n% sudo cp cookbook/nginx/conf.d/integration-http.conf \\\n  /etc/nginx/conf.d/integration-http.conf\n```\n\nReplace occurrences of `placeholder.example.com` in the HTTP configuration with your domain name (`integration.example.com`):\n\n```bash\n% sudo sed -i 's/placeholder.example.com/integration.example.com/g' \\\n  /etc/nginx/conf.d/integration-http.conf\n```\n\nTest the config, and reload NGINX:\n\n```bash\n% sudo nginx -t \u0026\u0026 sudo systemctl reload nginx\n```\n\nUse Certbot to request a Let's Encrypt certificate.\n\n\u003e ⚠️ Make sure to replace `integration.example.com` and `hello@example.com` before running this command:\n\n```bash\ncertbot certonly \\\n  -n --agree-tos --force-renewal \\\n  --webroot -w /var/www/_letsencrypt \\\n  -d integration.example.com \\\n  --email hello@example.com --no-eff-email\n```\n\nNow we can copy the HTTPS configuration from the cookbook directory to the NGINX config directory:\n\n```bash\n% sudo cp cookbook/nginx/conf.d/integration-https.conf \\\n  /etc/nginx/conf.d/integration-https.conf\n```\n\nReplace occurrences of `placeholder.example.com` in the HTTPS configuration with your domain name (`integration.example.com`):\n\n```bash\n% sudo sed -i 's/placeholder.example.com/integration.example.com/g' \\\n  /etc/nginx/conf.d/integration-https.conf\n```\n\nGenerate Diffie-Hellman keys on your server:\n\n```bash\n% sudo openssl dhparam -out /etc/nginx/dhparam.pem 2048\n```\n\nAgain, we test the config, and reload NGINX:\n\n```bash\n% sudo nginx -t \u0026\u0026 sudo systemctl reload nginx\n```\n\nIf everything worked, you'll see:\n\n- A **404 Not Found** on https://integration.example.com/\n- A **502 Bad Gateway** on https://integration.example.com/post\n\nFinally, in order to make Certbot automatically reload NGINX when it updates certificates, copy `reload.sh` from the cookbook directory to the Certbot hooks directory:\n\n```bash\n% sudo cp cookbook/nginx/reload.sh /etc/letsencrypt/renewal-hooks/post/nginx-reload.sh\n```\n\n## Installing Docker and Docker-Compose\n\n```bash\n% curl -fsSL https://download.docker.com/linux/ubuntu/gpg \\\n    | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg\n\n% echo \"deb [signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \\\nhttps://download.docker.com/linux/ubuntu `lsb_release -cs` stable\" \\\n    | sudo tee /etc/apt/sources.list.d/docker.list \u003e /dev/null\n\n% sudo apt-get update\n\n% sudo apt-get install docker-ce docker-ce-cli containerd.io\n\n% sudo systemctl enable docker containerd --now\n\n% sudo usermod -aG docker $USER \u0026\u0026 newgrp docker\n\n% mkdir -p ~/.docker/cli-plugins/\n\n% curl -fsSL https://github.com/docker/compose/releases/download/v2.1.1/docker-compose-linux-x86_64 \\\n    -o ~/.docker/cli-plugins/docker-compose\n\n% chmod +x ~/.docker/cli-plugins/docker-compose\n```\n\nIf everything worked, you can now run `docker run --rm hello-world` and you'll see something like:\n\n```\n[...]\nHello from Docker!\nThis message shows that your installation appears to be working correctly.\n[...]\n```\n\n## Ingest Pipeline\n\nThe ingest pipeline is responsible for accepting Webhooks from The Things Stack, doing some basic filtering, and then publishing the messages on Pub/Sub.\n\n![ingest](./img/ingest.mmd.svg)\n\nThe ingest pipeline consists of two files:\n\n- `ingest/benthos.yaml` defines the configuration for our Benthos pipeline.\n- `ingest/compose.yaml` defines the containers that will run in the deployment: Benthos, Redis, RabbitMQ and Mosquitto.\n\nEach of these files contains blocks that can be commented out if you only want to use one pub/sub system.\n\nBefore we can deploy the pipeline, we need to generate an API key that we'll give to The Things Stack to authenticate webhooks:\n\n```bash\n% openssl rand -base64 24 | tee cookbook/ingest/apikey.txt\n```\n\nNow let's deploy the ingest pipeline:\n\n```bash\n% cd cookbook/ingest\n\n% docker compose up -d\n```\n\n## Processing Pipelines\n\nThe processing pipelines are responsible for subscribing to Pub/Sub, and actually processing each message.\n\n![process](./img/process.mmd.svg)\n\nThere are currently three processing pipelines:\n\n- `process-stdout`: Writes messages to standard output. This pipeline also shows how to subscribe to RabbitMQ and MQTT. The other pipelines only use Redis Streams.\n- `process-clickhouse`: Writes messages to a [Clickhouse](https://clickhouse.com/) database.\n- `process-mongo`: Writes messages to a [Mongo](https://www.mongodb.com/) database.\n\nThe processing pipelines consist of two files:\n\n- `process-*/benthos.yaml` defines the configuration for our Benthos pipeline.\n- `process-*/compose.yaml` defines the containers that will run in the deployment: for now just Benthos.\n\nLet's deploy the processing pipeline that writes to standard output:\n\n```bash\n% cd cookbook/process-stdout\n\n% docker compose up -d\n```\n\nYou can now start following the logs to see messages come in when you enable the webhook integration in The Things Stack:\n\n```bash\n% docker compose logs -f\n```\n\n## Configuring the Webhook in The Things Stack\n\nIn The Things Stack, go to your application, then **Integrations** and **Webhooks**.\n\nClick **Add Webhook**, then **Custom webhook** and configure it as follows:\n\n- Choose a **Webhook ID**.\n- Set the **Webhook format** to **JSON**.\n- The **Base URL** is `https://integration.example.com/post`.\n- Add an **Additional header** `X-APIKey` with as value the API key that we generated into `cookbook/ingest/apikey.txt` earlier.\n- Under **Enabled messages** we'll only enable **Uplink message** and give it a path of `/up`.\n\nNow save the webhook configuration.\n\nIf you have active end devices in your application, you'll now start seeing messages come in. Otherwise you can go to a device in the Console of The Things Stack and simulate uplinks from the **Messaging** tab.\n\n## Future Work\n\nSee [enhancement issues](https://github.com/htdvisser/lorawan-stack-application-cookbook/labels/enhancement).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthethingsnetwork%2Florawan-stack-application-cookbook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthethingsnetwork%2Florawan-stack-application-cookbook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthethingsnetwork%2Florawan-stack-application-cookbook/lists"}