{"id":16729182,"url":"https://github.com/keithduncan/logplex-statsd","last_synced_at":"2025-07-08T08:35:30.776Z","repository":{"id":144668848,"uuid":"47550116","full_name":"keithduncan/logplex-statsd","owner":"keithduncan","description":"Report Heroku platform errors to a statsd collector for aggregation and storage.","archived":false,"fork":false,"pushed_at":"2015-12-16T02:09:10.000Z","size":67,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-15T16:37:23.128Z","etag":null,"topics":["heroku","logplex","metrics","statsd-client"],"latest_commit_sha":null,"homepage":"","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/keithduncan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-12-07T12:14:06.000Z","updated_at":"2019-02-08T12:03:10.000Z","dependencies_parsed_at":"2023-03-22T08:00:21.298Z","dependency_job_id":null,"html_url":"https://github.com/keithduncan/logplex-statsd","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/keithduncan/logplex-statsd","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Flogplex-statsd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Flogplex-statsd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Flogplex-statsd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Flogplex-statsd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/keithduncan","download_url":"https://codeload.github.com/keithduncan/logplex-statsd/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keithduncan%2Flogplex-statsd/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264232317,"owners_count":23576816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["heroku","logplex","metrics","statsd-client"],"created_at":"2024-10-12T23:27:24.594Z","updated_at":"2025-07-08T08:35:30.746Z","avatar_url":"https://github.com/keithduncan.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# logplex-statsd\n\nReport Heroku platform errors to a statsd collector for aggregation and storage.\n\n## Summary\n\nHeroku injects platform errors into your application’s Logplex stream. You can view\nthe last 24hrs worth of these errors in the application’s dashboard but can’t build\non top of them.\n\nIf you want to graph these error trends over a longer time scale, apply functions to them\nor configure nagios alerts based on their occurrence, you need to dump them into a database.\n\nlogplex-statsd is a Heroku deployable application that can be configured as a log drain\ndestination for your other Heroku applications. It will parse these errors from the logs\nand egress them over UDP into persistent storage like Graphite’s [carbon](https://github.com/graphite-project/)\nfor analysis.\n\n## Components\n\nlogplex-statsd builds on two primary components:\n\n- [keithduncan/logplex-parse](https://github.com/keithduncan/logplex-parse)\n- [keithduncan/statsd-client](https://github.com/keithduncan/statsd-client)\n\nThese are tied together using [Scotty](http://github.com/scotty-web/scotty) to receive, authenticate\nand respond to incoming logs from Logplex.\n\n## Development\n\n`script/development-bootstrap` checks your machine for the required components,\ncreates a cabal sandbox and installs the dependencies.\n\nlogplex-statsd uses `foreman` in development to run the processes for testing.\nRun `script/debug-server` to start the web server and two statsd listener\nprocesses.\n\nRunning `script/debug-server` you should see output like this when sending\nlogplex documents to localhost:\n\n```\nKeiths-MacBook-Pro:logplex-statsd keith$ script/debug-server\n20:18:32 web.1           | started with pid 19983\n20:18:32 listener-8126.1 | started with pid 19984\n20:18:32 listener-8127.1 | started with pid 19985\n20:18:32 listener-8126.1 | Listening on port 8126...\n20:18:32 listener-8127.1 | Listening on port 8127...\n20:18:32 web.1           | Preprocessing executable 'logplex-statsd' for logplex-statsd-0.1.0.0...\n20:18:33 web.1           | Running logplex-statsd...\n20:18:34 web.1           | Setting phasers to stun... (port 3000) (ctrl-c to quit)\n20:18:34 web.1           | POST /heaven/logs\n20:18:34 web.1           |   Request Body: 106 \u003c10\u003e123 2015-12-03T23:12:17+11:00 keiths-macbook-pro.local logplex-parse 420 - [foo bar=\"baz\"] error=1234597 \u003c10\u003e123 2015-12-03T23:12:17+11:00 keiths-macbook-pro.local logplex-parse 420 - - Error R99 heroku97 \u003c10\u003e123 2015-12-03T23:12:17+11:00 keiths-macbook-pro.local logplex-parse 420 - - Error R89 heroku97 \u003c10\u003e123 2015-12-03T23:12:17+11:00 keiths-macbook-pro.local logplex-parse 420 - - Error L99 heroku122 \u003c10\u003e123 2015-12-03T23:12:17+11:00 keiths-macbook-pro.local logplex-parse 420 - - at=error code=H12 desc=\"a thing happened\"\n20:18:34 web.1           |   Accept:\n20:18:34 web.1           |   Status: 201 Created 0.026851s\n20:18:34 listener-8126.1 | [\"heaven.heroku.errors.R89:1|c\", #\u003cAddrinfo: 127.0.0.1:52122 UDP\u003e, 0]\n20:18:34 listener-8127.1 | [\"heaven.heroku.errors.R99:1|c\", #\u003cAddrinfo: 127.0.0.1:53383 UDP\u003e, 0]\n20:18:34 listener-8127.1 | [\"heaven.heroku.errors.L99:1|c\", #\u003cAddrinfo: 127.0.0.1:53383 UDP\u003e, 0]\n20:18:34 listener-8127.1 | [\"heaven.heroku.errors.H12:1|c\", #\u003cAddrinfo: 127.0.0.1:53383 UDP\u003e, 0]\n```\n\n## Deploy to Heroku\n\nBefore you can deploy to Heroku you need to familiarise yourself with the\nexpected configuration variables.\n\n### Metrics Cluster Configuration\n\nYou first need to configure where the UDP packets containing the statsd metrics\nshould be sent.\n\nThe `METRICS_CLUSTER` environment variable is a comma separated list of\ncollector URIs. It defaults to two localhost collectors which you will want to\nreplace with something useful.\n\nlogplex-statsd uses [keithduncan/statsd-client](https://github.com/keithduncan/statsd-client)\nwhich supports consistently routing metrics between multiple collectors using a\nmodulus of a `CRC32(stat name)` and the number of collectors. By distributing\nthe metrics it helps prevents any single metrics collector from becoming\noverloaded.\n\nA cluster of one host is an acceptable configuration, all stats\nwill be sent to the cluster's single member. Measuring the load of your\ncollector cluster will indicate whether the collector cluster should be scaled\nhorizontally.\n\n### Amazon AWS S3 Configuration\n\nDeployment currently uses Heroku’s buildpack system however the app cannot be\nbuilt inside the quotas applied to the git push process. Instead deployment will\nprepare a slug containing the buildpack source that can be used to build the\napp dependencies and upload these to Amazon S3 for reuse.\n\nA postdeploy script is used to download GHC and Cabal, update the Cabal package\nlist, create a Cabal sandbox and install the dependencies. These artefacts are\nthen uploaded to S3 for reuse on subsequent deploys.\n\nThis requires an AWS S3 bucket to be configured along with an AWS IAM user. The\nIAM user's access should be tightly scoped using an IAM policy such as this with\nthe `bucket_name` replaced:\n\n```\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:DeleteObject\",\n                \"s3:GetObject\",\n                \"s3:GetObjectAcl\",\n                \"s3:ListBucket\",\n                \"s3:PutObject\",\n                \"s3:PutObjectAcl\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::bucket_name*\"\n            ]\n        }\n    ]\n}\n```\n\nHeroku’s deployment process will prompt you to configure the required fields.\n\nOnce this postdeploy script has completed you will need to deploy the app again\nusing `git commit --amend --no-edit \u0026\u0026 git push --force heroku` to force a slug\ncontaining the newly built environment and the app to be generated, phew.\n\nThis build process is less than ideal and I plan to replace it with Heroku’s\nDocker based workflow.\n\n[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy)\n\n## Piping a Heroku application's logs to logplex-parse\n\nOnce you have logplex-statsd deployed you’ll want to configure your other apps\nto send their logs to logplex-statsd.\n\n1. Prepare a new set of per-app credentials\n    - `password=\"$(ruby -rsecurerandom -e '$stdout.puts SecureRandom.hex(40)')\"`\n2. Set these in the environment of your logplex-statsd\n    - `heroku config:set API_CREDENTIALS_MYAPP=\"logplex:$password\" --app my-logplex-statsd`\n3. Configure the drain for `MYAPP`, assuming your logplex-statsd is available over https\n    - `heroku drain:add \"https://logplex:${password}@$(heroku domains --app my-logplex-statsd --json | jq --raw-output .[0].hostname)/myapp/logs\" --app my-app`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeithduncan%2Flogplex-statsd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkeithduncan%2Flogplex-statsd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeithduncan%2Flogplex-statsd/lists"}