{"id":29248599,"url":"https://github.com/rstudio-tech/gost","last_synced_at":"2025-07-04T00:08:16.058Z","repository":{"id":302494982,"uuid":"898911178","full_name":"rstudio-tech/Gost","owner":"rstudio-tech","description":"Gost is a Go implementation of the StatsD daemon.","archived":false,"fork":false,"pushed_at":"2024-12-05T09:16:54.000Z","size":225,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-02T17:55:28.316Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rstudio-tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-05T09:16:13.000Z","updated_at":"2024-12-16T21:41:11.000Z","dependencies_parsed_at":"2025-07-02T18:05:37.046Z","dependency_job_id":null,"html_url":"https://github.com/rstudio-tech/Gost","commit_stats":null,"previous_names":["rstudio-tech/gost"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rstudio-tech/Gost","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rstudio-tech%2FGost","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rstudio-tech%2FGost/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rstudio-tech%2FGost/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rstudio-tech%2FGost/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rstudio-tech","download_url":"https://codeload.github.com/rstudio-tech/Gost/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rstudio-tech%2FGost/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263421932,"owners_count":23464051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-04T00:08:15.391Z","updated_at":"2025-07-04T00:08:16.022Z","avatar_url":"https://github.com/rstudio-tech.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gost\n\nGost is a Go implementation of the [StatsD](https://github.com/etsy/statsd/)\ndaemon.\n\n## Usage\n\nInstall from source:\n\n    go install github.com/cespare/gost@latest\n\nRun `gost` with a conf file.\n\n    gost -conf my/config.toml\n\nBy default it uses `conf.toml`. This repo includes a [`conf.toml`](conf.toml)\nthat should get you started. It has a lot of comments that explain what all the\noptions are.\n\n### Messages\n\nGost is largely statsd compatible and any statsd library you want should work\nwith it out of the box. The main API difference is that gauges cannot be delta\nvalues (they are always interpreted as absolute).\n\nFor completeness, here is a summary of the supported messages. All messages are\nsent via UDP to localhost on a port configured by the `port` setting in the\nconfig file. Typically each message is a UDP packet, but multiple messages can\nbe sent in a single packet by separating them with `\\n` characters.\n\nThere are two data types involved: **keys** and **values**. **keys** are ascii\nstrings (see the Key Format section below for details). **values** are\nhuman-printed floats:\n\n    /^[+\\-]?\\d+(\\.\\d+)?$/\n\nCounters have a sampling rate, which is the same format as a value. This tells\ngost that the counter is being sampled at some rate, and gost divides the\ncounter value by the sampling rate to obtain an estimate of the true value.\n\n**Counters**\n\nA counter records occurrences of some event, or other values that can be\naccumulated by summing them.\n\nFor each counter, gost records two metrics:\n\n* `count`: the raw counts (scaled for sample rate)\n* `rate`: the rate, per second\n\nSyntax: `\u003ckey\u003e:\u003cvalue\u003e|c(|@\u003csampling-rate\u003e)?`\n\nExamples:\n\n    rails.requests:1|c\n    page_hits:135|c|@0.1\n\n**Timers**\n\nTimers are for measuring the elapsed time of some operation. These are more\ncomplex than the other kinds of stats. For each timer key, gost records the\nfollowing metrics during each flush period:\n\n* `timer.count`: the number of timer calls that have been recorded\n* `timer.rate`: the rate at which timer calls came in, per second\n* `timer.min`, `timer.max`: the min and max values of the timer during the flush\n  interval\n* `timer.mean`, `timer.median`, `timer.stdev`: the mean, median, and standard\n  deviation, respectively, of the timer values during the flush interval\n* `timer.sum`: the total sum of all timer values during the interval. This\n  value, in concert with `timer.count`, can be used (by some other system) to\n  compute mean values across flush buckets.\n\nSyntax: `\u003ckey\u003e:\u003cvalue\u003e|ms`\n\nExample: `s3_backup:1411|ms`\n\n**Gauges**\n\nA gauge is simply a value that varies over time. The most recent value of the\ngauge is the result gost emits during each flush.\n\nSyntax: `\u003ckey\u003e:\u003cvalue\u003e|g`\n\nExample: `active_users:992|g`\n\n**Sets**\n\nA set records the unique occurrences of some value. The metric sent to graphite\nis the number of unique values that were given under a particular key during a\nflush interval.\n\nSyntax: `\u003ckey\u003e:\u003cvalue\u003e|s`\n\nExample: `user_id:135|s`\n\n### Meta-stats\n\nGost sends back some stats about itself to graphite as well. This includes:\n\n* `gost.bad_messages_seen`: a counter for the number of malformed messages gost\n  has received\n* `gost.packets_received`: a counter for the number of packets gost has read\n* `gost.distinct_metrics_flushed`: a gauge for the number of stats sent to\n  graphite during this flush\n* `gost.distinct_forwarded_metrics_flushed`: a gauge for the number of stats\n  forwarded to another gost during this flush (see Counter Forwarding, below)\n\nThere are some other counters for various error conditions. Most of these also\nshow up in the stderr logs.\n\n### OS stats\n\nOne nice feature of gost is that, if you're running on a Linux system, it can\nautomatically send back statistics about the host, including memory, CPU,\nnetwork, and disk information. See [the example configuration file](conf.toml)\nfor how to set this up, and detailed information on what counters are sent.\n\n### Script stats\n\nGost is able to consume messages via scripts that emit statsd-formatted messages\nto stdout. See [the configuration file](conf.toml) for the options to specify\nthe script directory and the interval between runs.\n\nEach run interval, gost tries to list the script directory. For each regular\nfile in that directory, gost tries to run it as an executable (all at the same\ntime). The output is read line by line and each is parsed as a statsd message.\n\nIf one line is unable to be parsed, gost stops trying to parse the output of\nthat script. If the execution takes so long that the next run interval passes,\nthat script is not started again until it is finished (so at most one copy of\neach script is running at once).\n\nThe scripts are executed with no arguments from gost's current directory. Stdin\nand stderr are null devices. Only stdout is used. Scripts must be executable.\nAny errors running the script (including a non-zero exit status) trigger\ndebugging output and meta-stats.\n\n### Debug interface\n\nThe `debug_port` setting controls the port of a local server that gost starts up\nfor debugging. Gost will print its (UDP) input and (Graphite) output via TCP to\nany client that connects to this port. So if you're using `debug_port = 8126` as\nin the example config, then you can connect like this:\n\n    $ telnet localhost 8126\n\nand you will see gost's input and output. This is very handy for debugging. You\nmay want to filter out just a subset of the data; for instance:\n\n    $ nc localhost 8127 | grep '\\[out\\]' # just outbound messages\n\n## Key Format\n\nGost message keys are formed from printable ascii characters with a few\nrestrictions, listed below. The maximum size of an accepted UDP packet (which\nusually contains one message but may contain several separated by `\\n`) is 10Kb;\nthis sets the only limit on key length.\n\n| source char |  converted to  | reason                                                  |\n| ----------- | -------------- | ------------------------------------------------------- |\n|    newline  |      error     | newlines end gost messages                              |\n|     `:`     |      error     | colons end gost keys                                    |\n|    space    |       `_`      | graphite uses space in its message format               |\n|     `/`     |       `-`      | graphite can't handle `/` (keys are filenames)          |\n|   `\u003c`, `\u003e`  |     removed    | graphite doesn't handle `\u003c` (`\u003e` excluded for symmetry) |\n|     `*`     |     removed    | graphite uses `*` as a wildcard                         |\n|   `[`, `]`  |     removed    | graphite uses `[...]` for char set matching             |\n|   `{`, `}`, |     removed    | graphite uses `{...}` for matching multiple items       |\n\nAdditionally, note that a trailing `.` on a key will be ignored by Graphite, so\n`foo.` is the same as `foo`.\n\n## Counter Forwarding\n\nInstead of sending to graphite, gost can forward metrics (counters only) to\nanother gost, which in turn sends to graphite.\n\nEnable forwarding by setting the `forwarding_addr` option to the network address\nof the gost to which to forward. Then to forward a counter, prefix it with `f|`:\n\n    f|web.requests:1|c\n\nThis counter will not be flushed to graphite, but will be sent to the forwarder\ngost.\n\nTo enable gost to act as a forwarder (that is, it will accept forwarded messages\nin addition to normal UDP messages), set the `forwarder_listen_addr` to the bind\naddress to use to listen for forwarded messages. You can also use the\n`forwarded_namespace` setting to control the namespace applied to forwarded\nstats.\n\n**Motivation:** It's inconvenient to always have to sum your graphite queries\nacross all your servers -- often you only care about the global count. But\ngraphite doesn't add together counters for you when it ingests them. To get\naround this, some folks run a network topology where they forward all their\nmetrics into a single statsd across the network. This has some big downsides:\n\n* It's lossy (UDP)\n* The QPS is really limited in a setup like this\n* It's a lot of network traffic\n\nWith counter forwarding, you can get a lot of the advantages without the\ndisadvantages:\n\n* Gost-to-gost forwarding is over TCP\n* Gosts only flush once every N milliseconds, so the raw stats don't cross the\n  network\n* The forwarding protocol is an efficient binary format\n* You can handle a large volume of metrics this way\n\nOf course, this is still a single point of failure in your metrics collection\nsystem, but if you're using Graphite you've probably got that anyway.\n\nI suggest you host your forwarder gost instance alongside Graphite.\n\nIt is possible for a gost instance to forward counters to itself.\n\n## Tuning\n\nIf you're trying to push a lot of stats into gost, it may start dropping\nmessages. This may be because your OS is using a very limited amount of buffer\nfor the UDP socket. You can typically tune this; on my linux system, for\ninstance, I can bump the limits by using sysctl:\n\n```\n# 25MiB read and write buffer sizes\nnet.core.rmem_max=26214400\nnet.core.rmem_default=26214400\nnet.core.wmem_max=26214400\nnet.core.wmem_default=26214400\n```\n\nWith such of tuning your gost instance should be able to easily handle hundreds\nof thousands of messages per second on moderate hardware.\n\n(This will, of course, incur a lot of system load and typically you'll want to\nuse sampling to limit the gost qps to something reasonable.)\n\n## Differences with StatsD\n\n* Statsd only allows keys matching `/^[a-zA-Z0-9\\-_\\.]+$/`; gost is more\n  permissive (see Key Format, above).\n* Gauges cannot be deltas; they must be absolute values.\n* Timers don't return as much information as in statsd, and they're not\n  customizable.\n* gost can record OS stats from the host and deliver them to graphite as well.\n* The \"meta-stats\" gost sends back are different from StatsD (there are a lot\n  fewer of them)\n* Gost is very fast. It can handle several times the load statsd can before\n  dropping messages. In my unscientific tests on my Linux dev machine, I got\n  statsd up to about 80k qps before it started dropping messages, while gost got\n  to 350k+ qps without dropping any messages.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frstudio-tech%2Fgost","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frstudio-tech%2Fgost","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frstudio-tech%2Fgost/lists"}