{"id":22204382,"url":"https://github.com/innogames/pirate","last_synced_at":"2025-09-01T16:32:56.464Z","repository":{"id":57493147,"uuid":"85308912","full_name":"innogames/pirate","owner":"innogames","description":"Realtime metrics server written in Go","archived":false,"fork":false,"pushed_at":"2023-04-09T22:11:48.000Z","size":124,"stargazers_count":12,"open_issues_count":0,"forks_count":4,"subscribers_count":27,"default_branch":"master","last_synced_at":"2024-04-14T10:03:31.706Z","etag":null,"topics":["go","grafsy","graphite","metrics"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/innogames.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-17T12:22:31.000Z","updated_at":"2024-06-20T03:18:03.351Z","dependencies_parsed_at":"2024-06-20T03:29:24.492Z","dependency_job_id":null,"html_url":"https://github.com/innogames/pirate","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/innogames/pirate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/innogames%2Fpirate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/innogames%2Fpirate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/innogames%2Fpirate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/innogames%2Fpirate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/innogames","download_url":"https://codeload.github.com/innogames/pirate/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/innogames%2Fpirate/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267306730,"owners_count":24067035,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-27T02:00:11.917Z","response_time":82,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","grafsy","graphite","metrics"],"created_at":"2024-12-02T17:17:09.128Z","updated_at":"2025-07-27T05:32:57.813Z","avatar_url":"https://github.com/innogames.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pirate\n\n![Pirate Logo](doc/pirate-logo.png)\n\n## What is Pirate?\n\nPirate is a gateway, written in go, which accepts client-side metrics via UDP and makes them available to [grafsy](https://github.com/leoleovich/grafsy).\nIn the end you can have near-time dashboards with client-side metrics.\n\n\n## Architecture\n\n![Pirate Server Architecture](doc/architecture.png)\n\n- UDP packets are accepted by the UDP server\n- if compression is enabled, the packets are decompressed for the next step\n- plain text messages are parsed into header attributes and the metrics body (see [protocol](#protocol))\n- parsed messages are validated, invalid messages or metrics will be dropped (see [validation](#validation))\n- valid metric names are then resolved to their Graphite path\n- at the end the writer will take care of sending the metrics to Grafsy via TCP or filesystem\n- beside this there is a monitoring component, which tracks own metrics like `udp_received`, `metrics_received` or `metrics_dropped`,\n  which are also sent to the writer\n\nAll the components of the data pipeline are implemented as workers (goroutines), which scale with the amount of CPUs. Additionally\nthere are buffers (go channels) between those components. This means, even if some messages are processed a bit slower,\nthe rest of the system should not be affected.\n\nIn the rare case that all buffers should be full, incoming UDP packets will be dropped immediately.\n\n\n## Protocol\n\nThe client's message is a (GZIP-encoded) UDP packet, which consist of two parts: the header and the body.\n\nThe header is the first line of the message and contains information about the whole message (e.g. the project identifier and custom attributes).\nThe body contains the metrics, where each metric consists of a name, a numeric value and a timestamp.\n\n### Example\n\nIn plain text (before GZIP-compression is applied) the message could look like this:\n```\nproject=awesome_game; version=1.3.37;\nfps 55 1234567890\nfps 48 1234567900\nmemory_usage 209715200 1234567890\nfps 53 1234567950\nmemory_usage 205215400 1234568320\n```\n\nHere are two attributes defined by the header. The `project` is always required, because it determines, which rules\nare applied to the metrics. The `version` attribute is a custom attribute, which can be used in the `graphite_path`\nconfiguration via `{attr.version}`. Additional attributes, which are not used in the `graphite_path` will be ignored.\n\nIn total there are 5 metrics, which will be processed according to the configuration of the `awesome_game` project.\n\n\n## Output Format\n\nAccording to the project rules the metric name will be resolved to a graphite path (see [placeholders](#placeholders)\nfor more information). The resulting rows could then look like this:\n\n```\ngames.awesome_game.client.ios.1_3_37.fps 55 1234567890\ngames.awesome_game.client.ios.1_3_37.fps 48 1234567900\ngames.awesome_game.client.ios.1_3_37.memory_usage 209715200 1234567890\ngames.awesome_game.client.ios.1_3_37.fps 53 1234567950\ngames.awesome_game.client.ios.1_3_37.memory_usage 205215400 1234568320\n```\n\n\n## Validation\n\nThe incoming packages and the single metrics are validated. This includes:\n\n- the project identifier from the header must be configured, otherwise the message is dropped\n- all values of custom header fields must match their configured regex, otherwise the message is dropped\n- sent metric names must be configured, otherwise the metric is dropped\n- metric values must be within the configured min/max range to be valid, otherwise the metric is dropped\n- timestamp validation: metrics with future timestamps or too old timestamps (\u003e 1h) get dropped\n\nAll metrics which passed this validation will be processed and sent to Grafsy\n\n\n## Configuration\n\n### General\n\n| Key                  | Description                                              |\n|----------------------|----------------------------------------------------------|\n| `udp_address`        | The address to listen for UDP packages                   |\n| `graphite_target`    | The target, where the graphite data should be sent to, e.g. `tcp://localhost:3002` or `file:///tmp/metrics.log` |\n| `monitoring_enabled` | Whether Pirate should generate own monitoring metrics (received metrics, dropped metrics, etc.) |\n| `monitoring_path`    | Graphite path to use for monitoring metrics, only used when monitoring enabled (may contain [placeholders](#placeholders)) |\n| `gzip`               | Whether to use GZIP compressed messages |\n| `log_level`          | The log level (debug, info, notice, warning, error, critical) |\n| `per_ip_ratelimit`   | Rate limit per IP, allows up to `amount` UDP messages per `interval` from the same IP address |\n\n### Projects\n\nEvery project has its own custom sub-section within the configuration file under the key `projects.PROJECT_ID`,\nwhere `PROJECT_ID` is your own identifier, which is used from the message header to determine the target project\n\nThe projects sub-section then has the following keys:\n\n| Key               | Description                                              |\n|-------------------|----------------------------------------------------------|\n| `graphite_path`   | The path each incoming metric is written to. It might contain placeholders (see [placeholders](#placeholders) for more information) |\n| `attributes`      | Custom attributes, which can be used within [placeholders](#placeholders) |\n| `metrics`         | Allowed metric definitions with boundary check           |\n\n### Attributes\n\nEvery project may have custom attribute definitions under the key `projects.PROJECT_ID.attributes.ATTRIBUTE_ID`,\nwhere `ATTRIBUTE_ID` is your own identifier, which is used from the message header (analogous to the `project` attribute).\n\nThe value of every attribute is just a regular expression, which is tested on incoming message headers. If at least one\nof the attribute's expressions does not match, the whole message will be dropped.\n\n### Metrics\n\nEvery project may have one or more metric definitions under the key `projects.PROJECT_ID.metrics.METRIC_ID`,\nwhere `METRIC_ID` is your own metric identifier, which equals the metric name from the message body.\n\nThis definition includes a min and max value, which are used for boundary validation.\nMetrics, which are out of the configured boundary, will be dropped.\nOptionally the Graphite Path of the project can be overridden for single metrics.\n\nThe metrics sub-section supports the following options:\n\n| Key             | Description                                              |\n|-----------------|----------------------------------------------------------|\n| `graphite_path` | The Graphite path, which is used for this metric. This is optional: if left out, the `graphite_path` from the project is used |\n| `min`           | The minimum allowed value (float32) |\n| `max`           | The maximum allowed value (float32) |\n\n### Placeholders\n\nWithin your `graphite_path` configuration you can use two types of placeholders: attributes (`attr`) and metrics (`metric`).\nThe first one relates to attributes, which are sent with the message header and contain the project ID and arbitrary data.\nThe `metric` variable relates to the metric itself and currently only allows access to the metric sub-key `name`\n\nExample:\n```yaml\nprojects:\n  example_project:\n    graphite_path: games.awesome_game.client.{attr.platform}.{attr.version}.{metric.name}\n```\n\nSending the following message\n```\nproject=example_project; platform=ios; version=1.3.37;\nfps 55 1234567890\n```\n\nwould result in the path `games.awesome_game.client.ios.1_3_37.fps`\n\n*Notice:* during the placeholder resolution all dots are substituted by underscores in order to not influence your graphite path hierarchy\n\nIf one of the attributes is missing, the metrics won't be processed any further\n\n### Full Config Example\n```yaml\nudp_address: 0.0.0.0:33333\ngraphite_target: tcp://127.0.0.1:3002\nmonitoring_enabled: true\nmonitoring_path: games.awesome_game.pirate.{metric.name}\ngzip: true\nlog_level: debug # debug mode is very verbose and should only be used for - well - debugging purpose :)\nper_ip_ratelimit:\n  amount: 20\n  interval: 1m\nprojects:\n  awesome_client:\n    graphite_path: AVG.games.awesome_game.client.{attr.platform}.{attr.version}.{metric.name}\n    attributes:\n      platform: ^(ios|android)$\n      version: ^[0-9]+\\.[0-9]+$\n    metrics:\n      frames_per_second:\n        min: 0\n        max: 60\n      memory_usage:\n        min: 0\n        max: 2147483648\n      startup_time:\n        min: 0\n        max: 90\n      errors:\n        graphite_path: SUM.games.awesome_game.client.{attr.platform}.{attr.version}.{metric.name}\n        min: 0\n        max: 1000\n  awesome_backend:\n    graphite_path: servers.some_project.{attr.hostname}.{metric.name}\n    attributes:\n      hostname: ^[0-9a-f]{12}$\n    metrics:\n      requests_per_second:\n        min: 0\n        max: 100000\n      response_time:\n        min: 0\n        max: 100\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finnogames%2Fpirate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finnogames%2Fpirate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finnogames%2Fpirate/lists"}