{"id":20571878,"url":"https://github.com/rapidloop/sop","last_synced_at":"2025-04-14T17:08:14.423Z","repository":{"id":95145060,"uuid":"105507804","full_name":"rapidloop/sop","owner":"rapidloop","description":"Multi-purpose metrics management tool","archived":false,"fork":false,"pushed_at":"2017-10-03T10:54:43.000Z","size":77,"stargazers_count":88,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-14T17:07:58.692Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rapidloop.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-02T07:21:33.000Z","updated_at":"2023-11-01T20:43:51.000Z","dependencies_parsed_at":"2023-04-09T19:47:44.189Z","dependency_job_id":null,"html_url":"https://github.com/rapidloop/sop","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapidloop%2Fsop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapidloop%2Fsop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapidloop%2Fsop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapidloop%2Fsop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rapidloop","download_url":"https://codeload.github.com/rapidloop/sop/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248923764,"owners_count":21183953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T05:17:37.531Z","updated_at":"2025-04-14T17:08:14.386Z","avatar_url":"https://github.com/rapidloop.png","language":"Go","readme":"\n**sop** is a multi-purpose metrics storage and manipulation tool based on the\nPrometheus metrics data model.\n\n## Features\n\n### Inputs\n\nsop can accept inputs from various sources, simultaneously. The incoming metrics\ncan be **downsampled**. You can also use **filters** to store only a subset of\nthe incoming data. Time series data is stored in RocksDB, and is **compressed**\nby default. Metrics can be set to expire, to support **staleness handling**.\n\nThese input sources are implemented currently:\n\n* *Prometheus v1.x remote write*: An HTTP endpoint that can be set as the\n  remote write URL for Prometheus v1.x. Tested with 1.7.2.\n* *Prometheus v2.x remote write*: An HTTP endpoint that can be set as the\n  remote write URL for Prometheus v2.x. Tested with 2.0.0-beta.5; subject to change\n  as 2.0 evolves.\n* *InfluxDB*: An HTTP \"/write\" endpoint that can accept InfluxDB line protocol. \n  Tags are mapped to Prometheus-style labels and field values to multiple\n  metrics.\n* *NATS streaming server*: A durable NATS streaming client, which can ingest\n  and store metrics published to a NATS streaming server by another sop instance.\n  (See outputs section below.)\n\n### APIs\n\nsop supports APIs that are used to get at the data stored inside it's database.\nOne API is implemented currently:\n\n* *Prometheus HTTP API*: provides the standard Prometheus APIs, including\n  PromQL and federation support. sop can be configured as a Prometheus\n  data source in Grafana. Federation out of sop using the \"/federate\" endpoint\n  is also possible.\n\n### Outputs\n\nApart from storing internally, sop can stream the incoming data (after\ndownsampling, if downsampling is configured) to other servers. **Filters** can\nbe used to send only a subset of the data. Multiple outputs can be operated\nsimultaneously.\n\nThese output destinations are implemented currently:\n\n* *Prometheus v2.x remote write*: Write to a Prometheus v2.x remote write\n  HTTP endpoint.\n* *InfluxDB*: Write to an InfluxDB server using the HTTP \"/write\" endpoint.\n* *OpenTSDB*: Write to an OpenTSDB server using the HTTP \"/api/put\" endpoint.\n* *NATS streaming server*: Publish metrics to a NATS streaming server. The NATS\n  streaming server is used as a durable queue. The published messages can then\n  be read in by another sop instance. It can also be read in by multiple sop\n  instances, thereby redundantly duplicating the metrics stream. Using such a\n  queue also allows the sop output to on occasion burst above rates that\n  can be sustainedly ingested by the sop input.\n\n## Uses\n\n* *Long-term archiving*: downsample and store metrics for a configurable\n  retention period. Stored data can be queried anytime with Grafana and APIs.\n* *Unified storage*: metrics from multiple metrics system into sop. Or use sop\n  to send them all to another system, like OpenTSDB.\n* *Centralized storage*: collect and store metrics from multiple datacenters\n  or locations into a centralized store.\n\n## Status\n\nsop is currently in beta (1.0-beta.1).\n\nPerformance-wise, it can do sustained write rates of 13k+ samples/sec on a\n4-core 8GB DigitalOcean node, as measured using the `sopfilltest` tool.\n\n## Getting Started\n\nsop is available as a zero-dependency, single-binary executable for 64-bit\nLinux platforms. Pre-built binaries are available as [GitHub releases](https://github.com/rapidloop/sop/releases);\nstart off by downloading the latest release.\n\nsop is invoked with the path to a configuration file. You can generate the\ndefault configuration file from sop itself using the \"-p\" flag:\n\n```\n$ sop -p \u003e sop.cfg\n```\n\nYou can then edit the configuration file \"sop.cfg\" to setup inputs, APIs and\noutputs. The comments within the file should be self-explanatory. If not, ask!\n\nYou can run sop by invoking it with the path to the configuration file:\n\n```\n$ sop sop.cfg\n2017/10/03 09:11:26.190018 main.go:77: sop starting: version=1.0-beta.1, pid=11333\n2017/10/03 09:11:26.653607 main.go:115: started database: path=data (took 463.453866ms)\n2017/10/03 09:11:26.653681 main.go:128: started storer: ttl=4m0s, downsample=0s\n2017/10/03 09:11:26.653729 main.go:141: started reaper: retain=4320h0m0s, gc=24h0m0s\n2017/10/03 09:11:26.654129 main.go:162: started input: prometheus v2 remote write (listen=0.0.0.0:9096)\n2017/10/03 09:11:26.654366 main.go:188: started api: prometheus_http (listen=0.0.0.0:9095)\n2017/10/03 09:11:26.654386 main.go:202: sop open for business\n```\n\nUse `^C` to exit.\n\nSee [here](https://prometheus.io/docs/operating/configuration/#%3Cremote_write%3E)\nfor documentation about Prometheus' remote storage configuration.\n\nIf you want to build sop yourself, see the\n[sop-build](https://github.com/rapidloop/sop-build) repo.\n\n## Contributing\n\nsop is released under Apache 2.0 license, copyright (c) 2017 RapidLoop, Inc.\n\nsop is written in Go. Data is stored in RocksDB. It reuses parts of Prometheus,\nespecially PromQL. The build scripts are maintained separately in the\nsop-build repository. To hack on sop yourself, start by cloning sop-build and\nbuilding sop. It's fairly straight forward to add a new input, api or output.\n\nYour contributions are welcome. Start off by raising a issue here in GitHub.\n\nsop is developed and maintained by [RapidLoop](https://www.rapidloop.com/),\nthe makers of [OpsDash](https://www.opsdash.com/). Talk to us for support or\ncustomized monitoring solutions at hello@rapidloop.com.\n\n## So, hey, is _sop_ an acronym?\n\nAh. Um. Yeah, well, actually yes: _Son of Prometheus_. (ducks)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frapidloop%2Fsop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frapidloop%2Fsop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frapidloop%2Fsop/lists"}