{"id":18832677,"url":"https://github.com/datto/sisyphus","last_synced_at":"2025-04-14T04:24:26.247Z","repository":{"id":45332175,"uuid":"427144460","full_name":"datto/sisyphus","owner":"datto","description":"Kafka -\u003e Influx 2.x forwarder in Go","archived":false,"fork":false,"pushed_at":"2023-09-09T17:26:04.000Z","size":36764,"stargazers_count":4,"open_issues_count":3,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-27T18:21:25.500Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-11T21:11:37.000Z","updated_at":"2023-02-03T11:33:51.000Z","dependencies_parsed_at":"2023-01-18T16:45:15.623Z","dependency_job_id":null,"html_url":"https://github.com/datto/sisyphus","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datto%2Fsisyphus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datto%2Fsisyphus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datto%2Fsisyphus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datto%2Fsisyphus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datto","download_url":"https://codeload.github.com/datto/sisyphus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248820190,"owners_count":21166621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T01:58:43.097Z","updated_at":"2025-04-14T04:24:26.226Z","avatar_url":"https://github.com/datto.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sisyphus\n\nKafka -\u003e Influx 2.x forwarder in Golang\n\n# Expectations\n\n1. Incoming data is formatted as:\n  * \"Influx JSON\" format (telegraf's JSON output: https://github.com/influxdata/telegraf/tree/master/plugins/serializers/json)\n  * Influx line protocol (https://github.com/influxdata/telegraf/tree/master/plugins/serializers/influx)\n  * \"Prometheus JSON\" format (created from https://github.com/Telefonica/prometheus-kafka-adapter#json)\n2. Outbound data writes to an Influx v2 compatible endpoint.\n\n# Guarantees\n\n* Output data is formatted in an prometheus-compatible format (https://prometheus.io/docs/concepts/data_model/)\n* Failed writes are written to a defined Kafka topic for later inspection/processing\n\n# Config\n```\nconsumer_group: sisyphus-test\nclient_id: \u003chostname\u003e\n\nbrokers:\n  - localhost:9092\nstarting_offset_type: earliest\nnormalize_metrics: false\n\nwritepaths:\n  - influx_json_topics:\n      - test\n      - test2\n  - influx_line_topics:\n      - test5\n      - test6\n    prometheus_topics:\n      - test3\n      - test4\n    output_hostname: localhost\n    output_path: \"/insert/0:0/influx\"\n    output_port: 8480\n    go_channel_size: 10000\n    # number of points to send on each write\n    send_batch: 1000\n    # in seconds\n    tsd_flush_time: 10\n    # milliseconds before a write is considered timed out\n    write_timeout: 30\n    # individual thread sizes\n    kafka_reader_threads: 1\n    processor_threads: 1\n    filter_threads: 1\n    write_threads: 1\n    flip_single_fields: true\n    max_retries: 0\n\nfailed_writes_topic: influx-failed-writes\n\nstats_listen_address: 127.0.0.1\nstats_listen_port: 9999\n```\n\n## `normalize_metrics`\n\nForces all incoming stat names/tags/field names to lowercase strings. This is useful if you are trying to migrate a from a system that already did this.\n\n## `flip_single_fields`\n\nThis is a strange config, but needed when using VictoriaMetrics as a destination with `-influxSkipSingleField`. You can end up needed this in the following situation:\n\n1. You are ingesting data from influx sources\n2. you would like to add prometheus sources\n\nTo successfully change the data model, we must enable `-influxSkipSingleField` so prometheus stats like:\n```\n{\"value\": 2,\n \"timestamp\": 0,\n \"labels\": {\n   \"__name__\": \"up\",\n   \"dc\": \"dc1\"\n }\n}\n```\nWill not produce a value like `up_value` in VictoriaMetrics.\n\nBut this means that an influx stat like:\n```\n{\"fields\": {\n   \"lag\": 2\n },\n \"tags\": {\n   \"dc\": \"dc1\"\n },\n \"name\": \"kafka\",\n \"timestamp\": 0\n}\n```\nWill become `kafka` (dropping the `_lag` value that is descriptive and useful).\n\nIn this case, we can enable `flip_single_fields`, and `kafka_lag` will be submitted as `kafka_lag_value`, which VictoriaMetrics will then trim to `kafka_lag`.\n\n# Stats\n\nSisyphus uses https://github.com/VictoriaMetrics/metrics to produce Prometheus compatible stats\n\nStats available `http://\u003cstats_listen_address\u003e:\u003cstats_listen_port\u003e/\u003cstats_listen_path\u003e`.\n\nA quick breakdown:\n\n* ReceivedMsgs\n  * Messages that have been received and marshalled into objects for processing\n* SentMsgs\n  * Messages sent to our output\n* FilteredMsgs\n  * Messages that have been altered in some way to conform to standards (removed tag, changed key name, etc.)\n* DroppedMsgs\n  * Messages that can't be fixed to meet standards and must be dropped\n* FailedMsgs\n  * Messages that failed to send to the output\n  * failed messages are added to a dead-letter queue in Kafka\n* IngestMsgs\n  * Messages initially received from Kafka\n* MetricsCounted\n  * Individual metrics counted during the filtering process\n\n# Licensing\n\nsisyphus is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, under version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see \u003chttp://www.gnu.org/licenses/\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatto%2Fsisyphus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatto%2Fsisyphus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatto%2Fsisyphus/lists"}