{"id":26256559,"url":"https://github.com/netcracker/qubership-graphite-remote-adapter","last_synced_at":"2025-09-02T23:32:15.483Z","repository":{"id":275760515,"uuid":"905766637","full_name":"Netcracker/qubership-graphite-remote-adapter","owner":"Netcracker","description":"Adapter to receive metrics in Prometheus RemoteWrite protocol, convert and send to Graphite protocol","archived":false,"fork":false,"pushed_at":"2025-02-21T08:37:45.000Z","size":2019,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-10T11:23:14.928Z","etag":null,"topics":["monitoring","observability"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Netcracker.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE-OF-CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-19T13:33:34.000Z","updated_at":"2025-02-04T13:29:54.000Z","dependencies_parsed_at":"2025-02-04T13:38:30.085Z","dependency_job_id":null,"html_url":"https://github.com/Netcracker/qubership-graphite-remote-adapter","commit_stats":null,"previous_names":["netcracker/qubership-graphite-remote-adapter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Netcracker%2Fqubership-graphite-remote-adapter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Netcracker%2Fqubership-graphite-remote-adapter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Netcracker%2Fqubership-graphite-remote-adapter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Netcracker%2Fqubership-graphite-remote-adapter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Netcracker","download_url":"https://codeload.github.com/Netcracker/qubership-graphite-remote-adapter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243475362,"owners_count":20296714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["monitoring","observability"],"created_at":"2025-03-13T20:18:28.614Z","updated_at":"2025-09-02T23:32:15.441Z","avatar_url":"https://github.com/Netcracker.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Qubership Graphite Remote Storage Adapter\n\n[![Super-Linter](https://github.com/Netcracker/qubership-graphite-remote-adapter/actions/workflows/super-linter.yaml/badge.svg)](https://github.com/marketplace/actions/super-linter)\n\nThis is a read/write adapter that receives samples via Prometheus's remote write\nprotocol and stores them in remote storage like Graphite.\n\nInitially it was a fork of [graphite_remote_adapter](https://github.com/criteo/graphite-remote-adapter).\n\nBut after that, we made a lot of changes to its source code.\n\n## Differences from initial graphite-remote-adapter\n\n* Added LZ4 compression for Graphite protocol\n* Added performance fixes to improve performance\n* Added escaping symbols `;`, `~`, ' ' (space), `=` of label values to correct work with Graphite + ClickHouse stack\n* Added Helm chart to deploy this adapter in Kubernetes\n* Added Grafana dashboard to visualize self-metrics\n\n## Throughput\n\nTo reach 35k rps it is necessary at least:\n\n* 2 core\n* 2.5 GB\n* Handle write request duration on graphite carbon side less than 1 sec\n\n## Metric routing\n\nTo route metrics to graphite-remote-adapter change platform monitoring CRD(Custom Resource Definition) fields:\n\n* externalLabels\n* remoteWrite\n\nThe externalLabels to add to any time series or alerts when\ncommunicating with external systems (federation, remote\nstorage, Alertmanager).\n\nExample:\n\n```yaml\nexternalLabels:\n  cluster: test-cluster\n  environment: test-environment\n  project: development\n  team: test_team`\n```\n\nRemoteWriteSpec defines the remote_write configuration\nfor prometheus. The `remote_write` allows transparently\nsend samples to a long term storage.\n\nExample:\n\n```yaml\nremoteWrite:\n  - tlsConfig:\n      insecureSkipVerify: true\n    url: \u003e-\n      https://url_for_remote_write/write\n```\n\n### LZ4 compression\n\nLZ4 streaming compression can be turned on in configuration.\n\nExample:\n\n```yaml\nadditionalGraphiteConfig:\n  graphite:\n    write:\n      compress_type: lz4\n      lz4_preferences:\n        frame:\n          block_size: max256KB\n          block_mode: false\n          content_checksum: false\n          block_checksum: false\n        compression_level: 12\n        auto_flush: false\n        decompression_speed: false\n```\n\nParameters:\n\n* `compress_type` field support `plain`, `lz4` and empty (means `plain`) values.\n* `lz4_preferences` contains parameters for lz4 streaming compression.\n* `frame` - lz4 frame info.\n* `frame.block_size` - the larger the block size, the (slightly) better the compression ratio.\n  Larger blocks also increase memory usage on both compression and decompression sides.\n  Supported values: max64KB, max256KB, max1MB, max4MB. Default: max64KB.\n* `frame.block_mode` - linked blocks sharply reduce inefficiencies when using small blocks, they compress better.\n  However, some LZ4 decoders are only compatible with independent blocks. Default - false, i.e. blocks are linked.\n* `frame.content_checksum` - add a 32-bit checksum of frame's decompressed data. Default - false, i.e. disabled.\n* `frame.block_checksum` -  each block followed by a checksum of block's compressed data. Default - false, i.e. disabled.\n* `compression_level` - min value 3, max 12, default 9\n* `auto_flush` - always flush; reduces usage of internal buffers. Default - `false`\n* `decompression_speed` - parser favors decompression speed vs compression ratio.\n  Works for high compression modes (compression_level \u003e= 10) only.\n\n## Metrics list\n\n```prometheus\n# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.\n# TYPE go_gc_duration_seconds summary\ngo_gc_duration_seconds{quantile=\"0\"} 1.5378e-05\ngo_gc_duration_seconds{quantile=\"0.25\"} 2.4186e-05\ngo_gc_duration_seconds{quantile=\"0.5\"} 3.5375e-05\ngo_gc_duration_seconds{quantile=\"0.75\"} 5.7691e-05\ngo_gc_duration_seconds{quantile=\"1\"} 0.004353922\ngo_gc_duration_seconds_sum 0.68606493\ngo_gc_duration_seconds_count 474\n# HELP go_goroutines Number of goroutines that currently exist.\n# TYPE go_goroutines gauge\ngo_goroutines 11\n# HELP go_info Information about the Go environment.\n# TYPE go_info gauge\ngo_info{version=\"go1.13.8\"} 1\n# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.\n# TYPE go_memstats_alloc_bytes gauge\ngo_memstats_alloc_bytes 4.3156784e+07\n# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.\n# TYPE go_memstats_alloc_bytes_total counter\ngo_memstats_alloc_bytes_total 3.6797211216e+10\n# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.\n# TYPE go_memstats_buck_hash_sys_bytes gauge\ngo_memstats_buck_hash_sys_bytes 1.559402e+06\n# HELP go_memstats_frees_total Total number of frees.\n# TYPE go_memstats_frees_total counter\ngo_memstats_frees_total 4.63408148e+08\n# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.\n# TYPE go_memstats_gc_cpu_fraction gauge\ngo_memstats_gc_cpu_fraction 0.025775199892331825\n# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.\n# TYPE go_memstats_gc_sys_bytes gauge\ngo_memstats_gc_sys_bytes 2.413568e+07\n# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.\n# TYPE go_memstats_heap_alloc_bytes gauge\ngo_memstats_heap_alloc_bytes 4.3156784e+07\n# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.\n# TYPE go_memstats_heap_idle_bytes gauge\ngo_memstats_heap_idle_bytes 4.8390144e+08\n# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.\n# TYPE go_memstats_heap_inuse_bytes gauge\ngo_memstats_heap_inuse_bytes 1.18898688e+08\n# HELP go_memstats_heap_objects Number of allocated objects.\n# TYPE go_memstats_heap_objects gauge\ngo_memstats_heap_objects 312946\n# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.\n# TYPE go_memstats_heap_released_bytes gauge\ngo_memstats_heap_released_bytes 4.49609728e+08\n# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.\n# TYPE go_memstats_heap_sys_bytes gauge\ngo_memstats_heap_sys_bytes 6.02800128e+08\n# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.\n# TYPE go_memstats_last_gc_time_seconds gauge\ngo_memstats_last_gc_time_seconds 1.5917296489909294e+09\n# HELP go_memstats_lookups_total Total number of pointer lookups.\n# TYPE go_memstats_lookups_total counter\ngo_memstats_lookups_total 0\n# HELP go_memstats_mallocs_total Total number of mallocs.\n# TYPE go_memstats_mallocs_total counter\ngo_memstats_mallocs_total 4.63721094e+08\n# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.\n# TYPE go_memstats_mcache_inuse_bytes gauge\ngo_memstats_mcache_inuse_bytes 6944\n# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.\n# TYPE go_memstats_mcache_sys_bytes gauge\ngo_memstats_mcache_sys_bytes 16384\n# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.\n# TYPE go_memstats_mspan_inuse_bytes gauge\ngo_memstats_mspan_inuse_bytes 2.959088e+06\n# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.\n# TYPE go_memstats_mspan_sys_bytes gauge\ngo_memstats_mspan_sys_bytes 7.389184e+06\n# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.\n# TYPE go_memstats_next_gc_bytes gauge\ngo_memstats_next_gc_bytes 7.8598512e+07\n# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.\n# TYPE go_memstats_other_sys_bytes gauge\ngo_memstats_other_sys_bytes 2.08219e+06\n# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.\n# TYPE go_memstats_stack_inuse_bytes gauge\ngo_memstats_stack_inuse_bytes 1.179648e+06\n# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.\n# TYPE go_memstats_stack_sys_bytes gauge\ngo_memstats_stack_sys_bytes 1.179648e+06\n# HELP go_memstats_sys_bytes Number of bytes obtained from system.\n# TYPE go_memstats_sys_bytes gauge\ngo_memstats_sys_bytes 6.39162616e+08\n# HELP go_threads Number of OS threads created.\n# TYPE go_threads gauge\ngo_threads 14\n# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.\n# TYPE process_cpu_seconds_total counter\nprocess_cpu_seconds_total 340.33\n# HELP process_max_fds Maximum number of open file descriptors.\n# TYPE process_max_fds gauge\nprocess_max_fds 1.048576e+06\n# HELP process_open_fds Number of open file descriptors.\n# TYPE process_open_fds gauge\nprocess_open_fds 9\n# HELP process_resident_memory_bytes Resident memory size in bytes.\n# TYPE process_resident_memory_bytes gauge\nprocess_resident_memory_bytes 1.92991232e+08\n# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.\n# TYPE process_start_time_seconds gauge\nprocess_start_time_seconds 1.59172872096e+09\n# HELP process_virtual_memory_bytes Virtual memory size in bytes.\n# TYPE process_virtual_memory_bytes gauge\nprocess_virtual_memory_bytes 6.90434048e+08\n# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.\n# TYPE process_virtual_memory_max_bytes gauge\nprocess_virtual_memory_max_bytes -1\n# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.\n# TYPE promhttp_metric_handler_requests_in_flight gauge\npromhttp_metric_handler_requests_in_flight 1\n# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.\n# TYPE promhttp_metric_handler_requests_total counter\npromhttp_metric_handler_requests_total{code=\"200\"} 1\npromhttp_metric_handler_requests_total{code=\"500\"} 0\npromhttp_metric_handler_requests_total{code=\"503\"} 0\n# HELP remote_adapter_api_request_duration_seconds A histogram of latencies for requests.\n# TYPE remote_adapter_api_request_duration_seconds histogram\nremote_adapter_api_request_duration_seconds_bucket{handler=\"write\",method=\"post\",le=\"0.25\"} 27489\nremote_adapter_api_request_duration_seconds_bucket{handler=\"write\",method=\"post\",le=\"0.5\"} 28057\nremote_adapter_api_request_duration_seconds_bucket{handler=\"write\",method=\"post\",le=\"1\"} 29442\nremote_adapter_api_request_duration_seconds_bucket{handler=\"write\",method=\"post\",le=\"2.5\"} 34115\nremote_adapter_api_request_duration_seconds_bucket{handler=\"write\",method=\"post\",le=\"5\"} 45541\nremote_adapter_api_request_duration_seconds_bucket{handler=\"write\",method=\"post\",le=\"10\"} 63879\nremote_adapter_api_request_duration_seconds_bucket{handler=\"write\",method=\"post\",le=\"+Inf\"} 63879\nremote_adapter_api_request_duration_seconds_sum{handler=\"write\",method=\"post\"} 164068.22068123778\nremote_adapter_api_request_duration_seconds_count{handler=\"write\",method=\"post\"} 63879\n# HELP remote_adapter_api_requests_total A counter for requests to the wrapped handler.\n# TYPE remote_adapter_api_requests_total counter\nremote_adapter_api_requests_total{code=\"200\",handler=\"write\",method=\"post\"} 63879\n# HELP remote_adapter_api_response_size_bytes A histogram of response sizes for requests.\n# TYPE remote_adapter_api_response_size_bytes histogram\nremote_adapter_api_response_size_bytes_bucket{handler=\"write\",le=\"200\"} 63879\nremote_adapter_api_response_size_bytes_bucket{handler=\"write\",le=\"500\"} 63879\nremote_adapter_api_response_size_bytes_bucket{handler=\"write\",le=\"900\"} 63879\nremote_adapter_api_response_size_bytes_bucket{handler=\"write\",le=\"1500\"} 63879\nremote_adapter_api_response_size_bytes_bucket{handler=\"write\",le=\"+Inf\"} 63879\nremote_adapter_api_response_size_bytes_sum{handler=\"write\"} 1.27758e+06\nremote_adapter_api_response_size_bytes_count{handler=\"write\"} 63879\n# HELP remote_adapter_received_samples_total Total number of received samples.\n# TYPE remote_adapter_received_samples_total counter\nremote_adapter_received_samples_total{prefix=\"test.miniha_kubernetes.\"} 6.268375e+06\n# HELP remote_adapter_sent_batch_duration_seconds Duration of sample batch send calls to the remote storage.\n# TYPE remote_adapter_sent_batch_duration_seconds histogram\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"0.005\"} 24169\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"0.01\"} 26018\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"0.025\"} 26530\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"0.05\"} 26788\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"0.1\"} 27207\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"0.25\"} 27708\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"0.5\"} 28295\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"1\"} 29634\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"2.5\"} 34454\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"5\"} 47266\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"10\"} 63879\nremote_adapter_sent_batch_duration_seconds_bucket{remote=\"10.0.0.0:2003\",le=\"+Inf\"} 63879\nremote_adapter_sent_batch_duration_seconds_sum{remote=\"10.0.0.0:2003\"} 157999.59580358808\nremote_adapter_sent_batch_duration_seconds_count{remote=\"10.0.0.0:2003\"} 63879\n# HELP remote_adapter_sent_samples_total Total number of processed samples sent to remote storage.\n# TYPE remote_adapter_sent_samples_total counter\nremote_adapter_sent_samples_total{prefix=\"test.miniha_kubernetes.\",remote=\"10.0.0.0:2003\"} 6.268375e+06\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetcracker%2Fqubership-graphite-remote-adapter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetcracker%2Fqubership-graphite-remote-adapter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetcracker%2Fqubership-graphite-remote-adapter/lists"}