{"id":26059694,"url":"https://github.com/orange-cloudfoundry/custom_exporter","last_synced_at":"2025-04-11T06:50:28.123Z","repository":{"id":57523679,"uuid":"83023478","full_name":"orange-cloudfoundry/custom_exporter","owner":"orange-cloudfoundry","description":"A prometheus exporter that can retreive custom informations","archived":false,"fork":false,"pushed_at":"2020-01-02T11:50:43.000Z","size":147,"stargazers_count":17,"open_issues_count":0,"forks_count":2,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-25T04:41:31.906Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orange-cloudfoundry.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-24T09:13:42.000Z","updated_at":"2023-09-13T22:02:40.000Z","dependencies_parsed_at":"2022-08-28T06:00:22.822Z","dependency_job_id":null,"html_url":"https://github.com/orange-cloudfoundry/custom_exporter","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcustom_exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcustom_exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcustom_exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcustom_exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orange-cloudfoundry","download_url":"https://codeload.github.com/orange-cloudfoundry/custom_exporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248358548,"owners_count":21090401,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-08T13:27:50.197Z","updated_at":"2025-04-11T06:50:28.085Z","avatar_url":"https://github.com/orange-cloudfoundry.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prometheus custom_exporter\n\n## Intro\n\nThis project is aimed to retrieve specific metrics that can't be found in dedicated exporters.\n\n## Concepts\nThis exporter work with a dedicated config’s file that’s contain all needed metrics.\n \nEach dedicated metrics will use a collector to access to data and a custom commands list that's will be run into this collector.\nThe final command must expose a ready to be parsed result set to extract metric value and tags value. \nA mapping config allows to specify the tags included otherwise, only last data on each row or record will be extract.\n  \nA collector is define by his name and include the type (bash, mysql ...) and credential (data source name, user, password, uri ...). \n\nThe dedicated metrics are exposed with a prefix \"custom\" and the name of this metrics extract from the config file.\n \n## How it's work\nThe main process will load the config file and will register a dedicated collector into Prometheus client framework for each metrics. Each metric is composed of a collector helper who's include a specific type collector defined into the config file\n\nOn each call to the metrics path of the exporter (i.e. http://localhost:9213/metrics/), the main process will call each registered Prometheus collectors in multithreading and grab all results to expose them to the caller.\n\nIf a metrics is not available (errors on running command, result empty ...) a minimal result will be exposing. When this metric’s commands rise up, the result will appear. If the config of a metrics is not well defined, the metrics will be not registered into the main process. If no metrics are registered, the main process will exit with an error status.\n\n## Build from source \n\n\u003e Requirement : go version \u003e= 1.13 (using go mod)\n\nTo build from source, a makefile is available in the repos, so the easiest build process is :\n```bash\n\ngo get -u github.com/orange-cloudfoundry/custom_exporter\ncd $GOPATH/src/github.com/orange-cloudfoundry/custom_exporter\n\nmake\n\n```\n\nor building with [`promu` tools](https://github.com/prometheus/promu): \n```bash\n\ngo get -u github.com/prometheus/promu\ncd $GOPATH/src/github.com/prometheus/promu\n\nmake\n\ngo get -u github.com/orange-cloudfoundry/custom_exporter\ncd $GOPATH/src/github.com/orange-cloudfoundry/custom_exporter\n\n$GOPATH/bin/promu build --prefix $GOPATH/bin\n\n```\nNote: [a bosh release for cloudfoundry](https://github.com/orange-cloudfoundry/custom_exporter-boshrelease) is available at github\n\n## Configuration \n\nThe configuration is split in 2 separate parts:\n * **credentials**: provide credentials an data type to the custom export.\n * **metrics**: provide commands that are to be run to retrieve metrics and key-value mapping\n\n#### Credential\nThe credential section is composed at least as:\n\n  * **name**: name of the credential \n  * **type**: collector type (one of existing collector : redis, mysql, bash, ...). If the type is not understand the metrics connected to this credential will be ignored\n  \nThis other options depends of collectors:\n\n| Option Name | Description | Collector |\n| :---------: | :---------- | :-------: |\n| dsn | the DSN (Data Source Name) is an URL like string usually use to connect to database | mysql, redis | \n| user | the user to run command in shell process | bash |\n\nThe DSN form example for each collector: \n\n    * mysql: driver://user:password@protocol(addr:port|[addr_ip_v6]:port|socket)/database\n    * redis: protocol://\u003cempty\u003e:password@host:port/database\n \n#### Metric\nThe metrics section is composed at least as:\n\n  * **name**: name of the metrics\n  * **commands**: list of command to run to retrieve the metrics tags and value\n  * **credential**: the credential's name to use in this metrics (cannot be null : collector type is include in the credential)\n  * **value_type**: the prometheus value type (COUNTER, GAUGE, UNTYPED)\n  \nThis others options are optionals: \n\n| Option Name | Description | Collector |\n| :---------: | :---------- | :-------: |\n| mapping | the list of tags to be found in result set | all |\n| separator | the separator used in some collector like bash | bash |\n| value_name | the name of the metric value key who's be found in result of command | redis |\n\n## Manifest \u0026 result examples\n### First example\n#### Manifest\n```yaml\ncustom_exporter:\n  credentials:\n  - name: mysql_credential_tcp\n    type: mysql \n    dsn: mysql://user:password@tcp(127.0.0.1:3306)/database_name\n  - name: mysql_credential_socket\n    type: mysql \n    dsn: mysql://user:password@unix(/var/lib/mysql/mysql.sock)/database_name\n  - name: shell_credential\n    type: bash\n    user: root\n  - name: redis_credential\n    type: redis\n    dsn: tcp://:password@127.0.0.1:1234/0\n  metrics:\n  - name: node_database_size_bytes\n    commands:\n    - find /var/vcap/store/mysql/ -type d -name cf* -exec du -sb {} ;\n    - sed -ne s/^\\([0-9]\\+\\)\\t\\(\\/var\\/vcap\\/store\\/mysql\\/\\)\\(.*\\)$/\\3 \\1/p\n    credential: shell_credential\n    mapping:\n    - database\n    separator: ' '\n    value_type: UNTYPED\n  - name: node_database_provisioning_bytes\n    commands:\n    - select db_name,max_storage_mb*1024*1024 FROM mysql_broker.service_instances;\n    credential: mysql_credential\n    mapping:\n    - database\n    value_type: UNTYPED\n  - name: node_redis_info\n    commands:\n    - INFO REPLICATION\n    credential: redis_credential\n    mapping:\n    - role\n    value_name: value\n    value_type: UNTYPED\n```\n\n#### Results returned in the custom exporter\n\n```bash\n[08:53:09] BOSH MySQL ~ # curl -s 10.234.250.202:9100/metrics | grep -i 'node_database'\n# HELP node_database_provisioning_bytes Metric read from /var/vcap/jobs/node_exporter/config/database_provisioning.prom\n# TYPE node_database_provisioning_bytes untyped\ncustom_node_database_provisioning_bytes{database=\"cf_74df5b8f_e7fe_4151_8ec3_741296d42fbc\"} 1.048576e+09\ncustom_node_database_provisioning_bytes{database=\"cf_d7161ef3_e6fc_4a05_9631_834525f0f7ba\"} 1.048576e+09\ncustom_node_database_provisioning_bytes{database=\"cf_fa61054d_5c08_4734_a31e_4f2e6065897b\"} 1.048576e+08\n# HELP node_database_size_bytes Metric read from /var/vcap/jobs/node_exporter/config/database_size.prom\n# TYPE node_database_size_bytes untyped\ncustom_node_database_size_bytes{database=\"cf_74df5b8f_e7fe_4151_8ec3_741296d42fbc\"} 4157\ncustom_node_database_size_bytes{database=\"cf_d7161ef3_e6fc_4a05_9631_834525f0f7ba\"} 4157\ncustom_node_database_size_bytes{database=\"cf_fa61054d_5c08_4734_a31e_4f2e6065897b\"} 4157\n```\n\n### Another example \n#### Manifest \n```yaml\ncustom_exporter:\n  credentials:\n  - name: mysql_connector\n    type: mysql ##Possible types are for the moment shell mysql redis\n    dsn: mysql://root:password@1.2.3.4:1234/mydb\n  metrics:\n  - name: custom_metric\n    commands:\n    - 1\n    - 2\n    - 3\n    credential: mysql_connector\n    mapping:\n    - tag1\n    - tag2\n    value_type: UNTYPED\n    separator: \\t #useless for MySQL but can be usefull for shell\n```\n\n#### Result example (MySQL view)\n```mysql\n|  1 | chicken | 128 |\n|  2 | beef | 256 |\n|  3 | snails | 14 | \n```\n\n#### Result example (Exporter view)\n```bash\ncustom_metric{tag1=\"1\",tag2=\"chicken\",instance=\"ip:port\",job=\"custom_exporter\"}  128\ncustom_metric{tag1=\"2\",tag2=\"beef\",instance=\"ip:port\",job=\"custom_exporter\"}  256\ncustom_metric{tag1=\"3\",tag2=\"snails\",instance=\"ip:port\",job=\"custom_exporter\"}  14\n```\n\n## Port binding\nAccording to https://github.com/prometheus/prometheus/wiki/Default-port-allocations we will use TCP/9209\n\n## WIP : Working schema\n![custom_exporter_working_schema](custom_exporter.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forange-cloudfoundry%2Fcustom_exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forange-cloudfoundry%2Fcustom_exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forange-cloudfoundry%2Fcustom_exporter/lists"}