{"id":13582220,"url":"https://github.com/ribbybibby/s3_exporter","last_synced_at":"2025-10-24T00:34:38.948Z","repository":{"id":28425719,"uuid":"118349264","full_name":"ribbybibby/s3_exporter","owner":"ribbybibby","description":"Exports Prometheus metrics about S3 buckets and objects","archived":false,"fork":false,"pushed_at":"2024-07-18T13:35:56.000Z","size":6441,"stargazers_count":114,"open_issues_count":19,"forks_count":46,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-11T13:47:49.011Z","etag":null,"topics":["aws","aws-s3","metrics","monitoring","prometheus","prometheus-exporter","prometheus-metrics","s3-bucket"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ribbybibby.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-21T15:54:03.000Z","updated_at":"2025-03-17T14:42:49.000Z","dependencies_parsed_at":"2024-01-19T08:08:46.092Z","dependency_job_id":"f730b19c-a2ea-46a1-a162-942125252a82","html_url":"https://github.com/ribbybibby/s3_exporter","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/ribbybibby/s3_exporter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ribbybibby%2Fs3_exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ribbybibby%2Fs3_exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ribbybibby%2Fs3_exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ribbybibby%2Fs3_exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ribbybibby","download_url":"https://codeload.github.com/ribbybibby/s3_exporter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ribbybibby%2Fs3_exporter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261249775,"owners_count":23130494,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","metrics","monitoring","prometheus","prometheus-exporter","prometheus-metrics","s3-bucket"],"created_at":"2024-08-01T15:02:30.240Z","updated_at":"2025-10-24T00:34:38.879Z","avatar_url":"https://github.com/ribbybibby.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# AWS S3 Exporter\n\nThis exporter provides metrics for AWS S3 bucket objects by querying the API with a given bucket and prefix and constructing metrics based on the returned objects.\n\nI find it useful for ensuring that backup jobs and batch uploads are functioning by comparing the growth in size/number of objects over time, or comparing the last modified date to an expected value.\n\n## Building\n\n```\nmake\n```\n\n## Running\n\n```\n./s3_exporter \u003cflags\u003e\n```\n\nYou can query a bucket and prefix combination by supplying them as parameters to /probe:\n\n```\ncurl localhost:9340/probe?bucket=some-bucket\u0026prefix=some-folder/some-file.txt\n```\n\n### AWS Credentials\n\nThe exporter creates an AWS session without any configuration. You must specify credentials yourself as documented [here](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html).\n\nRemember, if you want to load credentials from `~/.aws/config` then you need to to set:\n\n```\nexport AWS_SDK_LOAD_CONFIG=true\n```\n\n### Docker\n\n```\ndocker pull ribbybibby/s3-exporter\n```\n\nYou will need to supply AWS credentials to the container, as mentioned in the previous section, either by setting the appropriate environment variables with `-e`, or by mounting your `~/.aws/` directory with `-v`.\n\n```\n# Environment variables\ndocker run -p 9340:9340 -e AWS_ACCESS_KEY_ID=\u003cvalue\u003e -e AWS_SECRET_ACCESS_KEY=\u003cvalue\u003e -e AWS_REGION=\u003cvalue\u003e s3-exporter:latest \u003cflags\u003e\n# Mounted volume\ndocker run -p 9340:9340 -e AWS_SDK_LOAD_CONFIG=true -e HOME=/ -v $HOME/.aws:/.aws s3-exporter:latest \u003cflags\u003e\n```\n\n## Flags\n\n```\n  -h, --help                     Show context-sensitive help (also try --help-long and --help-man).\n      --web.listen-address=\":9340\"\n                                 Address to listen on for web interface and telemetry.\n      --web.metrics-path=\"/metrics\"\n                                 Path under which to expose metrics\n      --web.probe-path=\"/probe\"  Path under which to expose the probe endpoint\n      --web.discovery-path=\"/discovery\"\n                                 Path under which to expose service discovery\n      --s3.endpoint-url=\"\"       Custom endpoint URL\n      --s3.disable-ssl           Custom disable SSL\n      --s3.force-path-style      Custom force path style\n      --log.level=\"info\"         Only log messages with the given severity or above. Valid levels: [debug, info, warn, error, fatal]\n      --log.format=\"logger:stderr\"\n                                 Set the log target and format. Example: \"logger:syslog?appname=bob\u0026local=7\" or \"logger:stdout?json=true\"\n      --version                  Show application version.\n```\n\nFlags can also be set as environment variables, prefixed by `S3_EXPORTER_`. For example: `S3_EXPORTER_S3_ENDPOINT_URL=http://s3.example.local`.\n\n## Metrics\n\n| Metric                             | Meaning                                                                                                     | Labels                    |\n| ---------------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------- |\n| s3_biggest_object_size_bytes       | The size of the largest object.                                                                             | bucket, prefix            |\n| s3_common_prefixes                 | A count of all the keys between the prefix and the next occurrence of the string specified by the delimiter | bucket, prefix, delimiter |\n| s3_last_modified_object_date       | The modification date of the most recently modified object.                                                 | bucket, prefix            |\n| s3_last_modified_object_size_bytes | The size of the object that was modified most recently.                                                     | bucket, prefix            |\n| s3_list_duration_seconds           | The duration of the ListObjects operation                                                                   | bucket, prefix, delimiter |\n| s3_list_success                    | Did the ListObjects operation complete successfully?                                                        | bucket, prefix, delimiter |\n| s3_objects_size_sum_bytes          | The sum of the size of all the objects.                                                                     | bucket, prefix            |\n| s3_objects                         | The total number of objects.                                                                                | bucket, prefix            |\n\n## Common prefixes\n\nRather than generating metrics for the objects with a particular prefix, you can\nset the `delimiter` parameter to produce a count of all the keys between the\nprefix and the next occurrence of the given delimiter.\n\nFor instance:\n```\n$ curl 'localhost:9340/probe?bucket=registry-bucket\u0026prefix=docker/registry/v2/blobs/sha256/\u0026delimiter=/'\n# HELP s3_common_prefixes A count of all the keys between the prefix and the next occurrence of the string specified by the delimiter\n# TYPE s3_common_prefixes gauge\ns3_common_prefixes{bucket=\"registry-bucket\",delimiter=\"/\",prefix=\"docker/registry/v2/blobs/sha256/\"} 133\n# HELP s3_list_duration_seconds The total duration of the list operation\n# TYPE s3_list_duration_seconds gauge\ns3_list_duration_seconds{bucket=\"registry-bucket\",delimiter=\"/\",prefix=\"docker/registry/v2/blobs/sha256/\"} 0.921488535\n# HELP s3_list_success If the ListObjects operation was a success\n# TYPE s3_list_success gauge\ns3_list_success{bucket=\"registry-bucket\",delimiter=\"/\",prefix=\"docker/registry/v2/blobs/sha256/\"} 1\n```\n\nSee [this\npage](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ListingKeysUsingAPIs.html)\nfor more information.\n\n## Prometheus\n\n### Configuration\n\nYou can pass the params to a single instance of the exporter using relabelling, like so:\n\n```yml\nscrape_configs:\n  - job_name: \"s3\"\n    metrics_path: /probe\n    static_configs:\n      - targets:\n          - bucket=stuff;prefix=thing.txt;\n          - bucket=other-stuff;prefix=another-thing.gif;\n    relabel_configs:\n      - source_labels: [__address__]\n        regex: \"^bucket=(.*);prefix=(.*);$\"\n        replacement: \"${1}\"\n        target_label: \"__param_bucket\"\n      - source_labels: [__address__]\n        regex: \"^bucket=(.*);prefix=(.*);$\"\n        replacement: \"${2}\"\n        target_label: \"__param_prefix\"\n      - target_label: __address__\n        replacement: 127.0.0.1:9340 # S3 exporter.\n```\n\n### Service Discovery\n\nRather than defining a static list of buckets you can use the `/discovery` endpoint\nin conjunction with HTTP service discovery to discover all the buckets the\nexporter has access to.\n\nThis should be all the config required to successfully scrape every bucket:\n\n```yml\nscrape_configs:\n  - job_name: \"s3\"\n    metrics_path: /probe\n    http_sd_configs:\n      - url: http://127.0.0.1:9340/discovery\n```\n\nUse `relabel_configs` to select the buckets you want to scrape:\n\n```yml\nscrape_configs:\n  - job_name: \"s3\"\n    metrics_path: /probe\n    http_sd_configs:\n      - url: http://127.0.0.1:9340/discovery\n    relabel_configs:\n      # Keep buckets that start with example-\n      - source_labels: [__param_bucket]\n        action: keep\n        regex: ^example-.*\n```\n\nThe prefix can be set too, but be mindful that this will apply to all buckets:\n\n```yml\nscrape_configs:\n  - job_name: \"s3\"\n    metrics_path: /probe\n    http_sd_configs:\n      - url: http://127.0.0.1:9340/discovery\n    params:\n      prefix: [\"thing.txt\"]\n```\n\n### Example Queries\n\nReturn series where the last modified object date is more than 24 hours ago:\n\n```\n(time() - s3_last_modified_object_date) / 3600 \u003e 24\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fribbybibby%2Fs3_exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fribbybibby%2Fs3_exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fribbybibby%2Fs3_exporter/lists"}