{"id":30740989,"url":"https://github.com/thannaske/s3usage","last_synced_at":"2025-09-04T00:18:33.739Z","repository":{"id":280039356,"uuid":"940808330","full_name":"thannaske/s3usage","owner":"thannaske","description":"This tool collects your Ceph RGW S3 bucket utilisations and calculates a monthly average from all datapoints for usage-based billing.","archived":false,"fork":false,"pushed_at":"2025-03-04T18:22:26.000Z","size":32,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-18T13:08:08.841Z","etag":null,"topics":["billing","ceph-radosgw","rgw","rgwadmin"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thannaske.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-28T20:42:23.000Z","updated_at":"2025-03-04T18:31:33.000Z","dependencies_parsed_at":"2025-02-28T23:56:30.515Z","dependency_job_id":"622ec258-6507-486c-a199-792cb5040e59","html_url":"https://github.com/thannaske/s3usage","commit_stats":null,"previous_names":["thannaske/s3usage"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/thannaske/s3usage","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thannaske%2Fs3usage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thannaske%2Fs3usage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thannaske%2Fs3usage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thannaske%2Fs3usage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thannaske","download_url":"https://codeload.github.com/thannaske/s3usage/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thannaske%2Fs3usage/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273530727,"owners_count":25122052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["billing","ceph-radosgw","rgw","rgwadmin"],"created_at":"2025-09-04T00:18:30.665Z","updated_at":"2025-09-04T00:18:33.724Z","avatar_url":"https://github.com/thannaske.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S3Usage - S3 Bucket Usage Monitor for Ceph\n![GitHub Downloads (all assets, all releases)](https://img.shields.io/github/downloads/thannaske/s3usage/total?style=flat)\n![GitHub License](https://img.shields.io/github/license/thannaske/s3usage)\n![GitHub Release](https://img.shields.io/github/v/release/thannaske/s3usage?include_prereleases)\n\nS3Usage is a command-line application that monitors storage usage of S3 buckets in a Ceph cluster for billing purposes. It uses Ceph's RGW Admin Ops API to efficiently retrieve bucket statistics, stores the data in a SQLite database, calculates monthly averages, and provides commands to query historical usage information.\n\n## Features\n\n- Connect to Ceph S3 RGW Admin API to efficiently retrieve bucket usage statistics\n- Store usage data in a SQLite database\n- Calculate monthly average bucket usage\n- Display monthly usage for all buckets\n- Query historical usage for specific buckets\n- Prune old data points while preserving monthly statistics\n\n## Implementation Details\n\nThis tool uses the Ceph RGW Admin Ops API and is not compatibile to any other S3-compatible system other than Ceph RGW.\n\nThe Admin API requests are properly signed using AWS Signature v4 authentication, which is required by Ceph RGW. Using the Admin Ops API provides significant performance improvements over listing all objects in buckets, making this tool suitable for monitoring Ceph S3 deployments with many large buckets.\n\n## Authentication Requirements\n\nThe tool uses the AWS Signature v4 authentication method for Admin API requests, identical to the authentication used by the `radosgw-admin` CLI tool. This requires:\n\n1. A Ceph user with administrative privileges\n2. The access and secret keys for that user\n3. The correct Ceph RGW endpoint URL\n\n## Installation\n\n### From Source\n\n```bash\ngit clone https://github.com/thannaske/s3usage.git\ncd s3usage\ngo build -o s3usage ./cmd/s3usage\n# Optionally, move to a directory in your PATH\nsudo mv s3usage /usr/local/bin/\n```\n\n## Usage\n\n### Environment Variables\n\nYou can configure S3Usage using environment variables:\n\n- `S3_ENDPOINT`: Ceph S3 endpoint URL (should be the RGW API endpoint)\n- `S3_ACCESS_KEY`: S3 access key (requires admin privileges for RGW Admin API)\n- `S3_SECRET_KEY`: S3 secret key\n- `S3_REGION`: S3 region (default: \"default\")\n- `S3_DB_PATH`: Path to SQLite database (default: `~/.s3usage.db`)\n\n### Required Permissions\n\nThe access key used must have administrative privileges on the Ceph RGW to access the Admin API endpoints. You can create a user with the appropriate permissions using:\n\n```bash\nradosgw-admin user create --uid=s3usage --display-name=\"S3 Usage Monitor\" --caps=\"buckets=*;users=*;usage=*;metadata=*;zone=*\"\nradosgw-admin key create --uid=s3usage --key-type=s3 --gen-access-key --gen-secret\n```\n\n### Collecting Usage Data\n\nTo collect bucket usage data and store it in the database, use the `collect` command:\n\n```bash\ns3usage collect --endpoint=https://s3.example.com --access-key=YOUR_ACCESS_KEY --secret-key=YOUR_SECRET_KEY\n```\n\nOr using environment variables:\n\n```bash\nexport S3_ENDPOINT=https://s3.example.com\nexport S3_ACCESS_KEY=YOUR_ACCESS_KEY\nexport S3_SECRET_KEY=YOUR_SECRET_KEY\ns3usage collect\n```\n\nThis command is meant to be scheduled via cron to collect data regularly. Running the collector more often will result in more datapoint and hence in a more precise monthly average usage.\n\n### Monthly Usage Report\n\nTo display the monthly average usage for all buckets:\n\n```bash\ns3usage list --year=2025 --month=2\n```\n\nIf no year/month is specified, the previous month's data is shown.\n\n### Bucket Usage History\n\nTo view historical usage data for a specific bucket:\n\n```bash\ns3usage history my-bucket-name\n```\n\nThis shows a year's worth of historical data for the specified bucket.\n\n### Pruning Old Data\n\nTo clean up individual data points from months that have already been aggregated into monthly averages:\n\n```bash\ns3usage prune\n```\n\nThis will prompt for confirmation before deleting data. To skip the confirmation:\n\n```bash\ns3usage prune --confirm\n```\n\nThe prune command only removes data points from completed months that already have monthly averages calculated. It preserves:\n- All monthly average statistics\n- Data points from the current month\n- Data points from months without calculated averages\n\nThis helps keep the database size manageable over time without losing valuable statistics.\n\n## Cron Setup\n\nTo collect data daily, add a cron job:\n\n```bash\n# Edit crontab\ncrontab -e\n\n# Add this line to run daily at 23:45\n45 23 * * * /usr/local/bin/s3usage collect --endpoint=https://s3.example.com --access-key=YOUR_ACCESS_KEY --secret-key=YOUR_SECRET_KEY\n```\n\n## Troubleshooting\n\nIf you encounter authentication issues:\n\n1. Verify your user has the correct admin capabilities\n2. Ensure your endpoint URL is correct (it should point to the RGW API endpoint)\n3. Check that you're using the correct access and secret keys\n4. Verify the region setting matches your Ceph configuration\n\n## License\n\nMIT \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthannaske%2Fs3usage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthannaske%2Fs3usage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthannaske%2Fs3usage/lists"}