{"id":36693144,"url":"https://github.com/jdevelop/s3kit","last_synced_at":"2026-01-12T11:24:06.378Z","repository":{"id":80673089,"uuid":"255490066","full_name":"jdevelop/s3kit","owner":"jdevelop","description":"AWS S3 CLI tooklit","archived":false,"fork":false,"pushed_at":"2020-10-30T22:14:48.000Z","size":91,"stargazers_count":22,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-11-23T18:05:24.842Z","etag":null,"topics":["aws-s3","cli-app"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jdevelop.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-14T02:24:46.000Z","updated_at":"2025-01-27T22:33:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"73d2dc59-2f29-4819-ae9c-733ad543d546","html_url":"https://github.com/jdevelop/s3kit","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/jdevelop/s3kit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdevelop%2Fs3kit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdevelop%2Fs3kit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdevelop%2Fs3kit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdevelop%2Fs3kit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jdevelop","download_url":"https://codeload.github.com/jdevelop/s3kit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdevelop%2Fs3kit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28338971,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T10:58:46.209Z","status":"ssl_error","status_checked_at":"2026-01-12T10:58:42.742Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","cli-app"],"created_at":"2026-01-12T11:24:05.609Z","updated_at":"2026-01-12T11:24:06.369Z","avatar_url":"https://github.com/jdevelop.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Purpose\n\nSimplify getting some basic statistics about content of AWS S3 buckets, that are missing or not very convenient to use in AWS CLI. Supports the following commands:\n\n|Command | Description|\n| -------- |:-----------|\n|  **cat**   | Print content of S3 file(s) to stdout|\n|  **help**    | Help about any command |\n|  **lock**    | Manage object locks |\n|  **logs**    | Print S3 access logs as JSON |\n|  **ls**      | List versions and/legal holds and locks |\n|  **parquet** | Parquet files explorer |\n|  **size**    | Calculate size of S3 location |\n|  **tag**     | Tag S3 object(s) |\n\nCould be quite useful for a fellow data engineer.\n\n## Build / install\n\n### Sources\n\n`go install`\n\n### MacOS\n\n```\nbrew tap jdevelop/s3kit\nbrew install s3kit\n```\n\n### Binary release\n\n- [1.1.0](https://github.com/jdevelop/s3kit/releases/tag/v1.1.0)\n- [1.2.0](https://github.com/jdevelop/s3kit/releases/tag/v1.2.0)\n- [1.3.0](https://github.com/jdevelop/s3kit/releases/tag/v1.3.0)\n\n## Usage\n\n```\nAWS S3 command line toolkit\n\nUsage:\n  s3kit [command]\n\nAvailable Commands:\n  cat         Print content of S3 file(s) to stdout\n  help        Help about any command\n  lock        Manage object locks\n  logs        Print S3 Access logs as JSON\n  parquet     Parquet files explorer\n  size        Calculate size of S3 location\n\nFlags:\n      --debug         print debug messages\n  -h, --help          help for s3kit\n      --quiet         print warnings and errors\n  -w, --workers int   number of concurrent threads (default 12)\n\nUse \"s3kit [command] --help\" for more information about a command.\n```\n\n### s3kit cat\n\nOften you want to view content of a file on S3, or perhaps *all* of them in a certain path. \nIf we're considering data engineering - often these files are represented as compressed CSV files.\nThere's no easy way to view the file, so basically the `cat` command can identify the file ( by extension ) and uncompress it to `stdout`.\n\n```\nPrint content of S3 file(s) to stdout\n\nUsage:\n  s3kit cat s3://bucket/key1 s3://bucket/prefix/ ... [flags]\n\nFlags:\n  -h, --help   help for cat\n\nGlobal Flags:\n  -w, --workers int   number of concurrent threads (default 12)\n```\n\n#### Example\n`s3kit cat s3://bucket/path` will print out content of all files under path prefix `s3://bucket/path`\n\n\n\n### s3kit logs\n\nLogs command will take the folder/file that contains the access logs for a [static website](https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html) hosted on an S3 bucket.\n\n```\nPrint S3 Access logs as JSON\n\nUsage:\n  s3kit logs s3://bucket/key1 s3://bucket/prefix/ ... [flags]\n\nFlags:\n  -e, --end Date     end date ( YYYY-MM-DD ) (default 2020-04-18 20:00:00 -0400 EDT)\n  -h, --help         help for logs\n  -s, --start Date   start date ( YYYY-MM-DD ) (default 0001-01-01 00:00:00 +0000 UTC)\n\nGlobal Flags:\n  -w, --workers int   number of concurrent threads (default 12)\n\n```\n\n#### Example\n```\ns3kit logs s3://logs-bucket/accesslog/\n```\n\nIt will print JSON objects ( one per line ) as:\n```\n{\n  \"time\": \"2020-04-13T13:27:30Z\",\n  \"remote_ip\": \"103.249.100.196\",\n  \"operation\": \"WEBSITE.GET.OBJECT\",\n  \"key\": \"web/wp-includes/wlwmanifest.xml\",\n  \"request_uri\": \"GET /web/wp-includes/wlwmanifest.xml HTTP/1.1\",\n  \"http_status\": 403,\n  \"bytes_sent\": 303,\n  \"object_size\": 0,\n  \"flight_time\": 59000000,\n  \"turnaround_time\": 0,\n  \"referer\": \"-\",\n  \"user_agent\": \"-\"\n}\n```\n\nWith `jq` it is quite easy to get top IP addresses that visited the website:\n```\nlogs s3://logs-bucket/accesslog/ | grep \"WEBSITE.GET.OBJECT\" | jq \".remote_ip\" | sort | uniq -c | sort -n\n```\n\n### s3kit size\n\nGetting size of a bucket / path prefix can be achieved with `aws s3 ls --summarize`, however it doesn't have the functionality to collect size of the top-level folders under S3 prefix, which might be useful.\n\n```\nCalculate size of S3 location\n\nUsage:\n  s3kit size  s3://bucket/key1 s3://bucket/prefix/ ... [flags]\n\nFlags:\n  -g, --group   group sizes by top-level folders\n  -h, --help    help for size\n      --json    output as JSON array\n      --raw     raw numbers, no human-formatted size\n\nGlobal Flags:\n  -w, --workers int   number of concurrent threads (default 12)\n```\n\n#### Example\n\n```\ns3kit size s3://dataeng-data/ -g\n+--------------------------------+-------+--------+\n|              PATH              | COUNT |  SIZE  |\n+--------------------------------+-------+--------+\n|  s3://dataeng-data/categories/ |    17 | 62 kB  |\n|  s3://dataeng-data/meetups/    |    18 | 15 MB  |\n|  s3://dataeng-data/members/    | 96599 | 8.2 GB |\n+--------------------------------+-------+--------+\n|             TOTAL:             | 96634 | 8.2 GB |\n+--------------------------------+-------+--------+\n```\n\nIt is also possible to get JSON output:\n```\n[\n  {\n    \"Path\": \"s3://dataeng-data/categories/\",\n    \"Count\": 17,\n    \"Size\": 61684\n  },\n  {\n    \"Path\": \"s3://dataeng-data/meetups/\",\n    \"Count\": 18,\n    \"Size\": 15408356\n  },\n  {\n    \"Path\": \"s3://dataeng-data/members/\",\n    \"Count\": 96599,\n    \"Size\": 8202309787\n  }\n]\n```\n\n### s3kit lock compliance\nAdds the [compliance lock](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html) to a given object identified by a prefix and applicable to all versions of the object(s), latest version of the object(s) or specific version of the object(s).\n\n```\nAdd compliance lock\n\nUsage:\n  s3kit lock compliance s3://bucket/key1 s3://bucket/prefix/ ... [flags]\n\nFlags:\n      --all               Apply to all versions of object(s)\n      --expire duration   compliance lock duration (1m, 1h etc)\n  -h, --help              help for compliance\n      --latest            Apply to latest version of object(s) (default true)\n      --version string    Apply to a specific version\n\nGlobal Flags:\n  -w, --workers int   number of concurrent threads (default 12)\n```\n\nThis operation will explicitly ask for confirmation prior to applying to the object version, because there's no way to revert the compliance lock:\n\n```\ns3kit lock compliance s3://demolocal123/go.mod --expire 10s\nLocking s3://bucket/key version .oNOa6ZVNDTUcKHaaaECJUmdA9XaBTI4 expires 2020-04-18 15:31:35, proceed? (y/N):\n```\n\nThe default answer is **NO**.\n\n### s3kit lock legal add / rm\n\nAdds or removes the [legal hold](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html) to the object(s) found by given S3 prefix(es).\n\n```\nAdd/remove legal hold\n\nUsage:\n  s3kit lock legal [command]\n\nAvailable Commands:\n  add         Add legal hold for given object(s)\n  rm          Remove legal hold for given object(s)\n\nFlags:\n      --all              Apply to all versions of object(s)\n  -h, --help             help for legal\n      --latest           Apply to latest version of object(s) (default true)\n      --version string   Apply to a specific version\n\nGlobal Flags:\n  -w, --workers int   number of concurrent threads (default 12)\n\nUse \"s3kit lock legal [command] --help\" for more information about a command.\n```\n\n### s3kit lock governance add / rm\n\nAdds or removes the [governance retention lock](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html) to the object(s) found by given S3 prefix(es).\n\n```\nAdd/remove governance lock\n\nUsage:\n  s3kit lock governance [command]\n\nAvailable Commands:\n  add         Add governance lock for given object(s)\n  rm          Remove governance lock for given object(s)\n\nFlags:\n      --all              Apply to all versions of object(s)\n  -h, --help             help for governance\n      --latest           Apply to latest version of object(s) (default true)\n      --version string   Apply to a specific version\n\nGlobal Flags:\n  -w, --workers int   number of concurrent threads (default 12)\n\nUse \"s3kit lock governance [command] --help\" for more information about a command.\n```\n\n### s3kit parquet schema \n```\nPrint parquet files schema\n\nUsage:\n  s3kit parquet schema s3://bucket/prefix/key s3://bucket/prefix/ ... [flags]\n\nFlags:\n  -h, --help       help for schema\n      --json       JSON output\n      --keys int   max parquet files to process (default 1)\n\nGlobal Flags:\n  -w, --workers int   number of concurrent threads (default 12)\n```\n\n### s3kit ls locks\n```\nList various locks on S3 object(s) (legal hold, governance/compliance retention)\n\nUsage:\n  s3kit ls locks s3://bucket/folder/ s3://bucket/folder/prefix ... [flags]\n\nFlags:\n      --all              Apply to all versions of object(s)\n  -h, --help             help for locks\n      --latest           Apply to latest version of object(s) (default true)\n      --version string   Apply to a specific version\n\nGlobal Flags:\n      --json          JSON output\n      --table         ASCII table output (default true)\n  -w, --workers int   number of concurrent threads (default 12)\n      --yaml          YAML output\n```\n\n### s3kit ls tags\n```\nList tags for object(s)\n\nUsage:\n  s3kit ls tags s3://bucket/folder/ s3://bucket/folder/prefix ... [flags]\n\nFlags:\n      --all              Apply to all versions of object(s)\n  -h, --help             help for tags\n      --latest           Apply to latest version of object(s) (default true)\n      --version string   Apply to a specific version\n\nGlobal Flags:\n      --json          JSON output\n      --table         ASCII table output (default true)\n  -w, --workers int   number of concurrent threads (default 12)\n      --yaml          YAML output\n```\n\n### s3kit ls versions\n```\nList object version(s)\n\nUsage:\n  s3kit ls versions s3://bucket/folder/ s3://bucket/folder/prefix ... [flags]\n\nFlags:\n  -h, --help   help for versions\n\nGlobal Flags:\n      --json          JSON output\n      --table         ASCII table output (default true)\n  -w, --workers int   number of concurrent threads (default 12)\n      --yaml          YAML output\n```\n\n### s3kit tag add\n```\nAdd tag(s) to S3 object(s)\n\nUsage:\n  s3kit tag add s3://bucket/folder/ s3://bucket/folder/prefix ... [flags]\n\nFlags:\n  -h, --help           help for add\n      --tags strings   tags as --tags 'tag1=value1,tag2=value2' or multiple --tags ... options\n\nGlobal Flags:\n      --all              Apply to all versions of object(s)\n      --latest           Apply to latest version of object(s) (default true)\n      --version string   Apply to a specific version\n  -w, --workers int      number of concurrent threads (default 12)\n```\n\n### s3kit tag rm\n```\nRemove tag(s) from S3 object(s)\n\nUsage:\n  s3kit tag rm s3://bucket/folder/ s3://bucket/folder/prefix ... [flags]\n\nFlags:\n  -h, --help           help for rm\n      --tags strings   tags as --tags 'tag1,tag2' or multiple --tags ... options\n\nGlobal Flags:\n      --all              Apply to all versions of object(s)\n      --latest           Apply to latest version of object(s) (default true)\n      --version string   Apply to a specific version\n  -w, --workers int      number of concurrent threads (default 12)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdevelop%2Fs3kit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjdevelop%2Fs3kit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdevelop%2Fs3kit/lists"}