{"id":29682190,"url":"https://github.com/koluku/s3s","last_synced_at":"2025-07-23T02:08:31.793Z","repository":{"id":37102158,"uuid":"496256210","full_name":"koluku/s3s","owner":"koluku","description":"Easy S3 select like searching in directories","archived":false,"fork":false,"pushed_at":"2024-06-17T23:30:36.000Z","size":252,"stargazers_count":17,"open_issues_count":6,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-06-19T06:54:25.004Z","etag":null,"topics":["aws","s3","s3select"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koluku.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-25T14:03:14.000Z","updated_at":"2024-04-26T09:32:03.000Z","dependencies_parsed_at":"2024-04-30T00:30:54.778Z","dependency_job_id":"5168478e-b8ae-49e4-93dd-37408d638dc0","html_url":"https://github.com/koluku/s3s","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/koluku/s3s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koluku%2Fs3s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koluku%2Fs3s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koluku%2Fs3s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koluku%2Fs3s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koluku","download_url":"https://codeload.github.com/koluku/s3s/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koluku%2Fs3s/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266604009,"owners_count":23954725,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","s3","s3select"],"created_at":"2025-07-23T02:08:31.205Z","updated_at":"2025-07-23T02:08:31.783Z","avatar_url":"https://github.com/koluku.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3s\n\n**s3s** is a go binary instead of [vast-engineering/s3select](https://github.com/vast-engineering/s3select).\n\n## Features\n\ns3s query all files lower than S3 prefix.\n\nAvailable below:\n\n- Input JSON to Output JSON\n- Input CSV to Output JSON\n- Input Application Load Balancer Logs to Output JSON\n- Input CloudFront Logs to Output JSON\n\n## Usage\n\n```console\n$ s3s --help\nNAME:\n   s3s - Easy S3 select like searching in directories\n\nUSAGE:\n   s3s [global options] command [command options] [arguments...]\n\nVERSION:\n   current\n\nCOMMANDS:\n   help, h  Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --debug        erorr check for developer (default: false)\n   --help, -h     show help\n   --version, -v  print the version\n\n   AWS:\n\n   --max-retries value, -M value   max number of api requests to retry (default: 20)\n   --region value                  region of target s3 bucket exist (default: ENV[\"AWS_REGION\"])\n   --thread-count value, -t value  max number of api requests to concurrently (default: 150)\n\n   Input Format:\n\n   --alb-logs, --alb_logs  (default: false)\n   --cf-logs, --cf_logs    (default: false)\n   --csv                   (default: false)\n\n   Query:\n\n   --count, -c              max number of results from each key to return (default: false)\n   --limit value, -l value  max number of results from each key to return (default: 0)\n   --query value, -q value  a query for S3 Select\n   --where value, -w value  WHERE part of the query\n\n   Run:\n\n   --delve               like directory move before querying (default: false)\n   --dry-run, --dry_run  pre request for s3 select (default: false)\n\n   Target:\n\n   --duration value  from current time if alb or cf (ex: \"2h3m\") (default: 0s)\n   --since value     end at if alb or cf (ex: \"2006-01-02 15:04:05\")\n   --until value     start at if alb or cf (ex: \"2006-01-02 15:04:05\")\n```\n\ns3s is execution S3 Select from json to json (default).\n\n```console\n$ s3s s3://bucket/prefix\n{\"time\":1654848930,\"type\":\"speak\"}\n{\"time\":1654848969,\"type\":\"sleep\"}\n\n// $ s3s s3://bucket/prefix_A s3://bucket/prefix_B s3://bucket/prefix_C\n```\n\n```console\n$ s3s -q 'SELECT * FROM S3Object s WHERE s.type = \"speak\"' s3://bucket/prefix\n{\"time\":1654848930,\"type\":\"speak\"}\n\n// alternate\n// $ s3s -w 's.type = \"speak\"' s3://bucket/prefix\n```\n\ns3s can execute S3 Select from csv to json when `--csv` option enabled.\n\n```console\n// 122, hello\n$ s3s s3://bucket/prefix\n{\"_1\":122,\"_2\":\"hello\"}\n```\n\n### ALB and CF logs support\n\n`--alb-logs` is a format for Application Load Balancer (ALB).\n`--cf-logs` is a format for CloudFront (CF).\n\nEach options are tagging available instead of `_1`, `_2`, etc.\n\n- [Application Load Balancer Format](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html)\n- [CloudFront Format](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html)\n\nAnd also, `--where` replace column names to column numbers.\nBut `--query` does not replace columns for execution raw query.\n\n```console\n// below query is same as $ s3s --alb-logs --query=\"'SELECT * FROM S3Object s WHERE s.`_2` = '2022-09-01T00:00:00.000000Z'\" s3://prefix\n$ s3s --alb-logs --where=\"s.`time` = '2022-09-01T00:00:00.000000Z'\" s3://prefix\n```\n\n|index|ALB|CF|\n|-|-|-|\n|_1|type|date|\n|_2|time|time|\n|_3|elb|x-edge-location|\n|_4|client:port|sc-bytes|\n|_5|target:port|c-ip|\n|_6|request_processing_time|cs-method|\n|_7|target_processing_time|cs(Host)|\n|_8|response_processing_time|cs-uri-stem|\n|_9|elb_status_code|sc-status|\n|_10|target_status_code|cs(Referer)|\n|_11|received_bytes|cs(User-Agent)|\n|_12|sent_bytes|cs-uri-query|\n|_13|request|cs(Cookie)|\n|_14|user_agent|x-edge-result-type|\n|_15|ssl_cipher|x-edge-request-id|\n|_16|ssl_protocol|x-host-header|\n|_17|target_group_arn|cs-protocol|\n|_18|trace_id|cs-bytes|\n|_19|domain_name|time-taken|\n|_20|chosen_cert_arn|x-forwarded-for|\n|_21|matched_rule_priority|ssl-protocol|\n|_22|request_creation_time|ssl-cipher|\n|_23|actions_executed|x-edge-response-result-type|\n|_24|redirect_url|cs-protocol-version|\n|_25|error_reason|fle-status|\n|_26|target:port_list|fle-encrypted-fields|\n|_27|target_status_code_list|c-port|\n|_28|classification|time-to-first-byte|\n|_29|classification_reason|x-edge-detailed-result-type|\n|_30||sc-content-type|\n|_31||sc-range-start|\n|_32||sc-range-end|\n\nSupport log range when alb and cf.\ntime format is `2006-01-02 15:04:05` as UTC.\n\n- `--duration` is a duration from now.\n- `--since` is start time\n- `--until` is end time\n\nHowever, s3s stop when you target cloudfront and using `--duration` or `--since` only, because s3s hit too many keys.\n\n### `-delve`, like directory move before querying\n\nsearch from prefix\n\n```console\n$ s3s -delve s3://bucket/prefix\n```\n\nsearch from bucket list\n\n```console\n$ s3s -delve\n```\n\n```\n  bucket/prefix/C/\n  bucket/prefix/B/\n  bucket/prefix/A/        # delve more lower path than this prefix\n  Query↵ (s3://bucket/prefix/) # choose and execute s3select this prefix\n\u003e ←Back upper path        # back to parent prefix\n5/5\n\u003e\n```\n\nQuerying after Enter.\n\n```\n{\"time\":1654848930,\"type\":\"speak\"}\n{\"time\":1654848969,\"type\":\"sleep\"}\n\n...\n\nbucket/prefix/A/ (print path to stderr at end)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoluku%2Fs3s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoluku%2Fs3s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoluku%2Fs3s/lists"}