{"id":20589979,"url":"https://github.com/doist/s3logger","last_synced_at":"2026-07-12T12:32:48.874Z","repository":{"id":29700393,"uuid":"120961078","full_name":"Doist/s3logger","owner":"Doist","description":null,"archived":false,"fork":false,"pushed_at":"2023-09-19T11:45:20.000Z","size":34,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-16T23:43:31.011Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://godoc.org/github.com/Doist/s3logger","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Doist.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-09T22:10:11.000Z","updated_at":"2022-01-15T15:17:37.000Z","dependencies_parsed_at":"2024-06-20T02:51:43.228Z","dependency_job_id":"96fec7ec-5af6-4295-90e6-f7f8749470ac","html_url":"https://github.com/Doist/s3logger","commit_stats":null,"previous_names":["artyom/s3logger"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Doist%2Fs3logger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Doist%2Fs3logger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Doist%2Fs3logger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Doist%2Fs3logger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Doist","download_url":"https://codeload.github.com/Doist/s3logger/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242218865,"owners_count":20091510,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T07:33:16.739Z","updated_at":"2025-10-24T10:49:38.710Z","avatar_url":"https://github.com/Doist.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"Program s3logger collects json messages over TCP, stores them into\ngzip-compressed files split by time and uploads these files to AWS s3\nbucket.\n\nOnce started, service accepts TCP connections and expects clients to send\nstreams of json objects over such connections. s3logger only closes\nconnection if it encounters malformed json or single object size exceeds 4\nMiB limit. s3logger only reads data from the client.\n\ns3logger accumulates received messages over predefined time window (-t flag)\nto a temporary log file creating new ones as needed; previous files are\nuploaded to s3 bucket in background and removed after successful upload.\nOptionally maximum size of input read can be specified in megabytes (-mb\nflag) to rotate file before reaching predefined time. Program only writes to\na single temporary log file at a time, so json messages received from\nmultiple concurrent connections are interleaved into a single json stream.\nIt does its best not to lose messages, but can still drop them if they're\ncoming faster than could be saved on disk or there's any disk write error.\nStored messages are separated by new line (0xa).\n\ns3logger uploads files to a specified bucket using predefined s3 object\nnaming scheme:\n\n    dt=2018-02-09/20180209T213803_df718a7818e53243.json.gz\n\nIt uses dt=YYYY-MM-DD \"directories\", object name base starting with date and\ntime when log file was created (UTC) followed by hex-encoded 64-bit random\nvalue and .json.gz suffix.\n\nSee\nhttps://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables\non how to configure s3 bucket access credentials.\n\ns3logger does not use TLS for its listener at the moment as it is expected\nto run on localhost or inside trusted network.\n\n    Usage of s3logger:\n      -addr string\n    \taddress to listen (default \"localhost:8080\")\n      -bucket string\n    \ts3 bucket to upload logs\n      -dir string\n    \tdirectory to keep unsent files (default \"/var/spool/s3logger\")\n      -mb int\n    \tmegabytes of input read until file is rotated (0 to disable) (default 512)\n      -prefix string\n    \ts3 object name prefix (directory in a bucket)\n      -t duration\n    \ttime to use single file (min 1m) (default 5m0s)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoist%2Fs3logger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoist%2Fs3logger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoist%2Fs3logger/lists"}