{"id":19330725,"url":"https://github.com/versent/kinesis-tail","last_synced_at":"2025-04-22T23:31:35.777Z","repository":{"id":50117450,"uuid":"108729270","full_name":"Versent/kinesis-tail","owner":"Versent","description":"Tool which provides tail for Kinesis","archived":false,"fork":false,"pushed_at":"2023-10-11T21:04:43.000Z","size":86,"stargazers_count":20,"open_issues_count":7,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-02T04:47:22.782Z","etag":null,"topics":["aws","golang","kinesis-stream"],"latest_commit_sha":null,"homepage":"https://github.com/Versent/kinesis-tail","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Versent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-29T11:42:44.000Z","updated_at":"2024-01-04T00:39:51.000Z","dependencies_parsed_at":"2024-06-19T09:55:33.132Z","dependency_job_id":"932cddcf-06cd-4a34-acce-6f379a514a58","html_url":"https://github.com/Versent/kinesis-tail","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Versent%2Fkinesis-tail","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Versent%2Fkinesis-tail/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Versent%2Fkinesis-tail/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Versent%2Fkinesis-tail/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Versent","download_url":"https://codeload.github.com/Versent/kinesis-tail/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250340235,"owners_count":21414503,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","golang","kinesis-stream"],"created_at":"2024-11-10T02:37:43.834Z","updated_at":"2025-04-22T23:31:35.514Z","avatar_url":"https://github.com/Versent.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kinesis-tai\n\nTool which provides tail for [Kinesis](https://aws.amazon.com/kinesis/streams/), it allows you to use one of two processors for the data returned, firstly one which decompresses and parses [CloudWatch Logs](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) data, and secondly one which just returns the raw data.\n\n# background\n\nThis cloudwatch logs reader works with a pattern used at Versent for log distribution and storage.\n\nFor more information on the setup for `cwlogs` sub command to function it assumes the logs are gzipped batches of log JSON records in Kinesis see [Real-time Processing of Log Data with Subscriptions](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CreateDestination.html)\n\n# installation\n\nYou can download `kinesis-tail` from [Releases](https://github.com/Versent/kinesis-tail/releases) or install it using npm.\n\n# usage\n\n```\nusage: kinesis-tail [\u003cflags\u003e] \u003ccommand\u003e [\u003cargs\u003e ...]\n\nFlags:\n      --help           Show context-sensitive help (also try --help-long and --help-man).\n  -t, --trace          Enable trace mode.\n  -r, --region=REGION  Configure the aws region.\n      --version        Show application version.\n\nCommands:\n  help [\u003ccommand\u003e...]\n    Show help.\n\n\n  cwlogs [\u003cflags\u003e] \u003cstream\u003e\n    Process cloudwatch logs data from kinesis.\n\n    --include=INCLUDE ...  Include anything in log group names which match the supplied string.\n    --exclude=EXCLUDE ...  Exclude anything in log group names which match the supplied string.\n\n  raw [\u003cflags\u003e] \u003cstream\u003e\n    Process raw data from kinesis.\n\n    --timeout=3600000  How long to capture raw data for before exiting in ms.\n    --count=0          How many records to capture raw data for before exiting.\n\n\n```\n\n# example\n\nList the kinesis streams in your account.\n\n```\naws kinesis list-streams\n```\n\nTo tail one of these streams and exit once you have captured 20 records.\n\n```\nkinesis-tail raw dev-1-stream --count 20\n```\n\nTo tail one of these streams and exit after 30 seconds, and write the data to a file.\n\n```\nkinesis-tail raw dev-1-stream --timeout 30000 | tee data.log\n```\n\n# license\n\nThis code is released under MIT License.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fversent%2Fkinesis-tail","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fversent%2Fkinesis-tail","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fversent%2Fkinesis-tail/lists"}