{"id":26285635,"url":"https://github.com/grumlimited/kinesis-tailr","last_synced_at":"2025-05-07T14:41:31.059Z","repository":{"id":161532716,"uuid":"636226553","full_name":"grumlimited/kinesis-tailr","owner":"grumlimited","description":"kinesis-tailr: a simple tool to tail a Kinesis stream built with Rust.","archived":false,"fork":false,"pushed_at":"2025-02-04T18:06:25.000Z","size":241,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-31T11:02:19.371Z","etag":null,"topics":["aws","kinesis","kinesis-c","kinesis-cli","rust","tail","tailf"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grumlimited.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-04T11:36:20.000Z","updated_at":"2025-02-04T18:05:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"31e6e12c-4ad7-449e-ad35-a26327274f61","html_url":"https://github.com/grumlimited/kinesis-tailr","commit_stats":null,"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grumlimited%2Fkinesis-tailr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grumlimited%2Fkinesis-tailr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grumlimited%2Fkinesis-tailr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grumlimited%2Fkinesis-tailr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grumlimited","download_url":"https://codeload.github.com/grumlimited/kinesis-tailr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252897367,"owners_count":21821430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","kinesis","kinesis-c","kinesis-cli","rust","tail","tailf"],"created_at":"2025-03-14T19:35:05.000Z","updated_at":"2025-05-07T14:41:31.036Z","avatar_url":"https://github.com/grumlimited.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kinesis-tailr\n\n![Continuous integration](https://github.com/grumlimited/kinesis-tailr/workflows/CI/badge.svg?branch=main)\n\nA simple tool to tail a Kinesis stream. Built with Rust.\n\n## Installation\n\n### Requirements\n\n* `rustc`\n* `make`\n\n### From source\n\n```bash\ncargo make install-local\n```\n\nInstalls a single binary to `/usr/local/bin/kinesis-tailr`. Alternatively, use\n\n```bash\ncargo install --git https://github.com/grumlimited/kinesis-tailr\n```\n\n### Releases\n\nThe [release page](https://github.com/grumlimited/kinesis-tailr/releases) provides packages for Debian and CentOS and\nArch Linux.\n\n## Usage\n\n    ❯ kinesis-tailr -help\n\n    Usage: kinesis-tailr [OPTIONS] --stream-name \u003cSTREAM_NAME\u003e\n    \n    Options:\n        -r, --region \u003cREGION\u003e                AWS Region\n        -s, --stream-name \u003cSTREAM_NAME\u003e      Name of the stream\n        --endpoint-url \u003cENDPOINT_URL\u003e        Endpoint URL to use\n        --from-datetime \u003cFROM_DATETIME\u003e      Start datetime position to tail from. ISO 8601 format\n        --to-datetime \u003cTO_DATETIME\u003e          End datetime position to tail up to. ISO 8601 format\n        --max-messages \u003cMAX_MESSAGES\u003e        Maximum number of messages to retrieve\n        --timeout \u003cTIMEOUT\u003e                  Exit if no messages received after \u003ctimeout\u003e seconds\n        --max-attempts \u003cMAX_ATTEMPTS\u003e        Maximum number of aws sdk retries. Increase if you are seeing throttling errors [default: 10]\n        --no-color                           Disable color output\n        --print-delimiter                    Print a delimiter between each payload\n        --print-key                          Print the partition key\n        --print-sequence-number              Print the sequence number\n        --print-shard-id                     Print the shard ID\n        --print-timestamp                    Print timestamps\n        --progress                           Print progress status\n        --shard-id \u003cSHARD_ID\u003e                Shard ID to tail from. Repeat option for each shard ID to filter on\n        -o, --output-file \u003cOUTPUT_FILE\u003e      Output file to write to\n        -c, --concurrent \u003cCONCURRENT\u003e        Concurrent number of shards to tail\n        -v, --verbose                        Display additional information\n            --base64                         Base64 encode payloads (eg. for binary data)\n            --utf8                           Forces UTF-8 printable payloads\n        -h, --help                           Print help\n        -V, --version                        Print version\n\n### Example\n\n```bash\nkinesis-tailr \\\n    --region eu-west-1 \\\n    --stream-name=ddb-stream-dev \\\n    --print-timestamp \\\n    --from-datetime '2023-05-04T20:57:12+00:00' \\\n    --max-messages 2\n```\n\n### UTF-8\n\n`kinesis-tailr` expects payloads to be UTF-8 encoded. If a payload is not UTF-8 encoded, it will be base64 encoded and\nprinted as such.\n\nIt might be useful to print the raw payload instead though. This can be achieved with the `--no-base64` flag.\n\nProperly UTF-8 encoded payloads will be printed as such and never base64 encoded.\n\n### Logging\n\nGeneral logging level for debugging can be turned on with:\n\n```bash\nexport RUST_LOG=\"INFO\"\n\nkinesis-tailr --stream-name mystream\n```\n\n    [2023-05-10T21:45:14Z INFO  aws_config::meta::region] load_region; provider=None\n    [2023-05-10T21:45:14Z INFO  aws_config::meta::region] load_region; provider=EnvironmentVariableRegionProvider { env: Env(Real) }\n    [2023-05-10T21:45:14Z INFO  tracing::span] lazy_load_credentials;\n    [2023-05-10T21:45:14Z INFO  aws_credential_types::cache::lazy_caching] credentials cache miss occurred; added new AWS credentials (took 24.934µs)\n    [...]\n\nSpecific logging for `kinesis-tailr` can be turned on with:\n\n```bash\nexport RUST_LOG=\"WARN,kinesis_tailr=INFO\"\n\nkinesis-tailr --stream-name mystream --from-datetime '2023-05-17T19:00:00Z' -o output.json\n```\n\n    [2023-05-17T20:37:35Z INFO  kinesis_tailr::kinesis::ticker] shardId-000000001119: 00:31:23\n    [2023-05-17T20:37:35Z INFO  kinesis_tailr::kinesis::ticker] shardId-000000001144: 00:31:27\n    [2023-05-17T20:37:35Z INFO  kinesis_tailr::kinesis::ticker] shardId-000000001085: 00:31:31\n    [2023-05-17T20:37:35Z INFO  kinesis_tailr::kinesis::ticker] shardId-000000001118: 00:32:33\n    [2023-05-17T20:37:35Z INFO  kinesis_tailr::kinesis::ticker] shardId-000000001156: 00:40:21\n    [2023-05-17T20:37:35Z INFO  kinesis_tailr::kinesis::ticker] shardId-000000001122: 00:41:46\n    [2023-05-17T20:37:35Z INFO  kinesis_tailr::kinesis::ticker] 10 shards behind\n    [...]\n\nIt is recommended to use `-o output.json` to write the output to a file, as the output can be quite verbose. This can\nthen be inspected with `jq` or similar.\n\nMoreover, it also frees the console output for informational messages. Use\n\n```bash\nexport RUST_LOG=\"WARN,kinesis_tailr=DEBUG\"\n```\n\nfor more debugging information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrumlimited%2Fkinesis-tailr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrumlimited%2Fkinesis-tailr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrumlimited%2Fkinesis-tailr/lists"}