https://github.com/grumlimited/kinesis-tailr
kinesis-tailr: a simple tool to tail a Kinesis stream built with Rust.
https://github.com/grumlimited/kinesis-tailr
aws kinesis kinesis-c kinesis-cli rust tail tailf
Last synced: 5 months ago
JSON representation
kinesis-tailr: a simple tool to tail a Kinesis stream built with Rust.
- Host: GitHub
- URL: https://github.com/grumlimited/kinesis-tailr
- Owner: grumlimited
- Created: 2023-05-04T11:36:20.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-02-04T18:06:25.000Z (9 months ago)
- Last Synced: 2025-03-31T11:02:19.371Z (7 months ago)
- Topics: aws, kinesis, kinesis-c, kinesis-cli, rust, tail, tailf
- Language: Rust
- Homepage:
- Size: 235 KB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# kinesis-tailr
A simple tool to tail a Kinesis stream. Built with Rust.
## Installation
### Requirements
* `rustc`
* `make`
### From source
```bash
cargo make install-local
```
Installs a single binary to `/usr/local/bin/kinesis-tailr`. Alternatively, use
```bash
cargo install --git https://github.com/grumlimited/kinesis-tailr
```
### Releases
The [release page](https://github.com/grumlimited/kinesis-tailr/releases) provides packages for Debian and CentOS and
Arch Linux.
## Usage
❯ kinesis-tailr -help
Usage: kinesis-tailr [OPTIONS] --stream-name
Options:
-r, --region AWS Region
-s, --stream-name Name of the stream
--endpoint-url Endpoint URL to use
--from-datetime Start datetime position to tail from. ISO 8601 format
--to-datetime End datetime position to tail up to. ISO 8601 format
--max-messages Maximum number of messages to retrieve
--timeout Exit if no messages received after seconds
--max-attempts Maximum number of aws sdk retries. Increase if you are seeing throttling errors [default: 10]
--no-color Disable color output
--print-delimiter Print a delimiter between each payload
--print-key Print the partition key
--print-sequence-number Print the sequence number
--print-shard-id Print the shard ID
--print-timestamp Print timestamps
--progress Print progress status
--shard-id Shard ID to tail from. Repeat option for each shard ID to filter on
-o, --output-file Output file to write to
-c, --concurrent Concurrent number of shards to tail
-v, --verbose Display additional information
--base64 Base64 encode payloads (eg. for binary data)
--utf8 Forces UTF-8 printable payloads
-h, --help Print help
-V, --version Print version
### Example
```bash
kinesis-tailr \
--region eu-west-1 \
--stream-name=ddb-stream-dev \
--print-timestamp \
--from-datetime '2023-05-04T20:57:12+00:00' \
--max-messages 2
```
### UTF-8
`kinesis-tailr` expects payloads to be UTF-8 encoded. If a payload is not UTF-8 encoded, it will be base64 encoded and
printed as such.
It might be useful to print the raw payload instead though. This can be achieved with the `--no-base64` flag.
Properly UTF-8 encoded payloads will be printed as such and never base64 encoded.
### Logging
General logging level for debugging can be turned on with:
```bash
export RUST_LOG="INFO"
kinesis-tailr --stream-name mystream
```
[2023-05-10T21:45:14Z INFO aws_config::meta::region] load_region; provider=None
[2023-05-10T21:45:14Z INFO aws_config::meta::region] load_region; provider=EnvironmentVariableRegionProvider { env: Env(Real) }
[2023-05-10T21:45:14Z INFO tracing::span] lazy_load_credentials;
[2023-05-10T21:45:14Z INFO aws_credential_types::cache::lazy_caching] credentials cache miss occurred; added new AWS credentials (took 24.934µs)
[...]
Specific logging for `kinesis-tailr` can be turned on with:
```bash
export RUST_LOG="WARN,kinesis_tailr=INFO"
kinesis-tailr --stream-name mystream --from-datetime '2023-05-17T19:00:00Z' -o output.json
```
[2023-05-17T20:37:35Z INFO kinesis_tailr::kinesis::ticker] shardId-000000001119: 00:31:23
[2023-05-17T20:37:35Z INFO kinesis_tailr::kinesis::ticker] shardId-000000001144: 00:31:27
[2023-05-17T20:37:35Z INFO kinesis_tailr::kinesis::ticker] shardId-000000001085: 00:31:31
[2023-05-17T20:37:35Z INFO kinesis_tailr::kinesis::ticker] shardId-000000001118: 00:32:33
[2023-05-17T20:37:35Z INFO kinesis_tailr::kinesis::ticker] shardId-000000001156: 00:40:21
[2023-05-17T20:37:35Z INFO kinesis_tailr::kinesis::ticker] shardId-000000001122: 00:41:46
[2023-05-17T20:37:35Z INFO kinesis_tailr::kinesis::ticker] 10 shards behind
[...]
It is recommended to use `-o output.json` to write the output to a file, as the output can be quite verbose. This can
then be inspected with `jq` or similar.
Moreover, it also frees the console output for informational messages. Use
```bash
export RUST_LOG="WARN,kinesis_tailr=DEBUG"
```
for more debugging information.