{"id":26457116,"url":"https://github.com/dinsaw/kines","last_synced_at":"2025-03-18T22:42:18.191Z","repository":{"id":56007265,"uuid":"210355228","full_name":"dinsaw/kines","owner":"dinsaw","description":"Friendly CLI for Amazon Kinesis Data Streams","archived":false,"fork":false,"pushed_at":"2020-12-01T17:06:41.000Z","size":8184,"stargazers_count":57,"open_issues_count":4,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-15T04:19:31.164Z","etag":null,"topics":["amazon-kinesis","aws","aws-kinesis","aws-kinesis-stream","cli","command-line","kinesis","kinesis-stream"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dinsaw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-09-23T12:55:58.000Z","updated_at":"2024-11-27T09:24:25.000Z","dependencies_parsed_at":"2022-08-15T11:20:12.045Z","dependency_job_id":null,"html_url":"https://github.com/dinsaw/kines","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dinsaw%2Fkines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dinsaw%2Fkines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dinsaw%2Fkines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dinsaw%2Fkines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dinsaw","download_url":"https://codeload.github.com/dinsaw/kines/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244130291,"owners_count":20402756,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-kinesis","aws","aws-kinesis","aws-kinesis-stream","cli","command-line","kinesis","kinesis-stream"],"created_at":"2025-03-18T22:42:17.525Z","updated_at":"2025-03-18T22:42:18.179Z","avatar_url":"https://github.com/dinsaw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Kines\n[![PyPI version](https://img.shields.io/pypi/v/kines.svg)](https://badge.fury.io/py/kines) [![PyPI downloads](https://img.shields.io/pypi/dm/kines.svg)](https://pypistats.org/packages/kines) [![Build Status](https://travis-ci.org/dinsaw/kines.svg?branch=master)](https://travis-ci.org/dinsaw/kines) [![codecov](https://codecov.io/gh/dinsaw/kines/branch/master/graph/badge.svg)](https://codecov.io/gh/dinsaw/kines)\n\nFriendly Command Line Interface for [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/)\n\n![Kines Demo](https://raw.githubusercontent.com/dinsaw/kines/master/demo/kines-demo.gif)\n\n#### Install\n- `pip install kines`\n\n#### Setup \n- `aws configure`\n\n#### Commands \n##### List all Kinesis Stream\n- `kines ls`\n\nThis command lists streams with their open shard count, enhanced consumers count, retention period and encryption type. Internally this command calls `list-streams` and `describe_stream_summary` methods of [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kinesis.html#id34) Kinesis client.\n\n![Kines ls](https://raw.githubusercontent.com/dinsaw/kines/master/demo/kines-ls.png)\n\n##### List Kinesis Stream Shards\n- `kines lss \u003cstream-name\u003e`\n\nThis command will show you open as well as closed shards of a Kinesis stream.\n\n- For detailed output Run `kines lss \u003cstream-name\u003e -d`\n\n![Kines lss](https://raw.githubusercontent.com/dinsaw/kines/master/demo/kines-lss-and-d.png)\n\n##### Find shard for partition key\n- `kines find prod-clickstream -p 123455 -p 8900`\n\nFrom [Kinesis Docs](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html)\n\u003e A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to. Partition keys are Unicode strings with a maximum length limit of 256 bytes. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards. When an application puts data into a stream, it must specify a partition key.\n\nThis command comes handy when you want to determine shard for a partition key. This Command can accept multiple partition keys.\n\n![Kines find](https://raw.githubusercontent.com/dinsaw/kines/master/demo/kines-find.png)\n\n#### Walk through kinesis records\n- `kines walk \u003cstream-name\u003e \u003cshard-id\u003e -s \u003csequence-number\u003e -n \u003cnumber-of-records-per-call\u003e`\n\nYou can use this command to debug kinesis records. This command internally creates a [Shard Iterator](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html). If you don't specify `sequence-number` then a ShardIterator is created with type `TRIM_HORIZON`, which allows you to fetch from oldest Kinesis records in shard. When you specify `sequence-number` a ShardIterator is created with `AT_SEQUENCE_NUMBER` type, which fetches data from the specified sequence number.\nThe Kinesis record's data is decoded using `base64` decoder. You can press ⏎ to fetch more records or type `n` to abort.\n\n![Kines Walk](https://raw.githubusercontent.com/dinsaw/kines/master/demo/kines-walk-demo.gif)\n\n- Use `-t` option to get records from 1 hour 10 minutes ago. Example: `kines walk click-stream 000000000000 -t '1h10m'`\n\n- Use `-l` option to start from latest records. Example: `kines walk click-stream 000000000000 -l`\n\n- Use `-f` option to poll records repeatedly. Example: `kines walk click-stream 000000000000 -l -f`\n\n#### Get report for Kinesis Stream\n\nThe report command gives you statistics about your Kinesis Stream. Internally, this command calls [`get_metric_data`](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch.html#CloudWatch.Client.get_metric_data) of Boto3 Cloudwatch client.\n\nPlease take a look at legends command for shortforms used in this report.\n\n![Kines Report 60 hours 6 hours](https://raw.githubusercontent.com/dinsaw/kines/master/demo/kines-report-h-60-p-60.png)\n\n#### View all short forms and legends\n- `kines legends`\n\n![Kines Legends](https://raw.githubusercontent.com/dinsaw/kines/master/demo/kines-legends.png)\n\n#### How to build in Dev?\n- `pip install --editable .`\n\n#### How to publish to pypi?\n- `python3 setup.py sdist bdist_wheel`\n- `twine upload --skip-existing dist/*`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdinsaw%2Fkines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdinsaw%2Fkines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdinsaw%2Fkines/lists"}