{"id":18975027,"url":"https://github.com/bufferapp/restream","last_synced_at":"2025-07-01T04:34:40.431Z","repository":{"id":146340171,"uuid":"94221581","full_name":"bufferapp/restream","owner":"bufferapp","description":"Replay Firehose Streams in Kinesis Streams! ","archived":false,"fork":false,"pushed_at":"2020-08-10T10:07:56.000Z","size":7,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-19T19:10:16.489Z","etag":null,"topics":["kinesis-stream","restream"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bufferapp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-06-13T14:24:31.000Z","updated_at":"2025-01-03T16:02:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"323d3717-1824-424c-9bf7-85450b5f1fdc","html_url":"https://github.com/bufferapp/restream","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bufferapp/restream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufferapp%2Frestream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufferapp%2Frestream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufferapp%2Frestream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufferapp%2Frestream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bufferapp","download_url":"https://codeload.github.com/bufferapp/restream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufferapp%2Frestream/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262898646,"owners_count":23381608,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kinesis-stream","restream"],"created_at":"2024-11-08T15:17:05.398Z","updated_at":"2025-07-01T04:34:40.411Z","avatar_url":"https://github.com/bufferapp.png","language":"Python","readme":"# Restream\n\nReplay Firehose Streams in Kinesis Streams!\n\n## Installation\n\nYou can use `pip` to install Restream.\n\n```bash\npip install git+https://github.com/bufferapp/restream\n```\n\nIf you prefer, you can clone it and run the setup.py file. Use the following commands to install Restream from Github:\n\n```bash\ngit clone https://github.com/bufferapp/restream\ncd restream\npython setup.py install\n```\n\n## Requirements\n\nThe script assumes you have data in S3 that follow the Firehose pattern (`bucket/key/\u003cyear\u003e/\u003cmonth\u003e/\u003cday\u003e/\u003chour\u003e`). Before running the script make sure you have a Kinesis Stream ready to consume a lot of data. Change the number of shards to fit the script throughput before restreaming large datasets!\n\n## Usage\n\nThe script requires the name of the bucket and the path (`key`) of the root Firehose folder as well as a delimited timeframe.\n\n```bash\nrestream [bucket] [key] [start-date] [end-date]\n```\n\nFor example, if the Firehose stream saves the records into `s3://my-bucket/my-data/` and we want to restream the data from `2017-06-12 20:00:00` to `2017-06-12 22:00:00` we can run the following:\n\n```bash\nrestream my-bucket my-data my-stream-name -sd '2017-06-12 20' -ed '2017 Jun 12 22:00' -y\n```\n\nDate input is flexible and has a resolution up to the hour.\n\n## Contributing\n\nWe appreciate all kind of contributions! Please open an issue to start a discussion around a feature idea or a bug. :smile:\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbufferapp%2Frestream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbufferapp%2Frestream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbufferapp%2Frestream/lists"}