{"id":24269040,"url":"https://github.com/chrisammon3000/aws-json-dataset","last_synced_at":"2026-04-19T02:33:37.525Z","repository":{"id":170204080,"uuid":"565621193","full_name":"chrisammon3000/aws-json-dataset","owner":"chrisammon3000","description":"Lightweight and simple Python package to quickly send JSON data to various AWS services.","archived":false,"fork":false,"pushed_at":"2024-02-03T23:25:40.000Z","size":126,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-26T11:36:57.741Z","etag":null,"topics":["aws-kinesis-firehose","aws-kinesis-stream","aws-sns","aws-sqs","data-engineering","json","streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrisammon3000.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-14T00:24:16.000Z","updated_at":"2023-05-28T02:23:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"7482b8c5-bfc8-4dc4-a690-ad29a9079e60","html_url":"https://github.com/chrisammon3000/aws-json-dataset","commit_stats":null,"previous_names":["abk7777/aws-json-dataset","chrisammon3000/aws-json-dataset"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chrisammon3000/aws-json-dataset","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-json-dataset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-json-dataset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-json-dataset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-json-dataset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrisammon3000","download_url":"https://codeload.github.com/chrisammon3000/aws-json-dataset/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-json-dataset/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31992011,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-kinesis-firehose","aws-kinesis-stream","aws-sns","aws-sqs","data-engineering","json","streaming"],"created_at":"2025-01-15T14:39:38.333Z","updated_at":"2026-04-19T02:33:37.506Z","avatar_url":"https://github.com/chrisammon3000.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aws-json-dataset\n[![Build](https://github.com/chrisammon3000/aws-json-dataset/actions/workflows/run_tests.yml/badge.svg?style=for-the-badge)](https://github.com/chrisammon3000/aws-json-dataset/actions/workflows/run_tests.yml) [![codecov](https://codecov.io/github/chrisammon3000/aws-json-dataset/branch/main/graph/badge.svg?token=QSZLP51RWJ)](https://codecov.io/github/chrisammon3000/aws-json-dataset?style=for-the-badge) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/release/python-3100/)\n\n\u003c!-- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](https://opensource.org/licenses/MIT) [![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg?style=for-the-badge)](https://www.python.org/downloads/release/python-3100/) --\u003e\n\nLightweight and simple Python package to quickly send batches of JSON data to various AWS services.\n\n## Description\nThe idea behind this library is to create an easy, quick way to send JSON data to AWS services.\n- SQS\n- SNS\n- Kinesis Firehose\n- Kinesis Data Streams (coming soon)\n\nJSON is an extremely common format and each AWS service has it's own API with different requirements for how to send data. \n\nThis library includes functionality for:\n- Automatically handling batch API calls to SNS, SQS and Kinesis Firehose\n- Manages available services based on record size\n- Base64 conversion for Kinesis streams\n\n### Roadmap\n- [ ] Support for Kinesis Data Streams\n- [ ] Support for DynamoDB inserts, updates and deletes\n- [ ] Support for S3, including gzip compression and JSON lines format\n- [ ] Support for FIFO SQS queues\n- [ ] Support for SNS topics\n\n## Quickstart\nSet up your [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) and environment variables and export them to the environment.\n```bash\nexport AWS_PROFILE=\u003cprofile\u003e\nexport AWS_REGION=\u003cregion\u003e\n```\n\n\nInstall the library using pip.\n```bash\npip install aws-json-dataset\n```\n\nSend JSON data to various AWS services.\n```python\nfrom awsjsondataset import AwsJsonDataset\n\n# create a list of JSON objects\ndata = [ {\"id\": idx, \"data\": \"\u003cdata\u003e\"} for idx in range(100) ]\n\n# Wrap using AwsJsonDataset\ndataset = AwsJsonDataset(data=data)\n\n# Send to SQS queue\ndataset.sqs(\"\u003csqs_queue_url\u003e\").send_messages()\n\n# Send to SNS topic\ndataset.sns(\"\u003csns_topic_arn\u003e\").publish_messages()\n\n# Send to Kinesis Firehose stream\ndataset.firehose(\"\u003cdelivery_stream_name\u003e\").put_records()\n```\n\n## Local Development\nFollow the steps to set up the deployment environment.\n\n### Prerequisites\n* AWS credentials\n* Python 3.10\n\n### Creating a Python Virtual Environment\nWhen developing locally, create a Python virtual environment to manage dependencies:\n```bash\npython3.10 -m venv .venv\nsource .venv/bin/activate\npip install -U pip\npip install .[dev,test]\n```\n\n### Environment Variables\nCreate a `.env` file in the project root.\n```bash\nAWS_REGION=\u003cregion\u003e\n```\n\n***Important:*** *Always use a `.env` file or AWS SSM Parameter Store or Secrets Manager for sensitive variables like credentials and API keys. Never hard-code them, including when developing. AWS will quarantine an account if any credentials get accidentally exposed and this will cause problems.* \u0026rarr; ***Make sure that `.env` is listed in `.gitignore`***\n\n### AWS Credentials\nValid AWS credentials must be available to AWS CLI and SAM CLI. The easiest way to do this is running `aws configure`, or by adding them to `~/.aws/credentials` and exporting the `AWS_PROFILE` variable to the environment.\n\nFor more information visit the documentation page:\n[Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)\n\n## Unit Tests\nFollow the steps above to create a Python virtual environment. Run tests with the following command.\n```bash\ncoverage run -m pytest\n```\n\n## Troubleshooting\n* Check your AWS credentials in `~/.aws/credentials`\n* Check that the environment variables are available to the services that need them\n* Check that the correct environment or interpreter is being used for Python\n\n\u003c!-- ## References \u0026 Links --\u003e\n\n## Authors\n**Primary Contact:** Gregory Christopher Lindsey (@chrisammon3000)\n\n## License\nThis library is licensed under the MIT-0 License. See the LICENSE file.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisammon3000%2Faws-json-dataset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrisammon3000%2Faws-json-dataset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisammon3000%2Faws-json-dataset/lists"}