https://github.com/chrisammon3000/aws-json-dataset
Lightweight and simple Python package to quickly send JSON data to various AWS services.
https://github.com/chrisammon3000/aws-json-dataset
aws-kinesis-firehose aws-kinesis-stream aws-sns aws-sqs data-engineering json streaming
Last synced: 3 months ago
JSON representation
Lightweight and simple Python package to quickly send JSON data to various AWS services.
- Host: GitHub
- URL: https://github.com/chrisammon3000/aws-json-dataset
- Owner: chrisammon3000
- License: mit
- Created: 2022-11-14T00:24:16.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-02-03T23:25:40.000Z (over 2 years ago)
- Last Synced: 2025-10-26T11:36:57.741Z (9 months ago)
- Topics: aws-kinesis-firehose, aws-kinesis-stream, aws-sns, aws-sqs, data-engineering, json, streaming
- Language: Python
- Homepage:
- Size: 123 KB
- Stars: 0
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# aws-json-dataset
[](https://github.com/chrisammon3000/aws-json-dataset/actions/workflows/run_tests.yml) [](https://codecov.io/github/chrisammon3000/aws-json-dataset?style=for-the-badge) [](https://opensource.org/licenses/MIT) [](https://www.python.org/downloads/release/python-3100/)
Lightweight and simple Python package to quickly send batches of JSON data to various AWS services.
## Description
The idea behind this library is to create an easy, quick way to send JSON data to AWS services.
- SQS
- SNS
- Kinesis Firehose
- Kinesis Data Streams (coming soon)
JSON is an extremely common format and each AWS service has it's own API with different requirements for how to send data.
This library includes functionality for:
- Automatically handling batch API calls to SNS, SQS and Kinesis Firehose
- Manages available services based on record size
- Base64 conversion for Kinesis streams
### Roadmap
- [ ] Support for Kinesis Data Streams
- [ ] Support for DynamoDB inserts, updates and deletes
- [ ] Support for S3, including gzip compression and JSON lines format
- [ ] Support for FIFO SQS queues
- [ ] Support for SNS topics
## Quickstart
Set up your [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) and environment variables and export them to the environment.
```bash
export AWS_PROFILE=
export AWS_REGION=
```
Install the library using pip.
```bash
pip install aws-json-dataset
```
Send JSON data to various AWS services.
```python
from awsjsondataset import AwsJsonDataset
# create a list of JSON objects
data = [ {"id": idx, "data": ""} for idx in range(100) ]
# Wrap using AwsJsonDataset
dataset = AwsJsonDataset(data=data)
# Send to SQS queue
dataset.sqs("").send_messages()
# Send to SNS topic
dataset.sns("").publish_messages()
# Send to Kinesis Firehose stream
dataset.firehose("").put_records()
```
## Local Development
Follow the steps to set up the deployment environment.
### Prerequisites
* AWS credentials
* Python 3.10
### Creating a Python Virtual Environment
When developing locally, create a Python virtual environment to manage dependencies:
```bash
python3.10 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install .[dev,test]
```
### Environment Variables
Create a `.env` file in the project root.
```bash
AWS_REGION=
```
***Important:*** *Always use a `.env` file or AWS SSM Parameter Store or Secrets Manager for sensitive variables like credentials and API keys. Never hard-code them, including when developing. AWS will quarantine an account if any credentials get accidentally exposed and this will cause problems.* → ***Make sure that `.env` is listed in `.gitignore`***
### AWS Credentials
Valid AWS credentials must be available to AWS CLI and SAM CLI. The easiest way to do this is running `aws configure`, or by adding them to `~/.aws/credentials` and exporting the `AWS_PROFILE` variable to the environment.
For more information visit the documentation page:
[Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
## Unit Tests
Follow the steps above to create a Python virtual environment. Run tests with the following command.
```bash
coverage run -m pytest
```
## Troubleshooting
* Check your AWS credentials in `~/.aws/credentials`
* Check that the environment variables are available to the services that need them
* Check that the correct environment or interpreter is being used for Python
## Authors
**Primary Contact:** Gregory Christopher Lindsey (@chrisammon3000)
## License
This library is licensed under the MIT-0 License. See the LICENSE file.