https://github.com/bh2smith/dune-aws
Python Package for Uploading Data to Dune Community Sources
https://github.com/bh2smith/dune-aws
Last synced: 11 months ago
JSON representation
Python Package for Uploading Data to Dune Community Sources
- Host: GitHub
- URL: https://github.com/bh2smith/dune-aws
- Owner: bh2smith
- Created: 2023-06-20T09:12:23.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-06-22T10:37:40.000Z (about 3 years ago)
- Last Synced: 2025-08-26T14:46:46.998Z (11 months ago)
- Language: Python
- Size: 16.6 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# dune-aws
Basic AWS Components for syncing off-chain data with Dune Community Sources
## Installation & Usage
```sh
pip install dune-aws
```
### Usage Option 1; Direct via AWSClient
This option provides the end user with many facets of functionality (delete, upload, download, view files).
One can mix and match the required components for their use case (utilizing also `aws.last_sync_block`).
For a very simple example utilizing this, see Option 2.
```py
import os
from dune_aws.aws import AWSClient
aws_client = AWSClient(
internal_role=os.environ["AWS_INTERNAL_ROLE"],
# Info below is provided by Dune team.
external_role=os.environ["AWS_EXTERNAL_ROLE"],
external_id=os.environ["AWS_EXTERNAL_ID"],
bucket=os.environ["AWS_BUCKET"],
)
data_set = [{"x": 1, "y": 2}, {"z": 3}]
aws_client.put_object(
data_set,
object_key="table_name/must_contain_dot_json_then_number.json",
)
```
### Usage Option 2; Simpler via RecordHandler
```py
from dune_aws.record_handler import RecordHandler
handler = RecordHandler(
file="table_name/moo_123",
data_set=[{"x": 1, "y": 2}, {"z": 3}] # should be list[dict[str, Any]],
# If AWSClient is not supplied to the handler, it will be loaded from environment.
)
handler.upload_content(delete_first=True)
```
Note that, one must first coordinate with Dune to
1. gain access to the AWS bucket (i.e. external credentials) and
2. define the table schema of your dataset.