{"id":23545529,"url":"https://github.com/bh2smith/dune-aws","last_synced_at":"2025-09-01T12:11:00.435Z","repository":{"id":182854118,"uuid":"656105021","full_name":"bh2smith/dune-aws","owner":"bh2smith","description":"Python Package for Uploading Data to Dune Community Sources","archived":false,"fork":false,"pushed_at":"2023-06-22T10:37:40.000Z","size":17,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-26T14:46:46.998Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bh2smith.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-06-20T09:12:23.000Z","updated_at":"2023-06-20T09:19:44.000Z","dependencies_parsed_at":"2023-07-21T21:52:53.685Z","dependency_job_id":null,"html_url":"https://github.com/bh2smith/dune-aws","commit_stats":null,"previous_names":["bh2smith/dune-aws"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/bh2smith/dune-aws","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bh2smith","download_url":"https://codeload.github.com/bh2smith/dune-aws/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-aws/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273122129,"owners_count":25049539,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-26T08:18:44.937Z","updated_at":"2025-09-01T12:11:00.403Z","avatar_url":"https://github.com/bh2smith.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dune-aws\n\nBasic AWS Components for syncing off-chain data with Dune Community Sources\n\n## Installation \u0026 Usage\n\n\n```sh\npip install dune-aws\n```\n\n\n### Usage Option 1; Direct via AWSClient\n\nThis option provides the end user with many facets of functionality (delete, upload, download, view files). \nOne can mix and match the required components for their use case (utilizing also `aws.last_sync_block`). \nFor a very simple example utilizing this, see Option 2.\n\n```py\nimport os\nfrom dune_aws.aws import AWSClient\n\naws_client = AWSClient(\n    internal_role=os.environ[\"AWS_INTERNAL_ROLE\"],\n    # Info below is provided by Dune team.\n    external_role=os.environ[\"AWS_EXTERNAL_ROLE\"],\n    external_id=os.environ[\"AWS_EXTERNAL_ID\"],  \n    bucket=os.environ[\"AWS_BUCKET\"],\n)\n\ndata_set = [{\"x\": 1, \"y\": 2}, {\"z\": 3}]\n\naws_client.put_object(\n    data_set,\n    object_key=\"table_name/must_contain_dot_json_then_number.json\",\n)\n```\n\n\n### Usage Option 2; Simpler via RecordHandler\n\n```py\nfrom dune_aws.record_handler import RecordHandler\n\nhandler = RecordHandler(\n    file=\"table_name/moo_123\",\n    data_set=[{\"x\": 1, \"y\": 2}, {\"z\": 3}]  # should be list[dict[str, Any]],\n    # If AWSClient is not supplied to the handler, it will be loaded from environment.\n)\n\nhandler.upload_content(delete_first=True)\n```\n\nNote that, one must first coordinate with Dune to \n1. gain access to the AWS bucket (i.e. external credentials) and \n2. define the table schema of your dataset.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbh2smith%2Fdune-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbh2smith%2Fdune-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbh2smith%2Fdune-aws/lists"}