{"id":23545514,"url":"https://github.com/bh2smith/dune-sync","last_synced_at":"2026-01-22T13:48:25.491Z","repository":{"id":258769573,"uuid":"868484195","full_name":"bh2smith/dune-sync","owner":"bh2smith","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-04T20:08:45.000Z","size":320,"stargazers_count":3,"open_issues_count":13,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-24T03:13:41.063Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bh2smith.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-06T13:59:17.000Z","updated_at":"2025-04-04T20:08:48.000Z","dependencies_parsed_at":"2025-04-24T03:13:52.466Z","dependency_job_id":"6ba0cdce-710a-40e8-aa9e-087ba7a1109f","html_url":"https://github.com/bh2smith/dune-sync","commit_stats":null,"previous_names":["bh2smith/dune-sync"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/bh2smith/dune-sync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bh2smith","download_url":"https://codeload.github.com/bh2smith/dune-sync/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fdune-sync/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28664004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T01:17:37.254Z","status":"online","status_checked_at":"2026-01-22T02:00:07.137Z","response_time":144,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-26T08:18:40.317Z","updated_at":"2026-01-22T13:48:25.475Z","avatar_url":"https://github.com/bh2smith.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dune Sync V2\n\nA general-purpose bidirectional data synchronization tool that enables seamless data transfer between [Dune Analytics](https://dune.com) and PostgreSQL databases. This project was developed as part of the [CoW DAO Grants Program](https://forum.cow.fi/t/grant-application-dune-sync-v2/2597).\n\n## Features\n\n- **Dune to PostgreSQL**: Archive data from Dune queries into your local database\n- **PostgreSQL to Dune**: Upload local data to Dune tables via the CSV upload endpoint\n- **Configuration-based**: Simple YAML configuration for defining sources, destinations, and jobs\n- **Docker-ready**: Easy deployment using pre-built container images\n\n## Usage\n\n### Create a configuration file\n\nConfiguration is provided in a single YAML file. Refer to the existing `config.yaml` for an overview.\nThe configuration file consists of three main sections:\n- `data_sources`: Defines available databases\n- `jobs`: Defines synchronization jobs that connect sources to destinations\n\nThe config file may contain environment variable placeholders in\n[envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html)-compatible format:\n- `$VAR_NAME`\n- `${VAR_NAME}`\n- `$varname`\n\n**Note**: Every variable referenced this way __must__ be defined at runtime,\notherwise the program exits with an error.\n\n#### Specifying configuration at runtime\n- By default, the program looks for a file called `config.yaml` next to `main.py`\n- You may pass a configuration file by using the `--config` parameter\n  - The argument to `--config` may be a filename, a file path, or a URL starting with `http://` or `https://`\n  - If a URL is passed, it's downloaded and its contents are assumed to be the configuration for the program\n- File or content served at the given URL must be valid YAML and encoded in UTF-8\n\n#### Data Source Definitions\n\nSources are defined as a list of configurations, each containing:\n- `name`: String. A unique identifier for referencing this source in jobs\n- `type`: String. Must be either `dune` or `postgres`\n- `key`: String. Connection details, supports environment variable templating using `${VAR_NAME}` syntax such as `${DB_URL}` or `${DUNE_API_KEY}` ([see environment setup](#define-environment))\n\n#### Job Parameters\n\nEach job in the `jobs` list should contain:\n- `name`: Optional String. A human-readable name for the job\n- `source`: Definition of which source to use and how to fetch data\n- `destination`: Definition of which destination to use and how to store data\n\n##### Source Configuration\n\nFor Dune sources (`ref: Dune1`):\n- `query_id`: Integer. ID of an existing Dune Query to execute\n- `query_engine`: Optional String. Either `medium` or `large`. Defaults to `medium`\n- `poll_frequency`: Optional Integer. Seconds between result polling. Defaults to `1`.\n- `parameters`: Optional list of Dune Query parameters\n    - `name`: String. Parameter name\n    - `type`: String. Must be one of: `TEXT`, `NUMBER`, `DATE`, `ENUM`\n    - `value`: Any. Value for the parameter\n\nFor Postgres sources (`ref: Postgres`):\n- `query_string`: String. SQL query or path to .sql file (relative to `main.py` or absolute)\n\n##### Destination Configuration\n\nFor Dune destinations (`ref: Dune`):\n- `table_name`: String. Name of Dune table to update\n- `request_timeout`: [optional] Request timeout for the dune client\n\nFor Postgres destinations (`ref: Postgres`):\n- `table_name`: String. Name of table to insert/append into\n- `if_exists`: String. One of `upsert`, `insert_ignore`, `replace` or `append`.\n  - `index_columns`: String[]. only relevant for `upsert` or `insert_ignore` a list of columns on which to detect conflict\n\n### Define environment\n\nCopy `.env.sample` to `.env` and fill out the two required variables\n\n- `DUNE_API_KEY` - Valid API key for [Dune](https://dune.com/)\n- `DB_URL` - Connection string for the source and/or destination PostgreSQL database,\n  in the form `postgresql://postgres:postgres@localhost:5432/postgres`\n- (Optional) `PROMETHEUS_PUSHGATEWAY_URL` - URL of a [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) which receives job-related metrics.\n\n### Mount the config and .env files into the container and run the script\n\nYou can download the image from GitHub Container Registry:\n\n```shell\ndocker pull ghcr.io/bh2smith/dune-sync:latest\n```\n\nOr build it yourself:\n\n```shell\nexport IMAGE_NAME=dune-sync # (or ghcr.io image above)\ndocker build -t ${IMAGE_NAME} .\n\n# Base docker command (using config.yaml mounted at /app/config.yaml)\ndocker run --rm \\\n    -v \"$(pwd)/config.yaml:/app/config.yaml\" \\\n    --env-file .env \\\n    ${IMAGE_NAME}\n\n# Optional additions:\n# - Mount custom config file (requires --config flag)\n    -v \"$(pwd)/my-config.yaml:/app/my-config.yaml\" \\\n# - Mount queries directory (if using SQL file paths)\n    -v \"$(pwd)/queries:/app/queries\" \\\n    --config /app/my-config.yaml\n# - Specify jobs to run (if not specified, all jobs will be run)\n    --jobs job1 job2\n```\n\nNote that postgres queries can also be file paths (they would also need to be mounted into the container).\n\n## Local Development\n\nFill out the empty fields in [Sample Env](.env.sample) (e.g. `DUNE_API_KEY` and `DB_URL`)\n\n```shell\ndocker-compose up -d # Starts postgres container (in the background)\npython -m src.main [--config config.yaml] [--jobs d2p-test-1 p2d-test]\n```\n\n### Development Commands\n\nTo get started with development:\n\n```shell\npython -m pip install poetry  # install poetry which is used to manage the project's dependencies\npoetry install  # Set up virtual environment\npoetry shell  # Activate virtual environment\n```\n\nThe project uses a Makefile to streamline development tasks. Here are the available commands:\n\n\n\n- `make fmt`: Formats code using black\n- `make lint`: Runs pylint for code quality checks\n- `make types`: Performs static type checking with mypy\n- `make check`: Runs formatting, linting, and type checking in sequence\n- `make test`: Runs pytest with coverage reporting (minimum 93% coverage required)\n- `make clean`: Removes Python cache files\n- `make run`: Executes the main application (although does not accespt run time parameters)\n\nTo pass run time arguments use, for example:\n\n```shell\npython -m src.main --jobs cow-solvers\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbh2smith%2Fdune-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbh2smith%2Fdune-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbh2smith%2Fdune-sync/lists"}