https://github.com/cowprotocol/dune-sync
Components for syncing off-chain data with Dune Community Sources
https://github.com/cowprotocol/dune-sync
Last synced: about 1 year ago
JSON representation
Components for syncing off-chain data with Dune Community Sources
- Host: GitHub
- URL: https://github.com/cowprotocol/dune-sync
- Owner: cowprotocol
- Archived: true
- Created: 2022-11-21T13:21:42.000Z (over 3 years ago)
- Default Branch: dev
- Last Pushed: 2024-11-23T02:49:25.000Z (over 1 year ago)
- Last Synced: 2025-04-12T23:13:44.705Z (about 1 year ago)
- Language: Python
- Size: 546 KB
- Stars: 5
- Watchers: 3
- Forks: 1
- Open Issues: 18
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# dune-sync
Components for syncing off-chain data with Dune Community Sources
# Local Development
1. Clone Repo `git clone git@github.com:cowprotocol/dune-sync.git`
2. If using VS Code, open in devcontainer to ensure same setup as the final container.
3. Several Makefile Commands `make XXX`
Key make commands are; `install, check, test`
To execute the main binary (inside the container) run
```
python3 -m src.main --sync-table
```
Another way to install the repo is with pip:
``
pip install git+https://github.com/cowprotocol/dune-sync.git@
```
This allows you to install the repo as a module to your local environment and automatically install the required dependencies.
# Docker
### Build
```shell
docker build -t local_dune_sync .
```
You must provide valid environment variables as specified in [.env.sample](.env.sample)
### Run Local
```shell
docker run -v ${PWD}/data:/app/data --env-file .env local_dune_sync
```
### Run Remote
You will need to attach a volume and have an env file configuration. This example
- mounts `$PWD/data`
- and assumes `.env` file is in `$PWD`
```shell
docker run -v ${PWD}/data:/app/data --env-file .env ghcr.io/cowprotocol/dune-sync:latest
```
### Breaking Changes
Whenever the schema changes, we must coordinate with Dune that the data must be dropped and the table rebuilt.
For this we have provided a script `scripts/empty_bucket.py` which can be called to delete all data from their
buckets and our backup volume. This should only be run whilst in coordination with their team about the changes.
They will "stop the stream", drop the table on their side and restart the stream.
In the event that a hard reset is performed without the proper coordination,
it is likely that duplicate records will appear in their production environment (i.e. the interface).
So, the process is:
- Contact a Dune Team Member (@dsalv)
- Mention that we need to rebuild table XYZ (because of a schema change)
- Once they are aware/prepared run
```shell
docker run -v ${PWD}/data:/app/data \
--env-file .env \
ghcr.io/cowprotocol/dune-sync \
--sync-table SYNC_TABLE
```
This will empty the buckets and repopulate with the appropriate changes.