{"id":22165596,"url":"https://github.com/textileio/demo-basin-wxm-query","last_synced_at":"2025-03-24T16:14:18.888Z","repository":{"id":223399761,"uuid":"751570756","full_name":"textileio/demo-basin-wxm-query","owner":"textileio","description":"Fetch Basin WeatherXM data, run queries with DuckDB, and store the history \u0026 visualizations in local files.","archived":false,"fork":false,"pushed_at":"2024-05-23T02:47:59.000Z","size":26877,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-29T21:51:04.102Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/textileio.png","metadata":{"files":{"readme":"README.md","changelog":"history.csv","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-01T21:41:12.000Z","updated_at":"2024-05-23T02:48:03.000Z","dependencies_parsed_at":"2024-02-26T01:25:28.974Z","dependency_job_id":"b6a77117-fa04-4d46-a106-5692f69af339","html_url":"https://github.com/textileio/demo-basin-wxm-query","commit_stats":null,"previous_names":["textileio/demo-basin-wxm-query"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/textileio%2Fdemo-basin-wxm-query","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/textileio%2Fdemo-basin-wxm-query/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/textileio%2Fdemo-basin-wxm-query/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/textileio%2Fdemo-basin-wxm-query/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/textileio","download_url":"https://codeload.github.com/textileio/demo-basin-wxm-query/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245304873,"owners_count":20593626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-02T05:15:42.239Z","updated_at":"2025-03-24T16:14:18.860Z","avatar_url":"https://github.com/textileio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Basin + WeatherXM Queries Demo\n\n[![standard-readme compliant](https://img.shields.io/badge/standard--readme-OK-green.svg)](https://github.com/RichardLitt/standard-readme)\n\n\u003e Compute queries for WeatherXM data pushed to Textile Basin + Vaults\n\n## Table of Contents\n\n- [Background](#background)\n  - [Data](#data)\n- [Install](#install)\n- [Usage](#usage)\n  - [Flags](#flags)\n  - [Makefile Reference](#makefile-reference)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Background\n\nThis project runs a simple analysis on [WeatherXM](https://weatherxm.com/) data that's pushed to Tableland [Basin](https://github.com/tablelandnetwork/basin-cli) (replicated to Filecoin). It fetches remote data and queries it with [DuckDB](https://duckdb.org).\n\n### Data\n\nThe script fetches data from the WeatherXM's `wxm.weather_data_dev` vault (created by `0x75c9eCb5309d068f72f545F3fbc7cCD246016fE0`) on a cron schedule with GitHub Actions. For every run, it will query data and write the results to:\n\n- [Data](./Data.md): Summary metrics for the run, including averages across all columns.\n- [History](./history.csv): A CSV file containing the full history of all runs, along with the run date and time.\n\n## Install\n\nTo set things up (for local development), you'll need to do the following. First, install `pipx` and `pipenv`:\n\n```\npython3 -m pip install pipx\npython3 -m pipx ensurepath\npipx install pipenv\npipenv run pip install --upgrade pip setuptools wheel\n```\n\nThen, install dependencies:\n\n```sh\npipenv install --dev\n```\n\nAnd then, activate the virtual environment and set up pre-commit hooks:\n\n```sh\npipenv shell\npipenv run pre-commit install -t pre-commit\npipenv run pre-commit install -t pre-push\n```\n\nNote the core dependencies installed are:\n\n- [`contextily`](https://contextily.readthedocs.io/en/latest/): Used for plotting data on a map.\n- [`duckdb`](https://duckdb.org/docs/api/python/overview): Creates an in-memory SQL database for querying parquet files extracted from Basin.\n- [`geopandas`](https://geopandas.org/en/stable/): Also used for plotting data on a map.\n- [`shapely`](https://pypi.org/project/shapely/): Required for `geopandas` to work.\n- [`requests`](https://pypi.org/project/requests/): Make requests to Basin HTTP API to fetch data.\n- [`polars`](https://pola.rs/): Used for DataFrame operations as part of post-query logic.\n- [`pyarrow`](https://pypi.org/project/pyarrow/): Required for DuckDB to work with parquet files.\n- [`rich`](https://github.com/Textualize/rich): Used for logging purposes.\n\nOnce you've done this, you'll also need to make sure the [Basin CLI](https://github.com/tablelandnetwork/basin-cli) is installed; it's part of the underlying application logic. You'll need [Go](https://go.dev/doc/install) 1.21 installed to do this, and then run:\n\n```sh\ngo install github.com/tablelandnetwork/basin-cli/cmd/vaults@latest\n```\n\nAlso, the [`go-car`](https://github.com/ipld/go-car) CLI is required to extract the underlying parquet files from the CAR files retrieved from Basin. You'll need Go 1.20 (note: different thant the Basin CLI) and can install it with:\n\n```sh\ngo install github.com/ipld/go-car/cmd/car@latest\n```\n\n## Usage\n\nRunning `basin_wxm/main.py` will fetch remote files from Tableland Basin, extract the contents with `go-car`, load the resulting parquet files into a DuckDB in-memory database, run queries on the data, and then collect them into a polars DataFrame for final operations (e.g., writing to files).\n\nTo use default time ranges (the full dataset), run:\n\n```sh\nmake run\n```\n\nOr, you can define a custom time range with `start` and `end` arguments (Unix epoch timestamps _or_ YYYY-MM-DD), which will be used to filter the data when queries are executed. Note: the timestamp range for the `wxm.weather_data_dev` vault starts on `1707436800`.\n\n```sh\nmake run start=1707436800 end=2024-02-15\n```\n\nThis range also defines which events/data is fetched; the `cache.json` file will store all previously extracted events, so only new events will be fetched on subsequent runs.\n\nOnce you run the command, it'll log information about the current status of each step in the run and the total time to complete upon finishing:\n\n```sh\n[18:45:46] INFO     Getting events for vault...done in 0.53s\n           INFO     Number of events found: 7\n           INFO     Number of new events: 7\n[18:51:04] INFO     Extracting data from events...done in 317.89s\n           INFO     Creating database with parquet files...done in 0.02s\n[18:51:11] INFO     Executing queries...done in 6.92s\n[18:51:29] INFO     Generating bbox plots...done in 17.88s\n⠙ Writing results to files...\n```\n\n\u003e Note: The program will download the files locally before creating the database and running queries, which will use up a bit of memory. For example, five wxm parquet files will total to ~1.2 GiB in terms of raw file size (each is 200-250 MiB).\n\n### Flags\n\nThe following flags are available for the `main.py` script:\n\n- `--start`: Start timestamp (in unix) for the query (e.g., 1707436800). Defaults to full range.\n- `--end`: End timestamp (in unix) for the query (e.g., 1707955200). Defaults to full range.\n- `--verbose`: Enable verbose logging to show stack traces for errors. Defaults to true.\n\n### Makefile Reference\n\nThe following defines all commands available in the Makefile:\n\n- `make run`: Run the `main.py` program to fetch Basin/wxm data, run queries, and write metrics to summary files.\n- `make install`: Install dependencies with `pip`, upgrading pip first.\n- `make setup`: Use the virtual environment and set up pre-commit hooks.\n- `make format`: Run `black`, `isort`, `mypy`, and `flake8` to format and lint the code.\n- `make basin`: Install the Basin CLI from the latest release.\n- `make ipfs-car`: Install `ipfs-car` from the latest release.\n- `make test`: Run the (dummy) tests.\n\n## Contributing\n\nPRs accepted.\n\nSmall note: If editing the README, please conform to the standard-readme specification.\n\n## License\n\nMIT AND Apache-2.0, © 2021-2024 Textile Contributors\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftextileio%2Fdemo-basin-wxm-query","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftextileio%2Fdemo-basin-wxm-query","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftextileio%2Fdemo-basin-wxm-query/lists"}