{"id":40699305,"url":"https://github.com/maris-development/beacon","last_synced_at":"2026-06-29T16:01:09.479Z","repository":{"id":281740845,"uuid":"920629743","full_name":"maris-development/beacon","owner":"maris-development","description":"A high-performance climate data lake supporting subsetting for zarr, netcdf, parquet, arrow ipc, csv and bbf","archived":false,"fork":false,"pushed_at":"2026-06-22T11:59:23.000Z","size":14706,"stargazers_count":9,"open_issues_count":35,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-06-22T12:25:02.023Z","etag":null,"topics":["blue-cloud2026","data-access","data-lake","docker","open-science","rest-api"],"latest_commit_sha":null,"homepage":"https://maris-development.github.io/beacon/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maris-development.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-01-22T13:53:22.000Z","updated_at":"2026-06-22T08:56:31.000Z","dependencies_parsed_at":"2025-04-18T17:42:34.672Z","dependency_job_id":"730573b3-63f0-488d-9e25-133ef4ef9cee","html_url":"https://github.com/maris-development/beacon","commit_stats":null,"previous_names":["maris-development/beacon"],"tags_count":23,"template":false,"template_full_name":null,"purl":"pkg:github/maris-development/beacon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maris-development%2Fbeacon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maris-development%2Fbeacon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maris-development%2Fbeacon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maris-development%2Fbeacon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maris-development","download_url":"https://codeload.github.com/maris-development/beacon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maris-development%2Fbeacon/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34933503,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-29T02:00:05.398Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blue-cloud2026","data-access","data-lake","docker","open-science","rest-api"],"created_at":"2026-01-21T12:00:18.238Z","updated_at":"2026-06-29T16:01:09.467Z","avatar_url":"https://github.com/maris-development.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Beacon — Data Lakehouse Query Engine\n\n[![Release](https://img.shields.io/github/v/release/maris-development/beacon?label=release\u0026color=success)](https://github.com/maris-development/beacon/releases)\n[![Docs](https://img.shields.io/github/actions/workflow/status/maris-development/beacon/pages.yml?label=docs)](https://maris-development.github.io/beacon/)\n[![codecov](https://codecov.io/gh/maris-development/beacon/branch/main/graph/badge.svg)](https://codecov.io/gh/maris-development/beacon)\n[![Docker](https://img.shields.io/badge/docker-ghcr.io-2496ED?logo=docker\u0026logoColor=white)](https://github.com/maris-development/beacon/pkgs/container/beacon)\n[![License](https://img.shields.io/github/license/maris-development/beacon)](LICENSE)\n[![Slack](https://img.shields.io/badge/slack-join-4A154B?logo=slack\u0026logoColor=white)](https://beacontechnic-wwa5548.slack.com/join/shared_invite/zt-2dp1vv56r-tj_KFac0sAKNuAgUKPPDRg)\n\nBeacon is a lightweight, high-performance **data lakehouse query engine** for scientific data. It lets you discover, read, transform, and serve large collections of array and tabular datasets **in place** — no copying into a warehouse, no rigid ETL pipeline. Point Beacon at a directory or object-storage bucket of files and query them directly over HTTP, with results streamed back in the format you ask for.\n\nIt is built on [Apache Arrow](https://arrow.apache.org/) and [Apache DataFusion](https://datafusion.apache.org/), so queries run on a columnar, vectorized engine while reading native scientific formats such as NetCDF, Zarr, Parquet, and ODV.\n\n\u003e 🚀 **In a hurry?** The [Quick Start guide](QUICKSTART.md) gets you running and querying — with the admin UI — in a couple of minutes.\n\n## Table of contents\n\n- [Quick Start guide](QUICKSTART.md)\n- [Why Beacon](#why-beacon)\n- [Features](#features)\n- [Concepts](#concepts)\n- [Quick start (Docker)](#quick-start-docker)\n- [Query examples](#query-examples)\n- [Configuration](#configuration)\n- [Documentation](#documentation)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Why Beacon\n\n- **Query files where they live.** Read NetCDF, Zarr, Parquet, ODV, CSV and more directly from a local volume or S3-compatible object store — no ingestion step.\n- **One API, many formats in and out.** Send a SQL or JSON query, choose your output format (Parquet, CSV, NetCDF, GeoParquet, Arrow IPC, ODV) and stream the result.\n- **Built for scale.** Columnar execution, predicate/projection pushdown, and statistics-based pruning on top of Arrow + DataFusion.\n- **Self-describing.** A built-in OpenAPI/Swagger UI documents every endpoint, and discovery endpoints expose available datasets, tables, columns, and functions.\n\n## Features\n\n- **Input formats:** Parquet, GeoParquet, NetCDF, Zarr, Atlas, ODV, CSV, Arrow IPC, GeoTIFF, Delta Lake, and the native Beacon Binary Format (BBF).\n- **Output formats:** Parquet, GeoParquet, NetCDF, ND-NetCDF, CSV, Arrow IPC, and ODV.\n- **Two query interfaces:** a structured **JSON query** API and raw **SQL** (enabled by default; toggle with `BEACON_ENABLE_SQL`).\n- **Arrow Flight SQL** endpoint for high-throughput clients (enabled by default).\n- **Storage backends:** local filesystem and S3-compatible object storage, with optional change-event watching.\n- **Interactive API docs** via Swagger UI (`/swagger`) and Scalar (`/scalar`).\n\n## Concepts\n\n- **Datasets** — the raw source files you make available to Beacon (e.g. `.nc`, `.zarr`, `.parquet`, `.csv`). Drop them into the mounted datasets directory and Beacon discovers them automatically.\n- **Tables** — named, queryable collections defined over one or more datasets, stored in the tables directory. A configurable **default table** (`BEACON_DEFAULT_TABLE`) is queried when no source is specified.\n- **Source functions** — table functions such as `read_netcdf(...)`, `read_parquet(...)`, and `read_csv(...)` let a query read specific files directly, without first defining a table.\n- **Query engine** — every request is parsed into a DataFusion logical plan and executed on the Arrow columnar engine, then encoded into the requested output format and streamed back.\n\nSee the [documentation](https://maris-development.github.io/beacon/) for the full data model.\n\n## Quick start (Docker)\n\nThe fastest way to try Beacon is a single `docker run`. Run it from the directory where you want the `datasets` and `tables` folders to live:\n\n```bash\ndocker run -d \\\n  --name beacon \\\n  -p 5001:5001 \\\n  -p 32011:32011 \\\n  -e BEACON_ADMIN_USERNAME=admin \\\n  -e BEACON_ADMIN_PASSWORD=securepassword \\\n  -v ./datasets:/beacon/data/datasets \\\n  -v ./tables:/beacon/data/tables \\\n  ghcr.io/maris-development/beacon:latest\n```\n\nThis maps the HTTP API (`5001`) and Arrow Flight SQL (`32011`) ports, sets the admin credentials, and mounts a local `./datasets` directory of files to query (plus an empty `./tables` directory you can omit if you won't use tables).\n\n### Docker Compose\n\nFor a reproducible setup, use Compose instead:\n\n```yaml\nservices:\n  beacon:\n    image: ghcr.io/maris-development/beacon:latest\n    container_name: beacon\n    restart: unless-stopped\n    ports:\n      - \"5001:5001\" # HTTP API\n      - \"32011:32011\" # Arrow Flight SQL\n    environment:\n      - BEACON_ADMIN_USERNAME=admin\n      - BEACON_ADMIN_PASSWORD=securepassword\n    volumes:\n      - ./datasets:/beacon/data/datasets # Mount a local directory of files to query\n      - ./tables:/beacon/data/tables # Mount an empty directory for tables, or omit if you won't use them\n```\n\nStart it with `docker compose up -d`, then open the interactive API docs at \u003chttp://localhost:5001/swagger/\u003e.\n\nAdd data by placing files (e.g. `.nc`, `.zarr`, `.parquet`, `.csv`) into `./datasets` — the container discovers them through the mounted volume.\n\n\u003e See the [installation guide](https://maris-development.github.io/beacon/docs/1.8.0/getting-started.html#local).\n\n## Query examples\n\nBoth examples below post to the same endpoint and stream back a file in the requested output format.\n\n### SQL\n\n\u003e SQL is enabled by default but can be disabled. Set `BEACON_ENABLE_SQL=false` to disable it.\n\n```http\nPOST http://localhost:5001/api/query\nContent-Type: application/json\n\n{\n  \"sql\": \"SELECT TEMP, PSAL, LONGITUDE, LATITUDE FROM read_netcdf(['data/2020.nc', 'data/2021.nc']) WHERE time \u003e '2020-01-01T00:00:00'\",\n  \"output\": { \"format\": \"parquet\" }\n}\n```\n\n### JSON\n\nThe JSON query API is read-only, always available, and needs no extra configuration.\n\n```http\nPOST http://localhost:5001/api/query\nContent-Type: application/json\n\n{\n  \"query_parameters\": [\n    { \"column_name\": \"TEMP\", \"alias\": \"temperature\" },\n    { \"column_name\": \"PSAL\", \"alias\": \"salinity\" },\n    { \"column_name\": \"TIME\" },\n    { \"column_name\": \"LONGITUDE\" },\n    { \"column_name\": \"LATITUDE\" }\n  ],\n  \"filters\": [\n    { \"for_query_parameter\": \"temperature\", \"min\": -2, \"max\": 35 },\n    { \"for_query_parameter\": \"salinity\", \"min\": 30, \"max\": 42 },\n    {\n      \"and\": [\n        { \"for_query_parameter\": \"LONGITUDE\", \"min\": -20, \"max\": 20 },\n        { \"for_query_parameter\": \"LATITUDE\", \"min\": 40, \"max\": 65 }\n      ]\n    }\n  ],\n  \"from\": {\n    \"netcdf\": { \"paths\": [\"data/2020.nc\", \"data/2021.nc\"] }\n  },\n  \"output\": { \"format\": \"csv\" }\n}\n```\n\nThe response is a streamed file in the chosen `output.format` (here, CSV). See the [query reference](https://maris-development.github.io/beacon/docs/1.8.0/api/querying/) for the full schema, all source types, and every output format.\n\n## Configuration\n\nBeacon is configured entirely through `BEACON_*` environment variables. The most common ones:\n\n| Variable | Default | Description |\n| --- | --- | --- |\n| `BEACON_HOST` | `0.0.0.0` | Address the HTTP server binds to. |\n| `BEACON_PORT` | `5001` | HTTP server port. |\n| `BEACON_ADMIN_USERNAME` | `beacon-admin` | Admin username for management endpoints. |\n| `BEACON_ADMIN_PASSWORD` | `beacon-password` | Admin password — **change this in production**. |\n| `BEACON_LOG_LEVEL` | `info` | Log verbosity (`trace`, `debug`, `info`, `warn`, `error`). |\n| `BEACON_VM_MEMORY_SIZE` | `8192` | Working memory (MB) available to the query engine. |\n| `BEACON_DEFAULT_TABLE` | `default` | Table queried when a request specifies no source. |\n| `BEACON_WORKER_THREADS` | `8` | Number of worker threads for the async runtime. |\n| `BEACON_ENABLE_SQL` | `true` | Enable the read-only raw SQL query interface. |\n| `BEACON_FLIGHT_SQL_ENABLE` | `true` | Enable the Arrow Flight SQL endpoint. |\n| `BEACON_FLIGHT_SQL_PORT` | `32011` | Arrow Flight SQL port. |\n\nS3-compatible storage, CORS, NetCDF caching, the crawler, and Flight SQL authentication have their own `BEACON_*` settings — see the [configuration reference](https://maris-development.github.io/beacon/docs/1.8.0/data-lake/configuration.html) for the complete list.\n\n## Documentation\n\n- Docs home: \u003chttps://maris-development.github.io/beacon/\u003e\n- Getting started: \u003chttps://maris-development.github.io/beacon/docs/1.8.0/getting-started.html#local\u003e\n- Query reference: \u003chttps://maris-development.github.io/beacon/docs/1.8.0/api/querying/\u003e\n- Community Slack: [join here](https://beacontechnic-wwa5548.slack.com/join/shared_invite/zt-2dp1vv56r-tj_KFac0sAKNuAgUKPPDRg)\n\n## Contributing\n\nBeacon is a Rust workspace. To build and test from source:\n\n```bash\ngit clone https://github.com/maris-development/beacon.git\ncd beacon\ncargo build --release\ncargo test\n```\n\nIssues and pull requests are welcome on [GitHub](https://github.com/maris-development/beacon/issues). For larger changes, please open an issue first to discuss the approach.\n\n## License\n\nBeacon is licensed under the **GNU Affero General Public License v3.0** (AGPL-3.0). See [LICENSE](LICENSE) for the full text.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaris-development%2Fbeacon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaris-development%2Fbeacon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaris-development%2Fbeacon/lists"}