{"id":18564772,"url":"https://github.com/luftdaten-at/luftdaten-api","last_synced_at":"2026-04-27T23:01:07.537Z","repository":{"id":250144912,"uuid":"833554059","full_name":"luftdaten-at/luftdaten-api","owner":"luftdaten-at","description":"Open source database, analytics and API for air quality data build on the FastAPI Framework.","archived":false,"fork":false,"pushed_at":"2026-04-20T20:40:37.000Z","size":388,"stargazers_count":1,"open_issues_count":7,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-04-20T22:40:22.905Z","etag":null,"topics":["database","fastapi"],"latest_commit_sha":null,"homepage":"https://api.luftdaten.at","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luftdaten-at.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-07-25T09:29:09.000Z","updated_at":"2026-04-20T20:40:15.000Z","dependencies_parsed_at":"2024-07-25T13:08:01.430Z","dependency_job_id":"791b5bc8-f68c-45f3-afe6-e4a08b49fe38","html_url":"https://github.com/luftdaten-at/luftdaten-api","commit_stats":null,"previous_names":["luftdaten-at/luftdaten-api"],"tags_count":47,"template":false,"template_full_name":null,"purl":"pkg:github/luftdaten-at/luftdaten-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luftdaten-at%2Fluftdaten-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luftdaten-at%2Fluftdaten-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luftdaten-at%2Fluftdaten-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luftdaten-at%2Fluftdaten-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luftdaten-at","download_url":"https://codeload.github.com/luftdaten-at/luftdaten-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luftdaten-at%2Fluftdaten-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32358509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T20:07:02.737Z","status":"ssl_error","status_checked_at":"2026-04-27T20:07:00.910Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","fastapi"],"created_at":"2024-11-06T22:16:17.385Z","updated_at":"2026-04-27T23:01:07.530Z","avatar_url":"https://github.com/luftdaten-at.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# luftdaten-api\n\n## About luftdaten-api\nluftdaten-api ist an open source database for air quality data build on the FastAPI Framework.\n\n## Documentation\n\n- **HTTP API (all endpoints):** [docs/endpoints.md](docs/endpoints.md) and [docs/README.md](docs/README.md) (links to interactive OpenAPI).\n- **OpenAPI / FastAPI documentation (maintainer guide):** [docs/fastapi-documentation/README.md](docs/fastapi-documentation/README.md).\n- **Database schema:** [docs/database/README.md](docs/database/README.md) (tables, materialized views, indexes).\n\n### Development\nEnvironment variables: copy **`.env.example`** to **`.env`** and adjust (database credentials, `DB_HOST`, optional `LOG_LEVEL`, monitoring vars for prod).\n\nDevelopment version:\n\n    cp .env.example .env\n    docker compose up -d\n\n\n#### Database migration\nSetup alembic folder and config files:\n    \n    docker compose exec app alembic init alembic\n\nGenerate and apply migrations:\n    \n    docker compose exec app alembic revision --autogenerate -m \"Initial migration\"\n    docker compose exec app alembic upgrade head\n\nRollback migrations:\n    \n    docker compose exec app alembic downgrade\n\n#### Database reset\n\n    docker compose down\n    docker volume ls\n    docker volume rm luftdaten-api_postgres_data\n    docker compose up -d\n    docker compose exec app alembic upgrade head\n\n#### Running Tests\n\nRun all unit tests via Docker:\n\n    docker compose run --rm test\n\nOr use the convenience script:\n\n    ./run_tests.sh\n\nRun specific test files:\n\n    docker compose run --rm test pytest tests/test_city.py -v\n    docker compose run --rm test pytest tests/test_health.py -v\n    docker compose run --rm test pytest tests/test_station.py -v\n\nRun tests with coverage (requires pytest-cov in requirements.txt):\n\n    docker compose run --rm test pytest tests/ --cov=. --cov-report=html --cov-report=term\n\nThe test service uses a separate test database (`db_test`) that is automatically set up and torn down.\n\n#### Station blacklist\n\nStations can be excluded from API responses via a blacklist config file. Blacklisted stations are omitted from all station, city, and statistics endpoints.\n\n**Location:** `config/station_blacklist.json`\n\n**Format:** JSON array of device IDs, e.g.:\n```json\n[\"12345\", \"67890\"]\n```\n\n**Editing:** Add or remove device IDs, save the file, then restart the app. With Docker Compose, the `config/` folder is mounted, so changes take effect after `docker compose restart app`.\n\n**Environment variable:** `STATION_BLACKLIST_FILE` — override the blacklist file path (e.g. `/app/config/station_blacklist.json` in Docker).\n\n**Behavior:**\n- Missing file or empty array → no stations excluded\n- Invalid JSON → startup fails\n- Blacklisted stations return 404 on `/station/info`; they are filtered from all other endpoints\n\n**Station ingest (measurements / status):** Use **POST** to `/v1/station/data` and `/v1/station/status` with the JSON body shape from OpenAPI (`/docs`). **GET** on these paths returns **405** — they will not show up as successful “reads” in traffic summaries. With or without a **trailing slash** is supported (a slash-only route used to **307**-redirect and break some embedded HTTP stacks). Set **`LOG_STATION_INGEST=true`** in `.env` to log each ingest attempt (`path` + HTTP status, including **422**).\n\n#### Monitoring\n\n**Built-in monitor endpoint** (`GET /v1/monitor`):\n- Database usage: size, connections, cache hit ratio, transactions, top tables by size\n- API stats: request counts by endpoint and status code (since startup)\n- Application: uptime, scheduler jobs, blacklist size\n\n**Prometheus metrics** (`GET /v1/metrics` or `/metrics` without `/v1` prefix):\n- **HTTP (instrumentator):** `http_requests_total` (labels `handler`, `method`, `status` with `2xx`/`4xx` buckets), `http_request_duration_seconds` (per `handler`), `http_request_duration_highr_seconds` (global latency)\n- **Custom `luftdaten_*`:** `luftdaten_http_requests_total{area,method,status}` for roll-up by API area; gauges `luftdaten_blacklist_size`, `luftdaten_scheduler_jobs`, `luftdaten_db_up` (refreshed every minute)\n- Scrape noise (`/metrics`, `/health/simple`, `/monitor`) is excluded from default HTTP metrics but still visible in logs; custom area counter follows the same exclusions\n- Design details and example PromQL: [`docs/PROMETHEUS_GRAFANA_ENDPOINTS_PLAN.md`](docs/PROMETHEUS_GRAFANA_ENDPOINTS_PLAN.md)\n\n**Optional monitoring stack** (Postgres exporter, Prometheus, Grafana):\n- Start with: `docker compose --profile monitoring up -d`\n- Grafana: http://localhost:3000 (default login: admin/admin) — datasource and dashboard **Luftdaten API** are provisioned from `monitoring/grafana/`\n- Prometheus: http://localhost:9090 — config in `monitoring/prometheus.yml`, optional recording rules in `monitoring/prometheus/rules.yml`\n- Grafana panels filter metrics with `job=\"\u003cname\u003e\"`. The **Luftdaten API** dashboard exposes **API Prometheus job** (from `label_values(http_requests_total, job)`); pick the value that matches your scrape config’s `job_name` (default in repo: `luftdaten-api`). If panels show *No data*, check **Status → Targets** in Prometheus and run `http_requests_total` in **Graph** to see the real `job` label. The app must register **`metrics.default()`** alongside the custom `luftdaten_*` instrumentation (see [`code/main.py`](code/main.py)); otherwise `http_requests_total` and latency histograms are never emitted and Grafana stays empty regardless of `job`.\n- **`handler=\"none\"`** for most traffic: prometheus-fastapi-instrumentator resolves the route **before** inner middleware runs. `/v1/...` must be stripped **outside** that middleware (see `VersionPrefixMiddleware` in [`code/main.py`](code/main.py)); otherwise routes registered as `/station/...` never match and Grafana shows `none` instead of `/station/current`, etc.\n\n**PostgreSQL query diagnostics (slow SQL / index hints):**\n- The `db` service loads **`pg_stat_statements`** via `shared_preload_libraries` (see `docker-compose.yml`). After pulling the change, **restart Postgres** (or recreate the volume) so the setting applies, then run **`alembic upgrade head`** — migration **`7f3a9c2e1d0b`** runs `CREATE EXTENSION IF NOT EXISTS pg_stat_statements`.\n- Existing data directories that were created **without** this `command` need a one-time restart of the `db` container after the compose update; if `CREATE EXTENSION` still errors, ensure `shared_preload_libraries` is active (`SHOW shared_preload_libraries;` in `psql`).\n- **`postgres_exporter`** (monitoring profile) enables **`--collector.stat_statements`** and **`--collector.statio_user_indexes`** with a 40-statement cap. Grafana dashboard **Luftdaten API** adds panels for cumulative time by `queryid`, seq scans by table, time rate, and index block read rate (`job=\"postgres\"`).\n- Map a **`queryid`** back to SQL in `psql`: `SELECT query FROM pg_stat_statements WHERE queryid = \u003cvalue\u003e;` (large cardinality — do not enable `--collector.stat_statements.include_query` on busy servers without care).\n- App connections set **`application_name`** (override with `POSTGRES_APPLICATION_NAME`). Optional dev-only SQL logging: **`DB_SQL_ECHO=true`** (see `.env.example`).\n\n#### Deployment\n\nBuild and push to Dockerhub.\n\n    docker build -f Dockerfile.prod -t luftdaten/api:tagname --platform linux/amd64 .\n    docker push luftdaten/api:tagname\n\nCurrently automaticly done by Github Workflow.\nTags:\n    - **staging**: latest version for testing\n    - **x.x.x**: released versions for production\n\n### Production\n\nCreate docker-compose.prod.yml from example-docker-compose.prod.yml by setting the secret key. Then run:\n\n    docker compose -f docker-compose.prod.yml up -d\n\nOptional **monitoring** (Prometheus, Grafana, postgres_exporter): copy `monitoring/` from the repo next to your compose file, set `GRAFANA_ADMIN_PASSWORD` in `.env`, then:\n\n    docker compose -f docker-compose.prod.yml --profile monitoring up -d\n\nProduction example compose exposes Grafana via Traefik at `grafana.staging.api.luftdaten.at` (override with `GF_SERVER_ROOT_URL` in `.env` if your host differs). Traefik’s `loadbalancer.server.port` for Grafana must be **3000** (Grafana’s default HTTP port), not 80 — otherwise Traefik returns **502 Bad Gateway**.\n\nCreate database structure:\n    \n    docker compose exec app alembic upgrade head    \n\n## API Documentation\n\nOpen API Standard 3.1\n\n/docs\nhttps://api.luftdaten.at/docs\n\n**Note:** `GET /v1/station/historical` requires **`station_ids`** with at least one device ID (comma-separated). Omitting it or sending an empty value returns **422**; use `GET /v1/station/all` (or similar) to discover IDs first.\n\n## License\nThis project is licensed under GNU General Public License v3.0.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluftdaten-at%2Fluftdaten-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluftdaten-at%2Fluftdaten-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluftdaten-at%2Fluftdaten-api/lists"}