{"id":45748756,"url":"https://github.com/frizzleqq/f1-dbt-duckdb","last_synced_at":"2026-02-25T17:42:25.201Z","repository":{"id":75144119,"uuid":"527218867","full_name":"frizzleqq/f1-dbt-duckdb","owner":"frizzleqq","description":"Data pipelines with dragster, dbt and duckdb using Ergast F1 data as source","archived":false,"fork":false,"pushed_at":"2025-04-11T13:44:45.000Z","size":819,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-11T14:10:09.495Z","etag":null,"topics":["dagster","dbt","duckdb","ergast-data"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/frizzleqq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-08-21T13:45:04.000Z","updated_at":"2024-12-30T12:30:31.000Z","dependencies_parsed_at":"2023-06-05T13:47:39.398Z","dependency_job_id":"5fc87529-a7bf-47e2-848a-f1e2bb1d10d8","html_url":"https://github.com/frizzleqq/f1-dbt-duckdb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/frizzleqq/f1-dbt-duckdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frizzleqq%2Ff1-dbt-duckdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frizzleqq%2Ff1-dbt-duckdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frizzleqq%2Ff1-dbt-duckdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frizzleqq%2Ff1-dbt-duckdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/frizzleqq","download_url":"https://codeload.github.com/frizzleqq/f1-dbt-duckdb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frizzleqq%2Ff1-dbt-duckdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29832976,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-25T17:17:09.781Z","status":"ssl_error","status_checked_at":"2026-02-25T17:16:50.421Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dagster","dbt","duckdb","ergast-data"],"created_at":"2026-02-25T17:42:24.126Z","updated_at":"2026-02-25T17:42:25.190Z","avatar_url":"https://github.com/frizzleqq.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# F1 warehouse with DuckDB\n\nProject with\n* Orchestration: [dagster](https://docs.dagster.io/)\n* Transformation \u0026 Testing: [dbt](https://docs.getdbt.com/)\n* Processing Engine \u0026 Database: [DuckDB](https://duckdb.org/)\n* Data Source: [Ergast API](http://ergast.com/mrd/) (Ergast is no longer being updated post 2024 season)\n\nCutout of the dagster lineage graph:\n![alt text](docs/dagster_lineage.png \"dagster lineage\")\n\nExample data in MotherDuck:\n![alt text](docs/motherduck.png \"Motherduck\")\n\n## Development\n\n* Python \u003e= 3.10 https://www.python.org/downloads/\n\nThis project uses `pyproject.toml` to describe package metadata and [uv](https://github.com/astral-sh/uv) to manage dependencies.\n\n### Install uv\n\nhttps://docs.astral.sh/uv/getting-started/installation/\n\n### Setup environment\n\nCreate virtual environment\n```bash\nuv venv\n```\n\nFollowing command installs Python dependencies:\n* The `[dev]` also installs development tools.\n* The `--editable` makes the CLI script available.\n```bash\nuv pip install --editable .[dev]\n```\n\nSync `uv` environment:\n```bash\nuv sync --extra dev\n```\n\n### Activate virtual environment\n\nBash:\n```bash\nsource .venv/bin/activate\n```\n\nWindows:\n```powershell\n.venv\\Scripts\\activate\n```\n\n### Dagster\n\nDagster uses environment variables located in [.env](.env).\nPlease copy the `.env_template` over to `.env` or set the `DATA_DIR` environment variable.\n\nStart local dagster server\n```bash\ndagster dev\n```\n\n### Dagster CLI\n\nLaunch dagster job without\n```bash\ndagster job execute -m foneplatform -j ergast_job\n```\n\n### Development Tools\n\n* Code linting/formatting: `ruff`\n* type checking: `mypy`\n* SQL linting/formatting: `sqlfluff`\n\n\n## Environments\n\nBased on the environment variable `ENVIRONMENT` we do:\n\n* `dev`: Locally a `f1.duckdb` database will be created within the `DATA_DIR` (defaults to \"data\" within project directory)\n* `md`: Connects to MotherDuck to store the `f1` database using the `MOTHERDUCK_TOKEN`\n\nUse the `.env` file to set environment variables\n\n\n## dbt\n\nInstall dbt_utils:\n```\ndbt deps --project-dir=\"./dbt\" --profiles-dir=\"./dbt\"\n```\n\nRun models:\n```\ndbt run --project-dir=\"./dbt\" --profiles-dir=\"./dbt\"\n```\n\nRun tests:\n```\ndbt test --project-dir=\"./dbt\" --profiles-dir=\"./dbt\"\n```\n\n### sqlfluff\n\nRun SQL linter on dbt models:\n\u003e **_NOTE:_** This may require setting the `DATA_DIR` environment variable to be set to the `data` directory containing the duckdb database.\n```\nsqlfluff lint ./dbt/models/core\n```\n\n## Staging\n\nStaging is done by a Dagster Multi-Asset ([./foneplatform/assets/ergast.py](./foneplatform/assets/ergast.py)):\n1. Downloading ZIP of CSV files (http://ergast.com/downloads/f1db_csv.zip)\n1. Read CSV using DuckDB and store the asset-result in DuckDB using the `DuckDBDuckDBIOManager`\n1. dbt will create models in the same DuckDB database\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrizzleqq%2Ff1-dbt-duckdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrizzleqq%2Ff1-dbt-duckdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrizzleqq%2Ff1-dbt-duckdb/lists"}