https://github.com/guidok91/aviation-analytics
Aviation analytics with DuckDB, dbt and dlt using AirLabs data.
https://github.com/guidok91/aviation-analytics
airlabs dbt dlt duckdb
Last synced: 4 months ago
JSON representation
Aviation analytics with DuckDB, dbt and dlt using AirLabs data.
- Host: GitHub
- URL: https://github.com/guidok91/aviation-analytics
- Owner: guidok91
- Created: 2023-09-08T09:40:24.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2026-02-27T09:14:20.000Z (4 months ago)
- Last Synced: 2026-02-27T14:28:23.089Z (4 months ago)
- Topics: airlabs, dbt, dlt, duckdb
- Language: Makefile
- Homepage:
- Size: 530 KB
- Stars: 7
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Aviation analytics with DuckDB, dbt and dlt using AirLabs data
Aviation analytics project with [DuckDB](https://duckdb.org/), [dbt](https://docs.getdbt.com/docs/introduction) and [dlt](https://dlthub.com/) using the data from [AirLabs API](https://airlabs.co).
- [Data Architecture](#data-architecture)
- [Running instructions](#running-instructions)
- [Spin up Docker containers](#spin-up-docker-containers)
- [Ingest source data from AirLabs REST API to DuckDB using dlt](#ingest-source-data-from-airlabs-rest-api-to-duckdb-using-dlt)
- [Run dbt models to transform and curate the data](#run-dbt-models-to-transform-and-curate-the-data)
- [Data exploration with the DuckDB UI](#data-exploration-with-the-duckdb-ui)
- [Countries with the highest number of airports](#countries-with-the-highest-number-of-airports)
- [Current number of flights by status](#current-number-of-flights-by-status)
- [Explore dbt project docs](#explore-dbt-project-docs)
- [Dependency management](#dependency-management)
- [CI/CD](#cicd)
## Data Architecture
Datasets are ingested as snapshots from the source API.
When loading to the curated tables, they are processed with a `merge` strategy (using a unique key per dataset).
This preserves the latest version of each record to keep it simple on the querying side.
## Running instructions
Run `make help` to see available commands together with their description.
### Spin up Docker containers
Build and spin up Docker containers needed for the app:
- `make docker-up`
### Ingest source data from AirLabs REST API to DuckDB using dlt
Get into the dlt container:
- `make docker-it-dlt`
For this step we first need to generate an AirLabs API key (see how to on their website), and set the environment variable `AIRLABS_API_KEY`. Then run:
- `make dlt-ingest-source-data`
### Run dbt models to transform and curate the data
Exit the dlt container and get into the dbt one by running `make docker-it-dbt`. Then:
- `make dbt-deps`
- `make dbt-run`
## Data exploration with the DuckDB UI
Once the models have been run and the data is ready, you can start exploring the data.
Run `make duckdb-ui` to lauch the DuckDB UI and access it via http://localhost:4213.
Example queries:
### Countries with the highest number of airports
```sql
SELECT
country_code,
COUNT(*)
FROM
curated.airports
GROUP BY
country_code
ORDER BY
count(*) DESC
LIMIT 10;
```
### Current number of flights by status
```sql
SELECT
flight_status,
COUNT(*)
FROM
curated.flight_positions
WHERE
processed_timestamp = (SELECT MAX(processed_timestamp) FROM curated.flight_positions)
GROUP BY
ALL
ORDER BY
COUNT(*) DESC;
```
## Explore dbt project docs
dbt provides auto-generated documentation for the project. Run `make dbt-docs` and access http://localhost:8080.
## Dependency management
Dependabot is configured to periodically upgrade repo dependencies. See [dependabot.yml](.github/dependabot.yml).
## CI/CD
A Github Actions CI/CD pipeline that runs the models, tests and code linting is defined [here](.github/workflows) and can be seen [here](https://github.com/guidok91/duckdb-dbt/actions).
Note that the `AIRLABS_API_KEY` is provided as a Github repository secret to be used in the CI/CD pipeline.