{"id":50751154,"url":"https://github.com/felipemchdev/data-dbt-batch","last_synced_at":"2026-06-11T01:03:56.261Z","repository":{"id":349908054,"uuid":"1204269953","full_name":"felipemchdev/data-dbt-batch","owner":"felipemchdev","description":"End-to-end batch pipeline using Olist data with Bronze/Silver/Gold layers,","archived":false,"fork":false,"pushed_at":"2026-04-08T03:28:50.000Z","size":43327,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-04-08T05:21:15.144Z","etag":null,"topics":["airflow","dags","data-engineering","dbt","docker","duckdb","great-expectations","pipeline"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/felipemchdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-07T21:16:47.000Z","updated_at":"2026-04-08T03:33:25.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/felipemchdev/data-dbt-batch","commit_stats":null,"previous_names":["felipemchdev/data-dbt-batch"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/felipemchdev/data-dbt-batch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fdata-dbt-batch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fdata-dbt-batch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fdata-dbt-batch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fdata-dbt-batch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/felipemchdev","download_url":"https://codeload.github.com/felipemchdev/data-dbt-batch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fdata-dbt-batch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34177449,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","dags","data-engineering","dbt","docker","duckdb","great-expectations","pipeline"],"created_at":"2026-06-11T01:03:55.455Z","updated_at":"2026-06-11T01:03:56.251Z","avatar_url":"https://github.com/felipemchdev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lakehouse Olist Batch Pipeline\n\nThis project is an end-to-end batch pipeline using Olist data with Bronze/Silver/Gold layers.\nIt combines Python scripts, dbt models, DuckDB, and Airflow in a practical local setup.\n\n## Stack\n\n- Python\n- DuckDB\n- dbt (dbt-duckdb)\n- Great Expectations\n- Apache Airflow\n- Docker Compose\n\n## Project Structure\n\n```text\nairflow/\n  dags/\ndbt/\n  models/staging/\n  models/marts/\nsrc/\n  ingest/\n  ge/\n  analytics/\ndata/\n  raw/\n  bronze/\n  gold/\n```\n\n## Expected Files in `data/raw`\n\n- `customers.csv` or `olist_customers_dataset.csv`\n- `orders.csv` or `olist_orders_dataset.csv`\n- `order_items.csv` or `olist_order_items_dataset.csv`\n- `order_payments.csv` or `olist_order_payments_dataset.csv`\n- `products.csv` or `olist_products_dataset.csv`\n- `sellers.csv` or `olist_sellers_dataset.csv`\n- `geolocation.csv` or `olist_geolocation_dataset.csv`\n\n## Run From Scratch (Docker + Airflow)\n\n1. Start Docker Desktop.\n2. Build images:\n\n```bash\ndocker-compose build dbt airflow-webserver airflow-scheduler\n```\n\n3. Initialize Airflow metadata DB:\n\n```bash\ndocker-compose run --rm airflow-webserver airflow db init\n```\n\n4. Create Airflow admin user:\n\n```bash\ndocker-compose run --rm airflow-webserver airflow users create \\\n  --username admin \\\n  --password admin \\\n  --firstname \"\u003cyour_first_name\u003e\" \\\n  --lastname \"\u003cyour_last_name\u003e\" \\\n  --role Admin \\\n  --email \"\u003cyour_email\u003e\"\n```\n\n5. Start services:\n\n```bash\ndocker-compose up -d airflow-webserver airflow-scheduler\n```\n\n6. Open Airflow UI:\n\n- URL: `http://localhost:8080`\n- User: `admin`\n- Password: `admin`\n\n7. Unpause and trigger the main DAG:\n\n```bash\ndocker-compose exec airflow-scheduler airflow dags unpause lakehouse_olist_pipeline\ndocker-compose exec airflow-scheduler airflow dags trigger lakehouse_olist_pipeline\n```\n\n## Optional: Run dbt Only\n\n```bash\ndocker-compose run --rm dbt dbt debug\ndocker-compose run --rm dbt dbt run\ndocker-compose run --rm dbt dbt test\n```\n\n## Outputs\n\n- DuckDB database: `data/lakehouse.duckdb`\n- Bronze parquet files: `data/bronze/*.parquet`\n- HTML report: `data/gold/report.html`\n- KPI summary: `data/gold/kpi_summary.json`\n- CSV exports: `data/gold/exports/*.csv`\n\n## See Results in DuckDB UI\n\nOpen the correct file:\n\n```bash\nduckdb -ui data/lakehouse.duckdb\n```\n\nIn SQL editor:\n\n```sql\nSHOW ALL TABLES;\n\nSELECT table_schema, table_name\nFROM information_schema.tables\nWHERE table_type = 'BASE TABLE'\nORDER BY 1,2;\n```\n\n## Useful Commands\n\nEnter Airflow container:\n\n```bash\ndocker-compose exec airflow-scheduler bash\n```\n\nStop everything:\n\n```bash\ndocker-compose down\n```\n\nClean unused Docker resources:\n\n```bash\ndocker system prune -f\n```\n\n## Quick Troubleshooting\n\n- DAGs are not showing in UI\n  - Make sure scheduler is running:\n  - `docker-compose up -d airflow-scheduler`\n\n- `ModuleNotFoundError: duckdb` in Airflow tasks\n  - Rebuild and restart Airflow services:\n  - `docker-compose build airflow-webserver airflow-scheduler`\n  - `docker-compose up -d --force-recreate airflow-webserver airflow-scheduler`\n\n- `Permission denied` writing under `/app/data`\n  - Recreate services with current compose config:\n  - `docker-compose up -d --force-recreate airflow-webserver airflow-scheduler`\n\n---\n/\n---\n\n# Lakehouse Olist Batch Pipeline (PT-BR)\n\nEste projeto é um pipeline batch ponta a ponta com dados da Olist, usando camadas Bronze/Silver/Gold.\nEle junta scripts Python, models dbt, DuckDB e Airflow em um fluxo local simples de subir e testar.\n\n## Stack\n\n- Python\n- DuckDB\n- dbt (dbt-duckdb)\n- Great Expectations\n- Apache Airflow\n- Docker Compose\n\n## Estrutura do Projeto\n\n```text\nairflow/\n  dags/\ndbt/\n  models/staging/\n  models/marts/\nsrc/\n  ingest/\n  ge/\n  analytics/\ndata/\n  raw/\n  bronze/\n  gold/\n```\n\n## Arquivos Esperados em `data/raw`\n\n- `customers.csv` ou `olist_customers_dataset.csv`\n- `orders.csv` ou `olist_orders_dataset.csv`\n- `order_items.csv` ou `olist_order_items_dataset.csv`\n- `order_payments.csv` ou `olist_order_payments_dataset.csv`\n- `products.csv` ou `olist_products_dataset.csv`\n- `sellers.csv` ou `olist_sellers_dataset.csv`\n- `geolocation.csv` ou `olist_geolocation_dataset.csv`\n\n## Rodar do Zero (Docker + Airflow)\n\n1. Abra o Docker Desktop.\n2. Faça o build das imagens:\n\n```bash\ndocker-compose build dbt airflow-webserver airflow-scheduler\n```\n\n3. Inicialize o banco de metadata do Airflow:\n\n```bash\ndocker-compose run --rm airflow-webserver airflow db init\n```\n\n4. Crie o usuário admin do Airflow:\n\n```bash\ndocker-compose run --rm airflow-webserver airflow users create \\\n  --username admin \\\n  --password admin \\\n  --firstname \"\u003cseu_nome\u003e\" \\\n  --lastname \"\u003cseu_sobrenome\u003e\" \\\n  --role Admin \\\n  --email \"\u003cseu_email\u003e\"\n```\n\n5. Suba os serviços:\n\n```bash\ndocker-compose up -d airflow-webserver airflow-scheduler\n```\n\n6. Abra a UI do Airflow:\n\n- URL: `http://localhost:8080`\n- Usuário: `admin`\n- Senha: `admin`\n\n7. Despause e dispare a DAG principal:\n\n```bash\ndocker-compose exec airflow-scheduler airflow dags unpause lakehouse_olist_pipeline\ndocker-compose exec airflow-scheduler airflow dags trigger lakehouse_olist_pipeline\n```\n\n## Opcional: Rodar só dbt\n\n```bash\ndocker-compose run --rm dbt dbt debug\ndocker-compose run --rm dbt dbt run\ndocker-compose run --rm dbt dbt test\n```\n\n## Saídas do Projeto\n\n- Banco DuckDB: `data/lakehouse.duckdb`\n- Arquivos parquet da bronze: `data/bronze/*.parquet`\n- Relatório HTML: `data/gold/report.html`\n- Resumo de KPI: `data/gold/kpi_summary.json`\n- CSVs de saída: `data/gold/exports/*.csv`\n\n## Ver Resultado no DuckDB UI\n\nAbra o arquivo correto:\n\n```bash\nduckdb -ui data/lakehouse.duckdb\n```\n\nNo editor SQL:\n\n```sql\nSHOW ALL TABLES;\n\nSELECT table_schema, table_name\nFROM information_schema.tables\nWHERE table_type = 'BASE TABLE'\nORDER BY 1,2;\n```\n\n## Comandos Úteis\n\nEntrar no container do Airflow:\n\n```bash\ndocker-compose exec airflow-scheduler bash\n```\n\nDesligar tudo:\n\n```bash\ndocker-compose down\n```\n\nLimpar recursos Docker não usados:\n\n```bash\ndocker system prune -f\n```\n\n## Troubleshooting Rápido\n\n- As DAGs não aparecem na UI\n  - Garanta que o scheduler está rodando:\n  - `docker-compose up -d airflow-scheduler`\n\n- Erro `ModuleNotFoundError: duckdb` nas tasks do Airflow\n  - Faça build e recrie os serviços:\n  - `docker-compose build airflow-webserver airflow-scheduler`\n  - `docker-compose up -d --force-recreate airflow-webserver airflow-scheduler`\n\n- Erro de permissão em `/app/data`\n  - Recrie os serviços com o compose atual:\n  - `docker-compose up -d --force-recreate airflow-webserver airflow-scheduler`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelipemchdev%2Fdata-dbt-batch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffelipemchdev%2Fdata-dbt-batch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelipemchdev%2Fdata-dbt-batch/lists"}