{"id":30886890,"url":"https://github.com/taeefnajib/car-listing-elt-analytics","last_synced_at":"2026-04-11T18:04:05.386Z","repository":{"id":311794199,"uuid":"1045073955","full_name":"taeefnajib/Car-Listing-ELT-Analytics","owner":"taeefnajib","description":"In this project, I've created an ELT pipeline using DLT, Dagster, Postgres and added interactive dashboard using Superset.","archived":false,"fork":false,"pushed_at":"2025-08-26T16:13:20.000Z","size":841,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-26T22:35:54.152Z","etag":null,"topics":["csv","dagster","dashboard","data-analytics","data-engineering","dlt","docker","docker-compose","elt","postgres","superset"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/taeefnajib.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-26T16:04:13.000Z","updated_at":"2025-08-26T16:34:30.000Z","dependencies_parsed_at":"2025-08-26T22:41:15.204Z","dependency_job_id":"5fc90b2c-8bec-4e44-9dfc-bcb51736ac30","html_url":"https://github.com/taeefnajib/Car-Listing-ELT-Analytics","commit_stats":null,"previous_names":["taeefnajib/car-listing-elt-analytics"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/taeefnajib/Car-Listing-ELT-Analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taeefnajib%2FCar-Listing-ELT-Analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taeefnajib%2FCar-Listing-ELT-Analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taeefnajib%2FCar-Listing-ELT-Analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taeefnajib%2FCar-Listing-ELT-Analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/taeefnajib","download_url":"https://codeload.github.com/taeefnajib/Car-Listing-ELT-Analytics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taeefnajib%2FCar-Listing-ELT-Analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274196114,"owners_count":25239148,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-08T02:00:09.813Z","response_time":121,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","dagster","dashboard","data-analytics","data-engineering","dlt","docker","docker-compose","elt","postgres","superset"],"created_at":"2025-09-08T13:52:38.517Z","updated_at":"2025-10-15T14:56:47.516Z","avatar_url":"https://github.com/taeefnajib.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# B2B SaaS Analytics Pipeline\n\nThis project sets up an end-to-end analytics pipeline that loads data from Supabase into Snowflake, transforms it with dbt, and visualizes it in Apache Superset.\n\n## Prerequisites\n\n- Python 3.8+\n- PostgreSQL\n- Supabase account and credentials\n- Snowflake account and credentials\n\n## Setup Instructions\n\n### 1. Set Up Python Environment\n\n```bash\n# Create a virtual environment\npython -m venv venv\n\n# Activate the virtual environment\n# On Windows:\nvenv\\Scripts\\activate\n# On Unix or MacOS:\nsource venv/bin/activate\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### 2. Start PostgreSQL Database in Docker\n\n```bash\n# Start postgres in docker\ndocker compose up --build\n```\n\nMake sure you have a `secrets.toml` file in the `.dlt` folder with the following structure:\n\n\n### 3. Set Up Dagster\n\n```bash\n# Start Dagster webserver\ndagster dev\n```\n\nThe Dagster UI will be available at http://localhost:3000\n\n### 4. Run the Pipeline\n\nIn the Dagster UI:\n1. Navigate to the Assets tab\n2. Select all assets\n3. Click \"Materialize Selected\"\n\nThis will:\n- Load data from CSV to PostgreSQL database using dlt\n- Run dbt transformations on the loaded data\n\n### 6. Set Up Apache Superset\n\n```bash\n# Change directory to superset folder\ncd superset\n\n# Make scripts executable\nchmod +x setup_db.sh run_superset.sh\n\n# Set up the database\n./setup_db.sh\n\n# Start Superset\n./run_superset.sh\n```\n\nWhen running `run_superset.sh`, you'll be prompted to create an admin user. Follow the prompts to set up your credentials.\n\nSuperset will be available at http://localhost:8088\n\n### 7. Create Dashboard in Superset\n\n1. Log in to Superset using your admin credentials\n2. Go to Data → Databases and add your PostgreSQL connection\n3. Create new datasets from your analytics table\n4. Create charts using these datasets\n5. Combine charts into a dashboard\n\n## Troubleshooting\n\n- If you encounter database connection issues, verify your credentials in `secrets.toml` and `profiles.yml`\n- For Superset connection issues, check `superset_config.py` settings\n- Make sure all required ports are available and not blocked by firewall\n\n## Security Note\n\nNever commit files containing credentials (`secrets.toml`, `profiles.yml`, `superset_config.py`) to version control. Add them to your `.gitignore` file. For demonstration purpose, we have unblocked some of these files.\n\n## Additional Resources\n\n- [dlt Documentation](https://dlthub.com/docs)\n- [dbt Documentation](https://docs.getdbt.com)\n- [Apache Superset Documentation](https://superset.apache.org/docs/intro)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaeefnajib%2Fcar-listing-elt-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftaeefnajib%2Fcar-listing-elt-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaeefnajib%2Fcar-listing-elt-analytics/lists"}