{"id":29178353,"url":"https://github.com/pragmaai/yelp-datapipeline","last_synced_at":"2026-05-04T01:33:23.864Z","repository":{"id":301722431,"uuid":"1010118514","full_name":"PragmaAI/yelp-datapipeline","owner":"PragmaAI","description":"🍽️ Yelp Data Pipeline \u0026 Analytics Dashboard  End-to-end data engineering pipeline processing Yelp dataset with Rust transforms, Apache Airflow orchestration, and interactive Streamlit analytics. Features business insights, user engagement analysis, and city performance comparisons.  🚀 Docker-ready • 📊 Interactive Dashboard • ⚡ High-performance R","archived":false,"fork":false,"pushed_at":"2025-06-28T11:43:36.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-28T12:38:05.171Z","etag":null,"topics":["airflow","data-engineering","data-pipeline","data-visualization","datafusion","docker","rust","streamlit","yelp","yelp-dataset"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PragmaAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-28T11:38:41.000Z","updated_at":"2025-06-28T11:48:46.000Z","dependencies_parsed_at":"2025-06-28T12:38:10.886Z","dependency_job_id":"749ab292-f055-48f3-800b-11d50bc149e7","html_url":"https://github.com/PragmaAI/yelp-datapipeline","commit_stats":null,"previous_names":["pragmaai/yelp-datapipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/PragmaAI/yelp-datapipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PragmaAI%2Fyelp-datapipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PragmaAI%2Fyelp-datapipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PragmaAI%2Fyelp-datapipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PragmaAI%2Fyelp-datapipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PragmaAI","download_url":"https://codeload.github.com/PragmaAI/yelp-datapipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PragmaAI%2Fyelp-datapipeline/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263019621,"owners_count":23400934,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","data-engineering","data-pipeline","data-visualization","datafusion","docker","rust","streamlit","yelp","yelp-dataset"],"created_at":"2025-07-01T18:46:07.976Z","updated_at":"2026-05-04T01:33:23.799Z","avatar_url":"https://github.com/PragmaAI.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🍽️ Yelp Data Pipeline \u0026 Analytics Dashboard\n\nA comprehensive data engineering pipeline that processes Yelp dataset JSON files, transforms them using Rust for high performance, orchestrates workflows with Apache Airflow, and provides interactive analytics through a Streamlit dashboard.\n\n## 🎯 Project Overview\n\nThis project demonstrates a modern data engineering stack for processing and analyzing Yelp business data:\n\n- **Data Ingestion**: JSON to Parquet conversion for efficient storage\n- **Data Transformation**: High-performance Rust-based data processing\n- **Workflow Orchestration**: Apache Airflow DAGs for reliable pipeline execution\n- **Data Visualization**: Interactive Streamlit dashboard for business insights\n- **Containerization**: Docker Compose for easy deployment\n\n## 🏗️ Architecture\n\n```\n┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐\n│   Raw JSON      │    │   Parquet       │    │   Analytics     │\n│   Data Files    │───▶│   Conversion    │───▶│   Dashboard     │\n└─────────────────┘    └─────────────────┘    └─────────────────┘\n                              │\n                              ▼\n                       ┌─────────────────┐\n                       │   Rust Data     │\n                       │   Transform     │\n                       └─────────────────┘\n                              │\n                              ▼\n                       ┌─────────────────┐\n                       │   Apache        │\n                       │   Airflow       │\n                       └─────────────────┘\n```\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Docker and Docker Compose\n- Python 3.8+\n- Rust (for local development)\n- Git\n\n### 1. Clone the Repository\n\n```bash\ngit clone https://github.com/PragmaAI/yelp-datapipeline.git\ncd yelp-datapipeline\n```\n\n### 2. Prepare Your Data\n\nPlace your Yelp dataset JSON files in the `data/raw/` directory:\n\n```\ndata/\n├── raw/\n│   ├── business.json\n│   ├── review.json\n│   ├── user.json\n│   └── tip.json\n└── processed/\n    └── (will be created automatically)\n```\n\n## 🐳 Running with Docker Compose\n\n### Start Airflow\n\n```bash\n# Start Airflow services\ndocker-compose up -d\n\n# Access Airflow UI\nopen http://localhost:8080\n# Default credentials: airflow/airflow\n```\n\n### Run the Data Pipeline\n\n1. **Navigate to Airflow UI**: http://localhost:8080\n2. **Enable DAGs**: Click the toggle switch next to each DAG\n3. **Trigger DAGs** in this order:\n   - `json_to_parquet_dag` - Converts JSON to Parquet\n   - `rust_transform_dag` - Runs Rust data transformations\n   - `yelp_rolling_etl` - Performs rolling ETL operations\n\n### Monitor Pipeline Execution\n\n- **DAGs Tab**: View all available workflows\n- **Graph View**: Visualize DAG dependencies\n- **Logs**: Check task execution logs\n- **XCom**: View data passed between tasks\n\n## 📊 Streamlit Analytics Dashboard\n\n### Start the Dashboard\n\n```bash\n# Navigate to streamlit app directory\ncd streamlit_app\n\n# Install dependencies\npip install -r requirements.txt\n\n# Run the dashboard\n./run_app.sh\n# or manually:\nstreamlit run app.py --server.port 8501 --server.address 0.0.0.0\n```\n\n### Access the Dashboard\n\nOpen your browser and navigate to: **http://localhost:8501**\n\n## 🎨 Dashboard Features\n\n### 📈 Dashboard Overview\n- **Key Metrics**: Business counts, user engagement, elite users\n- **Business Performance**: City-wise comparison charts\n- **User Engagement**: Distribution analysis\n- **Top Performers**: Best-rated businesses and active users\n\n### 🏢 Business Analytics\n- **Interactive Filtering**: Filter by city, category, and rating\n- **Performance Metrics**: Rating distribution, review analysis\n- **Category Insights**: Business category performance\n- **City Comparison**: Cross-city business analysis\n\n### 👥 User Analytics\n- **User Engagement**: Activity patterns and user categories\n- **Elite Users**: Analysis of elite user characteristics\n- **Sentiment Analysis**: User sentiment patterns\n- **User Compliments**: Recognition and engagement metrics\n- **Activity Timeline**: User activity over time\n\n## 🔧 Manual Development Setup\n\n### Local Airflow Setup\n\n```bash\n# Install Airflow\npip install apache-airflow\n\n# Initialize Airflow database\nairflow db init\n\n# Create admin user\nairflow users create \\\n    --username admin \\\n    --firstname Admin \\\n    --lastname User \\\n    --role Admin \\\n    --email admin@example.com \\\n    --password admin\n\n# Start Airflow webserver\nairflow webserver --port 8080\n\n# Start Airflow scheduler (in another terminal)\nairflow scheduler\n```\n\n### Rust Development\n\n```bash\n# Navigate to Rust project\ncd scripts/transform\n\n# Build the project\ncargo build --release\n\n# Run tests\ncargo test\n\n# Run the transform\ncargo run --release\n```\n\n### Python Dependencies\n\n```bash\n# Install Python dependencies\npip install -r requirements.txt\n\n# For development\npip install -r requirements-dev.txt  # if available\n```\n\n## 📁 Project Structure\n\n```\nyelp-datapipeline/\n├── airflow/                 # Airflow Docker configuration\n│   ├── Dockerfile\n│   └── entrypoint.sh\n├── dags/                    # Airflow DAGs\n│   ├── json_to_parquet_dag.py\n│   ├── rust_transform_dag.py\n│   ├── yelp_rolling_etl.py\n│   └── README.md\n├── data/                    # Data storage\n│   ├── raw/                 # Raw JSON files\n│   └── processed/           # Processed Parquet files\n├── notebooks/               # Jupyter notebooks\n│   └── analysis.ipynb\n├── scripts/                 # Data processing scripts\n│   ├── json_to_parquet.py   # Python JSON converter\n│   └── transform/           # Rust data transformer\n│       ├── Cargo.toml\n│       └── src/main.rs\n├── streamlit_app/           # Streamlit dashboard\n│   ├── app.py\n│   ├── requirements.txt\n│   ├── run_app.sh\n│   └── README.md\n├── docker-compose.yml       # Docker services\n├── requirements.txt         # Python dependencies\n├── run_pipeline.sh          # Pipeline runner\n└── start_airflow.sh         # Airflow starter\n```\n\n## 🔄 Data Pipeline Flow\n\n### 1. Data Ingestion\n- **Input**: Yelp JSON files (business, review, user, tip)\n- **Process**: Convert to Parquet format for efficient storage\n- **Output**: Parquet files in `data/processed/`\n\n### 2. Data Transformation\n- **Input**: Parquet files from ingestion\n- **Process**: Rust-based transformations for high performance\n- **Output**: Enhanced analytics datasets\n\n### 3. Analytics Processing\n- **Input**: Transformed data\n- **Process**: Generate business insights, user analytics, city comparisons\n- **Output**: Analytics-ready datasets for dashboard\n\n### 4. Visualization\n- **Input**: Analytics datasets\n- **Process**: Streamlit dashboard rendering\n- **Output**: Interactive web interface\n\n## 📊 Key Analytics Features\n\n### Business Insights\n- Top-performing businesses by city\n- Rating distribution analysis\n- Category performance comparison\n- Review sentiment analysis\n\n### User Analytics\n- User engagement patterns\n- Elite user characteristics\n- User sentiment analysis\n- Activity timeline tracking\n\n### City Performance\n- Cross-city business comparison\n- Rating tier analysis\n- Review volume analysis\n- Business density metrics\n\n## 🛠️ Configuration\n\n### Environment Variables\n\nCreate a `.env` file for custom configuration:\n\n```bash\n# Airflow Configuration\nAIRFLOW_UID=50000\nAIRFLOW_GID=0\n\n# Database Configuration\nPOSTGRES_USER=airflow\nPOSTGRES_PASSWORD=airflow\nPOSTGRES_DB=airflow\n\n# Data Paths\nDATA_RAW_PATH=./data/raw\nDATA_PROCESSED_PATH=./data/processed\n```\n\n### Docker Configuration\n\nThe `docker-compose.yml` includes:\n- **Airflow Webserver**: Web UI for DAG management\n- **Airflow Scheduler**: Executes DAGs\n- **PostgreSQL**: Metadata database\n- **Redis**: Celery backend (if using distributed execution)\n\n## 🔍 Troubleshooting\n\n### Common Issues\n\n1. **Port Conflicts**\n   ```bash\n   # Check if ports are in use\n   lsof -i :8080  # Airflow\n   lsof -i :8501  # Streamlit\n   ```\n\n2. **Permission Issues**\n   ```bash\n   # Fix file permissions\n   sudo chown -R $USER:$USER data/\n   chmod +x run_pipeline.sh start_airflow.sh\n   ```\n\n3. **Docker Issues**\n   ```bash\n   # Clean up Docker\n   docker-compose down -v\n   docker system prune -f\n   ```\n\n4. **Data Loading Errors**\n   - Ensure JSON files are in `data/raw/`\n   - Check file permissions\n   - Verify JSON format is valid\n\n### Logs and Debugging\n\n```bash\n# Airflow logs\ndocker-compose logs airflow-webserver\ndocker-compose logs airflow-scheduler\n\n# Streamlit logs\nstreamlit run app.py --logger.level debug\n```\n\n## 📈 Performance Optimization\n\n### Rust Transformations\n- **Parallel Processing**: Multi-threaded data processing\n- **Memory Efficiency**: Optimized for large datasets\n- **Type Safety**: Compile-time error checking\n\n### Data Storage\n- **Parquet Format**: Columnar storage for fast queries\n- **Compression**: Efficient data compression\n- **Partitioning**: Optimized data partitioning\n\n### Dashboard Performance\n- **Caching**: Streamlit caching for faster loading\n- **Lazy Loading**: Load data on demand\n- **Optimized Queries**: Efficient data filtering\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- **Yelp Dataset**: For providing the open dataset\n- **Apache Airflow**: For workflow orchestration\n- **Rust**: For high-performance data processing\n- **Streamlit**: For interactive data visualization\n\n## 📞 Support\n\nFor questions and support:\n- Create an issue on GitHub\n- Check the documentation in each component directory\n- Review the troubleshooting section above\n\n---\n\n**Happy Data Engineering! 🚀** ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpragmaai%2Fyelp-datapipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpragmaai%2Fyelp-datapipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpragmaai%2Fyelp-datapipeline/lists"}