{"id":22733728,"url":"https://github.com/ndiplacide7/air_quality_monitor","last_synced_at":"2026-02-25T19:11:37.941Z","repository":{"id":266455807,"uuid":"898389220","full_name":"ndiplacide7/air_quality_monitor","owner":"ndiplacide7","description":"Real-Time Air Quality Monitoring System using Django, Apache Hadoop, Apache Kafka, and AWS services.","archived":false,"fork":false,"pushed_at":"2024-12-11T07:41:16.000Z","size":141,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-26T16:41:42.428Z","etag":null,"topics":["apache-kafka","aws","css","django","hadoop","html","jupiter-notebook","mysql","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ndiplacide7.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-04T10:00:58.000Z","updated_at":"2024-12-11T07:41:21.000Z","dependencies_parsed_at":"2024-12-04T11:25:28.412Z","dependency_job_id":"48d93c16-8e1f-4e49-9f05-cc6f0972ae79","html_url":"https://github.com/ndiplacide7/air_quality_monitor","commit_stats":null,"previous_names":["ndiplacide7/air_quality_monitor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ndiplacide7/air_quality_monitor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndiplacide7%2Fair_quality_monitor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndiplacide7%2Fair_quality_monitor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndiplacide7%2Fair_quality_monitor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndiplacide7%2Fair_quality_monitor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ndiplacide7","download_url":"https://codeload.github.com/ndiplacide7/air_quality_monitor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndiplacide7%2Fair_quality_monitor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29835533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-25T19:08:47.527Z","status":"ssl_error","status_checked_at":"2026-02-25T18:59:04.705Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","aws","css","django","hadoop","html","jupiter-notebook","mysql","python3"],"created_at":"2024-12-10T20:15:50.448Z","updated_at":"2026-02-25T19:11:37.912Z","avatar_url":"https://github.com/ndiplacide7.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Air Quality Monitor: Comprehensive Environmental Data Solution\n\n### Team Contributions\n\n```markdown\n## GROUP NO : 7\n  ------------------------------------------------\n  | No. | Name              | Registration Number |\n  |-----|-------------------|---------------------|\n  | 1   | NDUWAYEZU Placide | 223027936           |\n  | 2   | UWASE Aline       | 218009283           |\n  | 3   | MUREMYI Samuel    | 223026694           |\n  -------------------------------------------------\n\n```\n\n## Project Case Study\n### Background and Motivation\nIn an era of increasing environmental concerns, real-time air quality monitoring has become crucial for public health and environmental policy. The Air Quality Monitor project aims to create a robust, scalable solution for tracking and analyzing air quality data.\n### Project Objectives\n\n1. **Real-time Data Collection**: Develop a system to continuously fetch air quality data from official sources.\n2. **Data Processing Pipeline**: Create an efficient mechanism to transform raw API data into meaningful insights.\n3. **Distributed Storage**: Implement a scalable storage solution using HDFS and cloud databases.\n4. **Data Visualization**: Build an interactive dashboard for accessible environmental insights.\n\n## Project Architecture\n### Technical Architecture\n\n- **Data Ingestion**: Retrieve data from ACT Government Air Quality API\n- **Stream Processing**: Apache Kafka for real-time data streaming\n- **Data Storage**:\n    - Distributed Storage: Apache Hadoop HDFS\n    - Persistent Storage: AWS MySQL RDS\n- **Web Framework**: Django\n- **Data Processing**: Pandas, PyArrow \n\n## System Design Diagram\n\n```bash \n\n[API Source] → [Kafka Stream] → [Data Processing] → [HDFS Storage] → [MySQL RDS] → [Django Dashboard]\n\n```\n\n## Challenges and Solutions\n\n1. **Dataset Availability**\n- **Challenge**: Lack of Accessible Air Quality Datasets for Rwanda\n    - Initial project goal was to develop an air quality monitoring system for Rwanda\n    - Significant obstacles encountered in obtaining comprehensive, reliable air quality data\n    - Limited public APIs and open data sources for environmental monitoring in Rwanda\n- **Solution**:\n    - Utilized ACT's robust air quality monitoring system as a proof-of-concept model\n\n2. **Local Infrastructure set-up Complexity**: Kafka and Hadoop Setup\n- **Challenge**: Overcoming Windows-Specific Installation Barriers\n    - Significant complexity in natively installing Kafka and Hadoop on Windows\n    - Multiple compatibility and configuration issues with distributed systems\n- **Solution**:\n  - Comprehensive Docker-Based Solution: Leveraged Docker containers to create a consistent, reproducible development environment\n  - ![img.png](img.png)\n\n### Team Contributions\n\n```markdown\n\n------------------------------------------------\n| No. | Name              | Registration Number |\n|-----|-------------------|---------------------|\n| 1   | NDUWAYEZU Placide | 223027936           |\n| 2   | UWASE Aline       | 218009283           |\n| 3   | MUREMYI Samuel    | 223026694           |\n-------------------------------------------------\n\n```\n\n### Data Source\n\nThe project retrieves real-time air quality data from the ACT (Australian Capital Territory) Ambient Air Quality Monitoring API: https://www.data.act.gov.au/resource/94a5-zqnn.json\n\n### Key Features\n  - Real-time data retrieval from official air quality API\n  - Data processing pipeline\n  - HDFS storage using Docker\n  - AWS MySQL RDS data persistence\n  - Interactive dashboard for air quality visualization\n\n### Technology Stack\n- Backend: Django\n- Data Processing:\n  - Pandas\n  - PyArrow\n- Data Storage:\n  - Apache Hadoop (HDFS)\n  - AWS MySQL RDS\n\n- Message Streaming: Apache Kafka\n- Additional Libraries:\n  - Requests\n  - Confluent Kafka\n  - Pytz\n\n### Prerequisites\n- Python 3.8+\n- Docker (optional, for HDFS)\n- AWS RDS MySQL instance\n- Apache Kafka\n- Apache Hadoop\n\n### Installation\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/your-username/air_quality_monitor.git\ncd air_quality_monitor\n```\n2. Create a virtual environment\n```bash\npython -m venv venv\nsource venv/bin/activate  # On Windows use `venv\\Scripts\\activate`\n```\n3. Install dependencies:\n```bash\npip install -r requirements.txt\n```\n4. Configure Database:\n- Set up your AWS MySQL RDS credentials in settings.py\n- Configure Kafka and Hadoop connection details\n\n5. Run Database Migrations:\n```bash\npython manage.py makemigrations\npython manage.py migrate\n```\n6. Create Superuser (Optional):\n```bash \npython manage.py createsuperuser\n```\n### Running the Application\n\n#### Start Data Pipeline (Manual Trigger)\n```bash\npython manage.py run_air_quality_pipeline\n```\n#### Run Django Development Server (On preferred port)\n```bash\npython manage.py runserver 9006\n```\n#### Docker HDFS Integration\nTo copy processed data to HDFS:\n```bash\ndocker cp \u003clocal_parquet_file\u003e \u003chadoop_container\u003e:/path/in/hdfs\n```\n\n### Configuration\n\nEnsure the following configurations are set:\n- API Endpoint\n- Kafka Bootstrap Servers\n- HDFS Connection\n- AWS RDS Credentials\n\n### Data Flow\n1. Fetch data from ACT Air Quality API\n2. Process raw data\n3. Generate unique Parquet filename\n4. Convert to PyArrow table\n5. Save to HDFS\n6. Persist in AWS MySQL RDS\n7. Visualize in Dashboard\n\n#### Environment Variables\n\nCreate a .env file for sensitive data (like Credential):\n```bash\nAPI_ENDPOINT=https://www.data.act.gov.au/resource/94a5-zqnn.json\nKAFKA_BOOTSTRAP_SERVERS=your-kafka-servers\nHDFS_HOST=your-hdfs-host\nAWS_RDS_HOST=your-rds-endpoint\nAWS_RDS_USER=your-username\nAWS_RDS_PASSWORD=your-password\n```\n### Troubleshooting\n\n- Ensure all services (Kafka, Hadoop, RDS) are running\n- Check network connectivity \n- Verify API access \n- Review logs for detailed error information\n\n### Contributing\n\n- Fork the repository\n- Create your feature branch (git checkout -b feature/AmazingFeature)\n- Commit your changes (git commit -m 'Add some AmazingFeature')\n- Push to the branch (git push origin feature/AmazingFeature)\n- Open a Pull Request\n\n### License\n- MIT\n\n### Acknowledgements\n\n- ACT Government for Open Data \n- Open Source Community\n\n### Contact\n- Placide - ndiplacide7@gailcom\n- Project Link: https://github.com/ndiplacide7/air_quality_monitor\n  \u003cimg src=\"https://t.bkit.co/w_675941ba62b4a.gif\" /\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndiplacide7%2Fair_quality_monitor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fndiplacide7%2Fair_quality_monitor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndiplacide7%2Fair_quality_monitor/lists"}