{"id":26167435,"url":"https://github.com/annenkovdmitrii/pi-data-pipeline","last_synced_at":"2026-04-12T14:43:44.634Z","repository":{"id":281747790,"uuid":"944630204","full_name":"annenkovdmitrii/pi-data-pipeline","owner":"annenkovdmitrii","description":"A distributed system for collecting and visualizing environmental \u0026 sensor data.","archived":false,"fork":false,"pushed_at":"2025-03-10T23:21:51.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-11T00:24:45.568Z","etag":null,"topics":["docker","postgresql","python","streamlit"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/annenkovdmitrii.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-07T17:28:11.000Z","updated_at":"2025-03-10T23:21:54.000Z","dependencies_parsed_at":"2025-03-11T00:24:58.949Z","dependency_job_id":"51f6ef38-badc-42b4-8a6f-718fda9d3887","html_url":"https://github.com/annenkovdmitrii/pi-data-pipeline","commit_stats":null,"previous_names":["annenkovdmitrii/pi-data-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/annenkovdmitrii%2Fpi-data-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/annenkovdmitrii%2Fpi-data-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/annenkovdmitrii%2Fpi-data-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/annenkovdmitrii%2Fpi-data-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/annenkovdmitrii","download_url":"https://codeload.github.com/annenkovdmitrii/pi-data-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243079762,"owners_count":20233051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","postgresql","python","streamlit"],"created_at":"2025-03-11T17:36:26.061Z","updated_at":"2025-12-16T12:51:26.640Z","avatar_url":"https://github.com/annenkovdmitrii.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Raspberry Pi Data Pipeline\n\nA distributed system for collecting and visualizing environmental sensor data.\n\n## Project Overview\n\n### Core Components\n| Service | Description | Port | Key Files |\n|---------|-------------|------|-----------|\n| **Sensor Collector** (`sensor_collector/`) | Collects Sense HAT metrics (temperature, humidity, pressure) | - | `Dockerfile`, `sensor_collector_host.py`, `sensor-test.py` |\n| **Weather Collector** (`weather_collector/`) | Fetches weather data from WeatherAPI.com | - | `Dockerfile`, `weather_collector.py`, `requirements.txt` |\n| **PostgreSQL** (`database/`) | Time-series data storage | 5432 | `init.sql` |\n| **Dashboard** (`dashboard/`) | Streamlit visualization interface | 8501 | `app.py`, `Dockerfile` |\n\n## Key Features\n- Real-time sensor data collection from Raspberry Pi Sense HAT\n- Environmental weather data integration via WeatherAPI.com\n- Interactive Streamlit dashboard with data comparison\n- Docker-based deployment with compose orchestration\n- PostgreSQL database with optimized sensor data schema\n\n## Getting Started\n\n### Prerequisites\n- Docker Engine 20.10+\n- Docker Compose 2.20+\n- Python 3.9+\n- WeatherAPI.com account (free tier)\n\n### Installation\n1. Clone repository:\n```bash\ngit clone https://github.com/your-username/raspberry-pi-data-pipeline.git\ncd raspberry-pi-data-pipeline\n```\n\n2. Configure environment:\n```bash\ncp .env.example .env\nnano .env  # Add your WeatherAPI key (WEATHER_API_KEY)\n```\n\n3. Start services:\n```bash\ndocker-compose up --build -d\n```\n\n## Operational Guide\n\n### Service Management\n| Command | Description |\n|---------|-------------|\n| `docker-compose up -d` | Start all services |\n| `docker-compose logs -f` | View live service logs |\n| `docker-compose exec sensor-collector bash` | Access sensor container |\n| `docker-compose down -v` | Stop and remove all containers |\n\n### Data Collection\n- Sensor data: Collected every 30 seconds via Sense HAT\n- Weather data: Collected every 5 minutes from WeatherAPI.com\n- Data retention: Configured via PostgreSQL table partitioning\n\n### Dashboard Features\n- Real-time sensor vs weather data comparison\n- Historical data visualization (1h/24h/7d)\n- Air quality index (AQI) monitoring\n- Raw data inspection tables\n\n## Database Schema Overview\n```mermaid\nerDiagram\n    sensor_readings {\n        TIMESTAMP timestamp\n        FLOAT temperature\n        FLOAT humidity\n        FLOAT pressure\n    }\n    \n    weather_api_data {\n        TIMESTAMP timestamp\n        FLOAT temperature\n        FLOAT humidity\n        FLOAT pressure\n        TEXT condition\n        FLOAT wind_speed\n        TEXT wind_direction\n        TEXT location\n        FLOAT aqi\n        FLOAT pm2_5\n        FLOAT pm10\n    }\n```\n\n## Development Setup\n\n### Testing Sensor Integration\n```bash\ndocker-compose exec sensor-collector python sensor-test.py\n```\n\n### Monitoring Data Flow\n```bash\n# View sensor data stream\ndocker-compose exec postgres psql -U postgres -d sensordata -c \"SELECT * FROM sensor_readings ORDER BY timestamp DESC LIMIT 10;\"\n\n# View weather data\ndocker-compose exec postgres psql -U postgres -d sensordata -c \"SELECT timestamp,temperature,humidity FROM weather_api_data ORDER BY timestamp DESC LIMIT 5;\"\n```\n\n### Debugging Tips\n1. Verify Sense HAT detection:\n```bash\ndocker-compose exec sensor-collector python -c \"from sense_hat import SenseHat; print(SenseHat().get_temperature())\"\n```\n\n2. Test weather API connection:\n```bash\ndocker-compose exec weather-collector python -c \"import os; print('API Key:', os.getenv('WEATHER_API_KEY'))\"\n```\n\n## Configuration Reference\n\n### Environment Variables\n| Variable | Service | Description |\n|----------|---------|-------------|\n| `WEATHER_API_KEY` | Weather Collector | WeatherAPI.com authentication key |\n| `WEATHER_CITY` | Weather Collector | Location for weather data collection |\n| `DB_*` | All | PostgreSQL connection parameters |\n\n### Port Mapping\n| Service | Host Port | Container Port |\n|---------|-----------|----------------|\n| Postgres | 5432 | 5432 |\n| Dashboard | 8501 | 8501 |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fannenkovdmitrii%2Fpi-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fannenkovdmitrii%2Fpi-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fannenkovdmitrii%2Fpi-data-pipeline/lists"}