{"id":28907597,"url":"https://github.com/sculpttechproject/servosense","last_synced_at":"2026-04-15T05:32:21.987Z","repository":{"id":300058545,"uuid":"1004995489","full_name":"SculptTechProject/ServoSense","owner":"SculptTechProject","description":"ServoSense — an end-to-end IoT sensor data pipeline: FastAPI simulator → Kafka → batch \u0026 streaming analytics (Pandas/Spark) → Delta Lake.","archived":false,"fork":false,"pushed_at":"2025-06-19T17:05:39.000Z","size":14,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-19T17:42:43.563Z","etag":null,"topics":["delta-lake","fastapi","iot","kafka","pandas","spark","streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SculptTechProject.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-19T13:53:02.000Z","updated_at":"2025-06-19T17:05:42.000Z","dependencies_parsed_at":"2025-06-19T17:43:25.491Z","dependency_job_id":null,"html_url":"https://github.com/SculptTechProject/ServoSense","commit_stats":null,"previous_names":["sculpttechproject/servosense"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SculptTechProject/ServoSense","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SculptTechProject%2FServoSense","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SculptTechProject%2FServoSense/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SculptTechProject%2FServoSense/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SculptTechProject%2FServoSense/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SculptTechProject","download_url":"https://codeload.github.com/SculptTechProject/ServoSense/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SculptTechProject%2FServoSense/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261153565,"owners_count":23116913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["delta-lake","fastapi","iot","kafka","pandas","spark","streaming"],"created_at":"2025-06-21T16:03:41.430Z","updated_at":"2025-10-14T07:12:48.423Z","avatar_url":"https://github.com/SculptTechProject.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ServoSense\n\n### Please check pinned project - Servo Sense App on my GitHub account! :)\n\n**ServoSense** is an end-to-end pipeline for processing and analyzing industrial machine sensor data. It integrates data generation, ingestion, streaming, batch processing, EDA, modeling, and monitoring in one cohesive toolkit.\n\n[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)](https://www.python.org/downloads/)\n[![Docker](https://img.shields.io/badge/docker-compose-blue)](https://docs.docker.com/compose/)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)\n\n## 🚀 Features\n\n* **Infrastructure**: Docker Compose setup with Kafka, Zookeeper, Prometheus, and Grafana.\n* **Serving API**: FastAPI service for sensor data CRUD, simulation, prediction, and health checks.\n* **Simulator**: Synthetic data generator (\\~10 Hz) with Prometheus metrics.\n* **Streaming**: PySpark Structured Streaming for real-time processing.\n* **Batch**: Python scripts with Pandas for CSV logging and rolling stats.\n* **EDA**: Jupyter notebooks for exploratory analysis and visualization.\n* **Models**: Train and serve Random Forest predictive-maintenance models.\n* **Monitoring**: Preconfigured Prometheus \u0026 Grafana dashboards and alerts.\n\n## 📋 Table of Contents\n\n1. [Prerequisites](#prerequisites)\n2. [Installation](#installation)\n3. [Quickstart](#quickstart)\n4. [API Endpoints](#api-endpoints)\n5. [Project Structure](#project-structure)\n6. [Contributing](#contributing)\n7. [License](#license)\n8. [Contact](#contact)\n\n---\n\n## 🛠️ Prerequisites\n\n* Python **3.9** or newer\n* Docker \u0026 Docker Compose\n* Java (for Spark)\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate  # Windows: .\\.venv\\Scripts\\activate\npip install --upgrade pip\npip install -r requirements.txt\n```\n\n\u003e ⚠️ Ensure `scikit-learn` is listed in `requirements.txt` before building Docker images.\n\n---\n\n## ⚙️ Installation\n\n```bash\ngit clone https://github.com/SculptTechProject/ServoSense.git\ncd ServoSense\n# Activate environment \u0026 install dependencies\n```\n\nBuild the simulator image:\n\n```bash\ndocker build -t servo-simulator:latest -f simulator/Dockerfile .\n```\n\nBring up core services:\n\n```bash\ndocker-compose -f infra/docker-compose.yml up -d\n```\n\n---\n\n## 🏃 Quickstart\n\n1. **Start Kafka \u0026 Zookeeper**\n\n   ```bash\n   ```\n\ndocker-compose -f infra/docker-compose.yml up -d kafka zookeeper\n\n````\n\n2. **Run the API Server**\n\n   ```bash\ncd serving\nuvicorn app:app --reload --port 8000\n````\n\n3. **(Optional) Launch Simulator**\n\n   ```bash\n   ```\n\ncd simulator\nuvicorn main\\:app --reload --port 8001\n\n````\n\n4. **Run Streaming Job**\n\n   ```bash\ncd streaming\nspark-submit \\\n  --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.1 \\\n  stream_job.py\n````\n\n5. **Batch Processing**\n\n   ```bash\n   ```\n\ncd batch\npython batch\\_job.py\n\n````\n\n6. **Open EDA Notebooks** in `Data_Analysis/`\n\n7. **Train Model** in `models/train_model.ipynb`\n\n8. **Start Monitoring**\n\n   ```bash\ndocker-compose -f infra/docker-compose.yml up -d prometheus grafana\n````\n\n* Prometheus: [http://localhost:9090](http://localhost:9090)\n* Grafana: [http://localhost:3000](http://localhost:3000) (admin/admin)\n\n---\n\n## 🔗 API Endpoints\n\nFastAPI serving endpoints:\n\n| Method | Path           | Description                                                                                   |\n| ------ | -------------- | --------------------------------------------------------------------------------------------- |\n| GET    | `/`            | Health check, returns `{ \"message\": \"Server is working!\" }`.                                  |\n| POST   | `/sensor`      | Add a new sensor reading (JSON body).                                                         |\n| GET    | `/sensor`      | Retrieve all stored sensor readings.                                                          |\n| GET    | `/simulate`    | Generate and return a single simulated reading.                                               |\n| GET    | `/predict_all` | Run model predictions on all CSV readings, returns list with `is_hot` flag and `probability`. |\n| GET    | `/metrics`     | Prometheus metrics endpoint for simulation gauges.                                            |\n| GET    | `/health`      | Service health endpoint, returns `{ \"status\": \"ok\" }`.                                        |\n\n---\n\n## 📂 Project Structure\n\n```\nServoSense/\n├── infra/            # Docker Compose for infra\n├── serving/          # FastAPI ingestion \u0026 prediction API\n├── simulator/        # Synthetic data generator\n├── streaming/        # PySpark streaming job\n├── batch/            # Batch processing scripts\n├── Data_Analysis/    # EDA notebooks\n├── models/           # ML training notebook \u0026 saved model\n├── monitoring/       # Prometheus \u0026 Grafana configs\n├── requirements.txt\n└── README.md\n```\n\n---\n\n## 🤝 Contributing\n\n1. Fork the repo\n2. Create a feature branch (`git checkout -b feat/my-feature`)\n3. Commit changes (`git commit -m \"Add feature\"`)\n4. Push to branch (`git push origin feat/my-feature`)\n5. Open a Pull Request\n\nPlease follow our [Code of Conduct](CODE_OF_CONDUCT.md).\n\n---\n\n## 📄 License\n\nThis project is licensed under the MIT License.\n\n---\n\n## 📬 Contact\n\nQuestions or feedback? Open an issue or ping us on GitHub:\n\n```\ngithub.com/SculptTechProject/ServoSense\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsculpttechproject%2Fservosense","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsculpttechproject%2Fservosense","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsculpttechproject%2Fservosense/lists"}