{"id":35762013,"url":"https://github.com/pyjeebz/helios","last_synced_at":"2026-01-13T20:47:12.023Z","repository":{"id":331220231,"uuid":"1125593384","full_name":"pyjeebz/helios","owner":"pyjeebz","description":"Predict","archived":false,"fork":false,"pushed_at":"2026-01-02T21:52:58.000Z","size":911,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-04T14:56:27.097Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pyjeebz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-31T02:22:59.000Z","updated_at":"2026-01-03T07:45:48.000Z","dependencies_parsed_at":"2026-01-05T02:10:34.245Z","dependency_job_id":null,"html_url":"https://github.com/pyjeebz/helios","commit_stats":null,"previous_names":["pyjeebz/helios"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/pyjeebz/helios","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyjeebz%2Fhelios","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyjeebz%2Fhelios/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyjeebz%2Fhelios/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyjeebz%2Fhelios/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pyjeebz","download_url":"https://codeload.github.com/pyjeebz/helios/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyjeebz%2Fhelios/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28400109,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T14:36:09.778Z","status":"ssl_error","status_checked_at":"2026-01-13T14:35:19.697Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-06T23:22:28.725Z","updated_at":"2026-01-13T20:47:12.017Z","avatar_url":"https://github.com/pyjeebz.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Helios\n\n**Predictive Infrastructure Intelligence Platform**\n\n[![CI](https://github.com/pyjeebz/helios/actions/workflows/ci.yml/badge.svg)](https://github.com/pyjeebz/helios/actions/workflows/ci.yml)\n[![Release](https://github.com/pyjeebz/helios/actions/workflows/release.yml/badge.svg)](https://github.com/pyjeebz/helios/actions/workflows/release.yml)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\nHelios uses machine learning to forecast infrastructure demand and provide proactive scaling recommendations, reducing reactive incidents and optimizing resource utilization.\n\n## 🚀 Quick Deploy\n\n**One-click deployment options:**\n\n[![Deploy to Render](https://render.com/images/deploy-to-render-button.svg)](https://render.com/deploy?repo=https://github.com/pyjeebz/helios)\n\n**Docker:**\n```bash\ndocker run -d -p 8000:8000 ghcr.io/pyjeebz/helios/inference:latest\n```\n\n**Kubernetes (Helm):**\n```bash\nhelm repo add helios https://pyjeebz.github.io/helios\nhelm install helios helios/helios\n```\n\n**Python SDK:**\n```bash\npip install helios-sdk helios-agent\n```\n\n---\n\n## 🎯 Overview\n\nHelios analyzes real-time metrics from your infrastructure, predicts future resource demands, detects anomalies, and provides actionable scaling recommendations—before problems occur.\n\n### Key Features\n\n| Feature | Description |\n|---------|-------------|\n| **Traffic Forecasting** | Predict CPU, memory, and request rates up to 1 hour ahead |\n| **Anomaly Detection** | Real-time detection of unusual patterns using XGBoost |\n| **Scaling Recommendations** | Actionable advice for replica counts and resource limits |\n| **Multi-Cloud Support** | Works on GKE, EKS, AKS, or any Kubernetes cluster |\n| **Prometheus Native** | Exposes metrics in Prometheus format for easy integration |\n\n---\n\n## 🏗️ Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────────────────┐\n│                         HELIOS PLATFORM                                      │\n├─────────────────────────────────────────────────────────────────────────────┤\n│                                                                             │\n│  ┌──────────────────┐    ┌─────────────────┐    ┌─────────────────────┐    │\n│  │ Metrics Adapter  │───▶│ ML Pipeline     │───▶│ Inference Service   │    │\n│  │ (pluggable)      │    │ (training)      │    │ (FastAPI)           │    │\n│  │ • GCP            │    │                 │    │ /predict            │    │\n│  │ • AWS            │    │ • Baseline      │    │ /detect             │    │\n│  │ • Azure          │    │ • Prophet       │    │ /recommend          │    │\n│  │ • Prometheus     │    │ • XGBoost       │    │ /metrics            │    │\n│  └──────────────────┘    └─────────────────┘    └──────────┬──────────┘    │\n│                                                            │               │\n│         ┌──────────────────────────────────────────────────┼───────────┐   │\n│         │                                                  │           │   │\n│         ▼                                                  ▼           ▼   │\n│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐  ┌───────────┐    │\n│  │ Prometheus  │    │ Alertmanager│    │ KEDA        │  │ Grafana   │    │\n│  │ (scraping)  │    │ (alerts)    │    │ (autoscale) │  │ (dashboard│    │\n│  └─────────────┘    └─────────────┘    └─────────────┘  └───────────┘    │\n│                                                                             │\n└─────────────────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## 📁 Project Structure\n\n```\nhelios/\n├── agent/                       # Helios Agent (metrics collection)\n│   ├── src/helios_agent/\n│   │   ├── agent.py             # Main agent orchestrator\n│   │   ├── cli.py               # Command-line interface\n│   │   ├── config.py            # Configuration loader\n│   │   ├── client.py            # Helios API client\n│   │   └── sources/             # Pluggable metric sources\n│   │       ├── base.py          # MetricsSource interface\n│   │       ├── registry.py      # Source registration\n│   │       ├── system.py        # Local system metrics\n│   │       ├── prometheus.py    # Prometheus backend\n│   │       ├── datadog.py       # Datadog backend\n│   │       ├── cloudwatch.py    # AWS CloudWatch\n│   │       ├── azure_monitor.py # Azure Monitor\n│   │       └── gcp_monitoring.py# GCP Cloud Monitoring\n│   └── pyproject.toml\n│\n├── ml/                          # Machine Learning\n│   ├── config.py                # Configuration\n│   ├── train.py                 # Training pipeline\n│   ├── pipeline/                # Data processing\n│   │   ├── data_fetcher.py      # Cloud metrics fetcher\n│   │   └── feature_engineering.py\n│   ├── models/                  # ML models\n│   │   ├── baseline.py          # Moving Average + Trend\n│   │   ├── prophet_model.py     # Prophet forecasting\n│   │   └── xgboost_anomaly.py   # XGBoost anomaly detection\n│   └── inference/               # Inference service (Phase 5)\n│       ├── app.py               # FastAPI application\n│       └── ...\n│\n├── infra/                       # Infrastructure\n│   ├── terraform/               # IaC for GCP/AWS/Azure\n│   │   └── gcp/\n│   │       ├── main.tf\n│   │       └── modules/\n│   │           ├── gke/         # Kubernetes cluster\n│   │           ├── cloudsql/    # PostgreSQL database\n│   │           ├── redis/       # Memorystore cache\n│   │           └── ...\n│   └── kubernetes/              # K8s manifests\n│       ├── saleor/              # Demo application\n│       ├── locust/              # Load testing\n│       ├── monitoring/          # Prometheus + Grafana\n│       └── helios-inference/    # ML inference service\n│\n├── loadtest/                    # Load testing\n│   └── locustfiles/             # Locust scenarios\n│       ├── locustfile.py        # Main test file\n│       └── personas/            # User personas\n│\n└── docs/                        # Documentation\n    └── architecture/\n        └── ARCHITECTURE.md\n```\n\n---\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Python 3.11+\n- Kubernetes cluster (GKE, EKS, AKS, or local)\n- kubectl configured\n- Terraform (for infrastructure provisioning)\n- Google Cloud SDK (for GCP deployment)\n\n### 1. Clone and Setup\n\n```bash\ngit clone https://github.com/your-org/helios.git\ncd helios\npython -m venv .venv\nsource .venv/bin/activate  # or .venv\\Scripts\\activate on Windows\npip install -r ml/requirements.txt\n```\n\n### 2. Configure Your Project\n\nUse the setup script to configure Helios with your GCP project:\n\n```bash\n# Linux/Mac\n./scripts/setup.sh YOUR_GCP_PROJECT_ID\n\n# Windows PowerShell\n.\\scripts\\setup.ps1 -ProjectId YOUR_GCP_PROJECT_ID\n```\n\nThis will:\n- Create a `.env` file with your project configuration\n- Update Kubernetes manifests with your project ID\n- Set up all required environment variables\n\nAlternatively, configure manually:\n\n```bash\n# Copy the example env file\ncp .env.example .env\n\n# Edit .env with your values\n# GCP_PROJECT_ID=your-project-id\n# GCP_REGION=us-central1\n```\n\n### 3. Authenticate with Cloud Provider\n\n```bash\n# For GCP\ngcloud config set project YOUR_GCP_PROJECT_ID\ngcloud auth application-default login\n\n# For AWS\nexport AWS_PROFILE=your-profile\n\n# For Azure\naz login\n```\n\n### 3. Train Models\n\n```bash\ncd ml\npython train.py\n```\n\n### 4. Deploy Infrastructure\n\n```bash\ncd infra/terraform/gcp\nterraform init\nterraform apply\n```\n\n### 5. Deploy Inference Service\n\n```bash\nkubectl apply -k infra/kubernetes/helios-inference/\n```\n\n---\n\n## 🤖 Helios Agent\n\nThe Helios Agent is a unified metrics collection daemon that can pull metrics from multiple backends and forward them to the Helios platform.\n\n### Installation\n\n```bash\n# Base installation (system metrics + Prometheus)\npip install helios-agent\n\n# With specific backends\npip install helios-agent[datadog]      # + Datadog support\npip install helios-agent[aws]          # + AWS CloudWatch\npip install helios-agent[azure]        # + Azure Monitor\npip install helios-agent[gcp]          # + GCP Cloud Monitoring\npip install helios-agent[all]          # All backends\n```\n\n### Supported Metrics Sources\n\n| Source | Description | Requirements |\n|--------|-------------|--------------|\n| `system` | Local CPU, memory, disk, network via psutil | Built-in |\n| `prometheus` | Query any Prometheus server | Built-in |\n| `datadog` | Pull metrics from Datadog API | `pip install helios-agent[datadog]` |\n| `cloudwatch` | AWS CloudWatch metrics | `pip install helios-agent[aws]` |\n| `azure_monitor` | Azure Monitor metrics | `pip install helios-agent[azure]` |\n| `gcp_monitoring` | Google Cloud Monitoring | `pip install helios-agent[gcp]` |\n\n### Quick Start\n\n```bash\n# Generate a configuration file\nhelios-agent init\n\n# List available metric sources\nhelios-agent sources\n\n# Test configured sources\nhelios-agent test\n\n# Run the agent (continuous collection)\nhelios-agent run\n\n# Check agent status\nhelios-agent status\n```\n\n### Configuration\n\nCreate a `helios-agent.yaml` file:\n\n```yaml\nagent:\n  collection_interval: 60      # seconds\n  batch_size: 100\n  log_level: INFO\n\nsources:\n  # Local system metrics (always recommended)\n  - type: system\n    enabled: true\n    config:\n      collect_cpu: true\n      collect_memory: true\n      collect_disk: true\n      collect_network: true\n\n  # Prometheus server\n  - type: prometheus\n    enabled: true\n    config:\n      url: http://prometheus:9090\n      queries:\n        - name: container_cpu\n          query: rate(container_cpu_usage_seconds_total[5m])\n        - name: container_memory\n          query: container_memory_usage_bytes\n\n  # AWS CloudWatch\n  - type: cloudwatch\n    enabled: false\n    config:\n      region: us-east-1\n      namespace: AWS/EC2\n      metrics:\n        - CPUUtilization\n        - NetworkIn\n        - NetworkOut\n\n  # Datadog\n  - type: datadog\n    enabled: false\n    config:\n      api_key: ${DATADOG_API_KEY}\n      app_key: ${DATADOG_APP_KEY}\n      queries:\n        - avg:system.cpu.user{*}\n\nhelios:\n  endpoint: http://localhost:8080\n  api_key: ${HELIOS_API_KEY}\n```\n\n### Environment Variables\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `HELIOS_CONFIG_FILE` | Path to config file | `./helios-agent.yaml` |\n| `HELIOS_ENDPOINT` | Helios API endpoint | `http://localhost:8080` |\n| `HELIOS_API_KEY` | API key for authentication | - |\n| `DATADOG_API_KEY` | Datadog API key | - |\n| `DATADOG_APP_KEY` | Datadog application key | - |\n| `AWS_REGION` | AWS region for CloudWatch | - |\n\n### CLI Commands\n\n```bash\n# Initialize configuration\nhelios-agent init [--output FILE]\n\n# Run agent with custom config\nhelios-agent run --config /path/to/config.yaml\n\n# List registered source plugins\nhelios-agent sources\n\n# Test all configured sources\nhelios-agent test\n\n# Show agent status\nhelios-agent status\n\n# Run single collection (for testing)\nhelios-agent run --once\n```\n\n### Creating Custom Sources\n\nYou can create custom metric sources by implementing the `MetricsSource` interface:\n\n```python\nfrom helios_agent.sources import MetricsSource, register_source, MetricSample\n\n@register_source(\"custom\")\nclass CustomSource(MetricsSource):\n    \"\"\"Custom metrics source.\"\"\"\n    \n    async def initialize(self) -\u003e None:\n        # Connect to your data source\n        pass\n    \n    async def collect(self) -\u003e CollectionResult:\n        # Fetch and return metrics\n        samples = [\n            MetricSample(\n                name=\"custom_metric\",\n                value=42.0,\n                labels={\"env\": \"prod\"}\n            )\n        ]\n        return CollectionResult(samples=samples)\n    \n    async def health_check(self) -\u003e bool:\n        return True\n    \n    async def close(self) -\u003e None:\n        # Cleanup\n        pass\n```\n\n---\n\n## 📊 ML Models\n\n### Model Performance (167 data points, 24-hour training)\n\n| Model | MAE | MAPE | Notes |\n|-------|-----|------|-------|\n| **Baseline (MA+Trend)** | 2.5M | 2.6% | Simple, fast, reliable |\n| **Prophet** | 25M | 21.1% | Better with more data, seasonality |\n| **XGBoost Anomaly** | - | 0.69% rate | 1 anomaly detected |\n\n### Feature Engineering\n\n- **Lag features**: 1, 3, 6, 12 periods\n- **Rolling statistics**: mean, std, min, max (windows: 3, 6, 12)\n- **Time features**: hour, day_of_week, is_weekend, sin/cos encoding\n- **Percent changes**: 1, 3 periods\n\n---\n\n## 🔌 API Endpoints\n\n| Endpoint | Method | Description |\n|----------|--------|-------------|\n| `/health` | GET | Service health check |\n| `/models` | GET | List loaded models |\n| `/predict` | POST | Forecast future metrics |\n| `/detect` | POST | Anomaly detection scoring |\n| `/recommend` | POST | Scaling recommendations |\n| `/metrics` | GET | Prometheus metrics |\n\n### Example: Get Predictions\n\n```bash\ncurl -X POST http://helios-inference:8080/predict \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"metrics\": {\n      \"cpu_utilization\": 0.45,\n      \"memory_utilization\": 0.62,\n      \"db_connections\": 15\n    },\n    \"periods\": 12\n  }'\n```\n\n### Example: Detect Anomalies\n\n```bash\ncurl -X POST http://helios-inference:8080/detect \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"metrics\": {\n      \"cpu_utilization\": 0.95,\n      \"memory_utilization\": 0.88,\n      \"db_connections\": 150\n    }\n  }'\n```\n\n---\n\n## 🔧 Configuration\n\n### Environment Variables\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `GCP_PROJECT_ID` | GCP project (for Cloud Monitoring) | - |\n| `METRICS_LOOKBACK_HOURS` | Hours of historical data | 24 |\n| `AGGREGATION_INTERVAL` | Metric aggregation (minutes) | 5 |\n| `ANOMALY_THRESHOLD_SIGMA` | Standard deviations for anomaly | 2.5 |\n\n### Scaling Thresholds\n\n```yaml\nscaling:\n  cpu_scale_up_threshold: 0.80\n  cpu_scale_down_threshold: 0.20\n  memory_warning_threshold: 0.85\n  min_replicas: 1\n  max_replicas: 10\n```\n\n---\n\n## 📈 Roadmap\n\n### Completed\n\n- [x] **Phase 1**: Infrastructure setup (Terraform + GKE)\n- [x] **Phase 2**: Demo application (Saleor e-commerce)\n- [x] **Phase 3**: Observability (Prometheus + Grafana)\n- [x] **Phase 4**: ML pipeline (Baseline, Prophet, XGBoost)\n\n### In Progress\n\n- [ ] **Phase 5**: Inference service \u0026 auto-scaling integration\n  - [ ] FastAPI inference service\n  - [ ] Real-time scoring loop\n  - [ ] KEDA predictive autoscaling\n  - [ ] Grafana dashboards\n  - [ ] Alertmanager integration\n\n### Future\n\n- [ ] **Phase 6**: Multi-cloud adapters (AWS, Azure)\n- [ ] **Phase 7**: Deep learning models (LSTM, Transformer)\n- [ ] **Phase 8**: Kubernetes operator\n\n---\n\n## 🧪 Testing\n\n### Run Unit Tests\n\n```bash\ncd ml\npytest tests/\n```\n\n### Run Load Tests\n\n```bash\n# Deploy Locust to cluster\nkubectl apply -k infra/kubernetes/locust/base\n\n# Port-forward to UI\nkubectl port-forward -n loadtest svc/locust-master 8089:8089\n\n# Start test via API\ncurl -X POST http://localhost:8089/swarm \\\n  -d \"user_count=100\u0026spawn_rate=10\u0026host=http://saleor-api.saleor.svc\"\n```\n\n---\n\n## 📝 License\n\nApache 2.0 - See [LICENSE](LICENSE) for details.\n\n---\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing`)\n3. Commit changes (`git commit -m 'Add amazing feature'`)\n4. Push to branch (`git push origin feature/amazing`)\n5. Open a Pull Request\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyjeebz%2Fhelios","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpyjeebz%2Fhelios","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyjeebz%2Fhelios/lists"}