{"id":50750767,"url":"https://github.com/felipemchdev/helios-eeip","last_synced_at":"2026-06-11T01:02:47.052Z","repository":{"id":361056880,"uuid":"1252912205","full_name":"felipemchdev/helios-eeip","owner":"felipemchdev","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-29T02:06:56.000Z","size":477,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-29T04:07:56.769Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://eeip-dashboard.wonderfulbush-04a41aac.eastus.azurecontainerapps.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/felipemchdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-29T01:56:10.000Z","updated_at":"2026-05-29T02:14:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/felipemchdev/helios-eeip","commit_stats":null,"previous_names":["felipemchdev/helios-eeip"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/felipemchdev/helios-eeip","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fhelios-eeip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fhelios-eeip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fhelios-eeip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fhelios-eeip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/felipemchdev","download_url":"https://codeload.github.com/felipemchdev/helios-eeip/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felipemchdev%2Fhelios-eeip/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34177445,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-11T01:02:46.216Z","updated_at":"2026-06-11T01:02:47.032Z","avatar_url":"https://github.com/felipemchdev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ☀️ Helios EEIP - Enterprise Ecosystem Intelligence Platform\n[Dashboard](https://eeip-dashboard.wonderfulbush-04a41aac.eastus.azurecontainerapps.io/) \n\n[![Python](https://img.shields.io/badge/Python-3.11-blue.svg)](https://python.org)\n[![dbt](https://img.shields.io/badge/dbt-1.7-orange.svg)](https://docs.getdbt.com)\n[![DuckDB](https://img.shields.io/badge/DuckDB-embedded-yellow.svg)](https://duckdb.org)\n[![Streamlit](https://img.shields.io/badge/Streamlit-latest-red.svg)](https://streamlit.io)\n[![Azure](https://img.shields.io/badge/Azure-East%20US-0078D4.svg)](https://azure.microsoft.com)\n[![License](https://img.shields.io/badge/license-MIT-lightgrey.svg)](./LICENSE)\n\nHelios EEIP is a self-hosted, Azure-native **supply chain security intelligence platform** that ingests open-source package metadata from [deps.dev](https://deps.dev) (Google Open Source Insights), maps the complete transitive dependency graph, cross-references known CVEs, and surfaces actionable risk analytics through an interactive dashboard.\n\n**What you can answer in under 10 seconds:**\n- Which packages in my portfolio have CRITICAL or HIGH CVEs in their dependency tree?\n- Is the vulnerability direct or transitive? What's the propagation chain?\n- What's the blast radius — how many other packages share this same vulnerable dependency?\n- Should I install, monitor, or avoid this package right now?\n\n---\n\n## Architecture Overview\n\n```\n┌──────────────────────────────────────────────────────────────────────┐\n│                          HELIOS EEIP                                  │\n├───────────────┬──────────────────────────────────────────────────────┤\n│  INGESTION    │  Azure Functions (Python v2)                          │\n│               │  Timer trigger every 3 days                           │\n│               │  → Pulls deps.dev API for 10+ high-impact packages   │\n│               │  → Writes raw JSON to ADLS Gen2 (Bronze)             │\n│               │  → Downloads Bronze to local temp storage            │\n│               │  → Runs dbt-duckdb: Bronze → Silver → Gold           │\n│               │  → Exports Gold tables as Parquet to ADLS            │\n├───────────────┼──────────────────────────────────────────────────────┤\n│  TRANSFORM    │  dbt 1.7 + DuckDB (embedded OLAP)                    │\n│               │  Medallion architecture: Bronze / Silver / Gold       │\n│               │  → stg_packages, stg_dependencies, stg_edges          │\n│               │  → stg_advisories, stg_dependency_advisories          │\n│               │  → dim_dependency_chain, dim_dependency_edges         │\n│               │  → fct_dependency_risk (risk score per package)      │\n├───────────────┼──────────────────────────────────────────────────────┤\n│  STORAGE      │  Azure Data Lake Storage Gen2                          │\n│               │  Container: bronze (raw JSON), gold (Parquet)        │\n│               │  Hierarchical namespace enabled                       │\n├───────────────┼──────────────────────────────────────────────────────┤\n│  DASHBOARD    │  Streamlit (Azure Container App, Consumption tier)    │\n│               │  → Reads Gold Parquet via pandas/pyarrow              │\n│               │  → Package impact cards with severity badges         │\n│               │  → CVE details, blast radius, dependency chains       │\n│               │  → Bubble chart risk radar                            │\n│               │  → Actionable recommendations per severity            │\n├───────────────┼──────────────────────────────────────────────────────┤\n│  SECURITY     │  Azure Key Vault, Managed Identity, RBAC             │\n│  MONITORING   │  Application Insights, Azure Monitor                  │\n│  REGISTRY     │  Azure Container Registry (eeipregistry)             │\n└───────────────┴──────────────────────────────────────────────────────┘\n```\n\n**Cost:** Under R$6 (~$1 USD) per month. Entire stack runs on Azure Consumption tier with scale-to-zero. No VMs. No always-on compute.\n\n---\n\n## Tech Stack\n\n| Layer | Technology | Version |\n|---|---|---|\n| Language | Python | 3.11 |\n| Ingestion + Orchestration | Azure Functions (Python v2) | 1.18.0 |\n| Transformation | dbt + dbt-duckdb adapter | 1.7.4 |\n| Execution Engine | DuckDB | embedded |\n| Dashboard | Streamlit + pandas + PyArrow | latest |\n| Storage | Azure Data Lake Storage Gen2 | — |\n| Container Registry | Azure Container Registry | — |\n| Secrets | Azure Key Vault | — |\n| Auth | Managed Identity (system-assigned) | — |\n| Monitoring | Application Insights | — |\n| Container Runtime | Azure Container Apps (Consumption) | — |\n| External API | deps.dev v3alpha (Google) | — |\n| Ecosystems Tracked | PyPI, NPM, Maven | — |\n| SQL Dialect | DuckDB | — |\n| dbt Packages | dbt_utils | ≥1.0.0, \u003c2.0.0 |\n\n---\n\n## Repository Structure\n\n```\neeip/\n├── README.md\n├── .gitignore\n├── .env.example\n│\n├── ingestion/                     # Azure Functions (Python v2)\n│   ├── function_app.py            # Timer trigger + deps.dev scraper\n│   ├── requirements.txt\n│   ├── host.json\n│   └── transformation/            # embedded dbt project (deployed with Function)\n│       ├── dbt_project.yml\n│       ├── profiles.yml\n│       ├── packages.yml\n│       └── models/\n│           ├── bronze/sources.yml\n│           ├── silver/            # stg_packages, stg_dependencies, stg_edges,\n│           │                      # stg_advisories, stg_dependency_advisories\n│           └── gold/              # dim_dependency_chain, dim_dependency_edges,\n│                                  # fct_dependency_risk, schema.yml\n│\n├── orchestration/                 # Apache Airflow (legacy — pipeline now self-contained in Function)\n│   ├── Dockerfile\n│   ├── docker-compose.yml\n│   └── dags/\n│       └── eeip_pipeline.py       # Unused — ingestion runs dbt internally\n│\n├── transformation/                # dbt project (standalone copy)\n│   ├── dbt_project.yml\n│   ├── profiles.yml\n│   ├── packages.yml\n│   └── models/                    # Same structure as ingestion/transformation/\n│\n├── dashboard/                     # Streamlit\n│   ├── Dockerfile                 # python:3.11-slim\n│   ├── app.py                     # Full dashboard with HTML cards\n│   ├── requirements.txt           # streamlit, pandas, pyarrow, azure-storage-blob\n│   └── fonts/                     # CSS font files (deprecated)\n│\n└── airflow-config.yaml            # Exported Container App configuration\n```\n\n---\n\n## Medallion Architecture\n\n| Layer | Description | Storage |\n|---|---|---|\n| **Bronze** | Raw JSON from deps.dev, partitioned by `YYYY-MM-DD/ecosystem_package.json` | ADLS: `bronze/raw/deps_dev/` |\n| **Silver** | Deduplicated, normalized views — packages, dependencies, edges, advisories | DuckDB views |\n| **Gold** | Business-ready tables — dependency chains, dependency edges with CVE mapping, risk per root package | ADLS: `gold/{model}/*.parquet` |\n\n### dbt Models\n\n**Silver:**\n| Model | Purpose |\n|---|---|\n| `stg_packages` | Normalized package + advisory data, deduplicated by `ingested_at DESC` |\n| `stg_dependencies` | Unnested dependency graph with direct/indirect classification |\n| `stg_dependency_edges` | Parent → child edge resolution from `dependencies.edges[]` |\n| `stg_advisories` | CVE metadata with severity derived from CVSS score |\n| `stg_dependency_advisories` | Dependency → advisory ID mapping |\n\n**Gold:**\n| Model | Purpose |\n|---|---|\n| `dim_dependency_chain` | Denormalized dependency → advisory chain (root package → dep → CVE) |\n| `dim_dependency_edges` | Denormalized edge → advisory chain (parent → child → CVE) |\n| `fct_dependency_risk` | Per-package risk metrics: dep counts, advisory exposure, risk density, severity level |\n\n### dbt Data Quality Tests\n\n| Column | Test |\n|---|---|\n| `root_package` | `not_null`, `unique` |\n| `risk_level` | `accepted_values: ['HIGH', 'MEDIUM', 'LOW']` |\n| `risk_density` | `not_null`, `\u003e= 0` |\n| `total_dep_count` | `not_null`, `\u003e= 1` |\n\n---\n\n## Dashboard Features\n\nThe Streamlit dashboard provides a platform operations view of supply chain risk:\n\n**KPI Bar:** Total packages · Total dependencies · Total vulnerabilities · Critical/High count · Affected packages\n\n**Package Impact Analysis:**\n- Cards sorted by severity (CRITICAL → HIGH → MEDIUM), with emoji badges\n- Severity derived from actual CVE data, not ratio thresholds\n- Blast radius: how many other packages share each vulnerable transitive dependency\n- Dependency chain visualization: `root → parent → vulnerable_dep` paths\n- Top 5 CVEs per package with title, CVSS score, and direct/transitive indicator\n- Actionable recommendations per severity: install, monitor, or avoid\n\n**Safe Packages:** Green cards for packages with zero known advisories\n\n**Blast Radius View:** Shared vulnerable dependencies ranked by number of affected packages\n\n**CVE Explorer:** Full per-CVE detail with affected packages, dependency chains, and fix recommendations\n\n**Package Risk Radar:** Bubble chart — X = advisory exposure, Y = critical dependencies, size = total dependencies, color = risk level. Raw data available in expander.\n\n---\n\n## Deployment\n\n### Prerequisites\n\n- Azure CLI 2.x with `az login`\n- Docker (for building Container App images)\n- Python 3.11 (for local development)\n\n### Dashboard (most common deploy)\n\n```bash\ncd eeip/dashboard\ndocker build -t eeipregistry.azurecr.io/dashboard:vXX .\ndocker push eeipregistry.azurecr.io/dashboard:vXX\naz containerapp update \\\n  --name eeip-dashboard \\\n  --resource-group rg-eeip \\\n  --image eeipregistry.azurecr.io/dashboard:vXX\n```\n\n### Ingestion + Transformation (Azure Functions)\n\n```bash\ncd eeip/ingestion\nfunc azure functionapp publish eeip-ingestion --python\n```\n\nThe Function handles everything end-to-end: pulls deps.dev API, downloads bronze from ADLS, runs dbt silver → gold with DuckDB, and exports gold Parquet back to ADLS. No separate orchestrator needed.\n\n---\n\n## Local Development\n\n### 1. Environment\n\n```bash\ncp .env.example .env     # Edit with your values\naz login                 # Authenticate to Azure\n```\n\n### 2. Ingestion\n\n```bash\ncd ingestion\npip install -r requirements.txt\ncurl -X POST http://localhost:7071/admin/functions/ingest_deps_dev\n```\n\n### 3. Transformations (dbt)\n\n```bash\ncd transformation\npip install dbt-duckdb\ndbt deps\ndbt run\ndbt test\n```\n\n### 4. Dashboard\n\n```bash\ncd dashboard\npip install -r requirements.txt\nstreamlit run app.py\n```\n\n### 5. Airflow (legacy — optional)\n\n```bash\ncd orchestration\ndocker compose up --build -d\n# UI at http://localhost:8080\n```\n\n\u003e Airflow is no longer part of the active pipeline. The Function runs dbt\n\u003e internally and exports gold Parquet to ADLS.\n\n---\n\n## Auth \u0026 Security\n\n- **Production:** System-assigned Managed Identity for Function → ADLS and Container Apps → ACR. Secrets in Azure Key Vault.\n- **Current dashboard:** `ADLS_CONNECTION_STRING` via environment variable (migration to Managed Identity planned).\n- **Development:** `az login` + `DefaultAzureCredential()`. `.env` and `local.settings.json` are in `.gitignore`.\n\n---\n\n## Cost Breakdown\n\n| Resource | Tier | Monthly Cost (est.) |\n|---|---|---|\n| Azure Functions | Consumption, timer + HTTP trigger | $0.00 (well within free grant) |\n| Container Apps (dashboard) | Consumption, 0.5 vCPU, 1 GB, scale-to-zero | ~$0.50 |\n| ADLS Gen2 | ~10 MB storage + ~20 writes/run | ~$0.05 |\n| Key Vault | Free tier | $0.00 |\n| Container Registry | Basic | ~$0.15 |\n| **Total** | | **~$1.00 / R$6.00** |\n\n---\n\n## Roadmap\n\n### Phase 1 — Production Hardening\n- [ ] Managed Identity for all ADLS access (no connection strings)\n- [ ] Terraform/Bicep IaC for all resources\n- [ ] GitHub Actions CI/CD pipeline\n- [ ] Azure Monitor alert rules\n\n### Phase 2 — Data Enrichment\n- [ ] OSV.dev API integration for full CVSS vectors and exploit availability\n- [ ] Weighted risk score: severity �- depth in dependency graph �- exploitability\n- [ ] dbt incremental models for silver and gold layers\n- [ ] Historical snapshot tracking\n\n### Phase 3 — Expansion\n- [ ] Dynamic package discovery (scrape top-downloaded, not hardcoded)\n- [ ] Additional ecosystems: Go, Cargo, NuGet\n- [ ] SBOM export (CycloneDX/SPDX)\n- [ ] Slack/Teams notification integration\n- [ ] OpenSSF Scorecard metrics\n\n### Phase 4 — Platform\n- [ ] REST API layer (FastAPI)\n- [ ] Multi-tenant support\n- [ ] Replace DuckDB with Databricks/Synapse at scale\n\n---\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelipemchdev%2Fhelios-eeip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffelipemchdev%2Fhelios-eeip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelipemchdev%2Fhelios-eeip/lists"}