{"id":48541891,"url":"https://github.com/nolieravioli/eve_data_framework3","last_synced_at":"2026-04-08T05:01:24.429Z","repository":{"id":291294110,"uuid":"977197033","full_name":"NolieRavioli/eve_data_framework3","owner":"NolieRavioli","description":"A scalable and secure backend framework for EVE Online data, built for ME, industrial pilots, market traders, and corporations. ","archived":false,"fork":false,"pushed_at":"2026-04-04T23:31:32.000Z","size":10869,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-05T01:21:07.124Z","etag":null,"topics":["esi","eve","eve-online","eve-swagger-interface"],"latest_commit_sha":null,"homepage":"https://nolie.space/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NolieRavioli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-05-03T16:37:12.000Z","updated_at":"2026-04-04T23:31:33.000Z","dependencies_parsed_at":"2025-05-03T17:41:03.667Z","dependency_job_id":"72256b51-9462-457d-9e2b-3c555b2b1fbd","html_url":"https://github.com/NolieRavioli/eve_data_framework3","commit_stats":null,"previous_names":["nolieravioli/eve_data_framework3"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/NolieRavioli/eve_data_framework3","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolieRavioli%2Feve_data_framework3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolieRavioli%2Feve_data_framework3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolieRavioli%2Feve_data_framework3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolieRavioli%2Feve_data_framework3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NolieRavioli","download_url":"https://codeload.github.com/NolieRavioli/eve_data_framework3/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NolieRavioli%2Feve_data_framework3/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31540826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T16:28:08.000Z","status":"online","status_checked_at":"2026-04-08T02:00:06.127Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["esi","eve","eve-online","eve-swagger-interface"],"created_at":"2026-04-08T05:01:11.070Z","updated_at":"2026-04-08T05:01:24.421Z","avatar_url":"https://github.com/NolieRavioli.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# EVE Data Framework\n\nA self-hosted web application and data platform for EVE Online. The framework provides:\n\n- **EVE SSO authentication** — multi-character OAuth 2.0 with Fernet-encrypted token storage\n- **Shared DuckDB warehouse** — SDE dimension tables, live market orders, structure data, and ESI spec metadata\n- **Per-character private SQLite databases** — skills, wallet, assets, and token credentials per owner\n- **Background task queue** — two FIFO executor queues with live SSE log streaming to the browser\n- **Cron-style scheduler** — recurring background jobs for market/structure/character data collection\n- **Auto-discovered Flask applications** — pluggable web tools with role-based access control\n\n---\n\n## Table of Contents\n\n1. [Installation](#installation)\n2. [First-Run Setup](#first-run-setup)\n3. [Accessing the Dashboard](#accessing-the-dashboard)\n4. [Site Admin Guide](#site-admin-guide)\n5. [Configuration Reference](#configuration-reference)\n6. [Built-In Applications](#built-in-applications)\n7. [Background Scheduler](#background-scheduler)\n8. [Architecture Overview](#architecture-overview)\n9. [Developer Guide — Creating Applications](#developer-guide--creating-applications)\n10. [Developer Guide — Creating Analysis Collectors](#developer-guide--creating-analysis-collectors)\n11. [Developer Guide — Core Layer](#developer-guide--core-layer)\n12. [ESI Client \u0026 Code Generation](#esi-client--code-generation)\n13. [Security Notes](#security-notes)\n14. [Licensing](#licensing)\n\n---\n\n## Installation\n\n### Prerequisites\n\n- Python 3.12 or later\n- `pip` (included with Python)\n- Network access to EVE Online ESI (`esi.evetech.net`) and SSO (`login.eveonline.com`)\n- An active EVE Online developer application (register at [developers.eveonline.com](https://developers.eveonline.com))\n\n### Clone and install dependencies\n\n```bash\ngit clone https://github.com/NolieRavioli/eve_data_framework3.git\ncd eve_data_framework3\npip install -r requirements.txt\n```\n\n### Configure\n\nCopy the example config and open it in your editor:\n\n```bash\n# Linux / macOS\ncp example.config.yaml config.yaml \u0026\u0026 nano config.yaml\n\n# Windows (PowerShell)\nCopy-Item example.config.yaml config.yaml ; notepad config.yaml\n```\n\nThe minimum required change is to enter your EVE developer application credentials (see [First-Run Setup](#first-run-setup)).\n\n\u003e **Note:** `config.yaml` is gitignored. It will never be committed. Keep your `client_id` and `client_secret` out of source control.\n\n### Start the server\n\n```bash\npython main.py\n```\n\nThe server starts on `http://127.0.0.1:5000` by default. To accept connections from other machines on your network, change `host` in `config.yaml`:\n\n```yaml\nRuntime:\n  host: \"0.0.0.0\"\n  port: 5000\n```\n\n---\n\n## First-Run Setup\n\nOn the very first launch with a fresh database, the framework will display a **setup wizard** at `http://127.0.0.1:5000/setup`. This wizard walks through:\n\n1. **Registering your EVE developer application credentials** — paste in your `client_id` and `client_secret` from [developers.eveonline.com](https://developers.eveonline.com). The callback URL you register on CCP's portal should be `http://\u003cyour-host\u003e/auth/callback`.\n2. **Logging in with your EVE character** — the first character to log in becomes the **site owner** and gains unconditional admin access.\n3. **Triggering the initial SDE load** — downloads and unpacks the Static Data Export from CCP's servers into `_sde/` and builds the DuckDB warehouse. This can take several minutes depending on which SDE sections are enabled.\n\nAfter setup, subsequent starts skip directly to the login page.\n\n---\n\n## Accessing the Dashboard\n\nNavigate to `http://\u003chost\u003e:\u003cport\u003e/` in a browser. If you are not logged in you will be redirected to the login page. Click **Login with EVE Online** to start the SSO flow.\n\nOnce authenticated, you will land on the **Dashboard**, which shows your linked characters and quick-links to enabled applications.\n\nTo add additional characters to your account, use the **Add Toon** link on the dashboard.\n\n---\n\n## Site Admin Guide\n\n### User Management\n\nSite admins can manage other users through the **Admin Panel** (`/admin`):\n\n- **Promote to site admin** — grants a user admin-level access across all applications.\n- **Grant / revoke roles** — control which applications each user can access. Roles map 1-to-1 to application role names.\n- **View logs** — the admin panel exposes a live log console streamed via SSE.\n\n### Default Roles\n\nEvery new user is automatically granted the roles listed under `Auth.default_roles` in `config.yaml`:\n\n```yaml\nAuth:\n  default_roles:\n    - dashboard\n    - queue\n```\n\nChange this list to grant or restrict access to applications for new users by default.\n\n### Granting Roles Programmatically\n\n```python\nfrom core.auth import grant_user_roles, revoke_user_role\n\ngrant_user_roles(owner_id, [\"dashboard\", \"database\", \"sde\", \"tasks\"], granted_by=admin_owner_id)\nrevoke_user_role(owner_id, \"tasks\")\n```\n\n### Triggering Manual Data Collection\n\nData collection happens automatically on the scheduler (see [Background Scheduler](#background-scheduler)). To trigger a refresh manually, visit the **Scheduler** tab (`/tasks/scheduler/`) and click **Run Now** on any job. You can also use the **Task Manager** (`/tasks`) to monitor running jobs and stream their logs live.\n\n### Regenerating the ESI Spec\n\nIf ESI adds new routes or changes its schema, regenerate the typed client:\n\n```powershell\npython build.py --force\n```\n\nThis fetches the latest OpenAPI spec, regenerates the typed Python client in `core/esi/generated/`, and regenerates the domain-specific wrappers in `core/esi/personal/`, `core/esi/corp/`, and `core/esi/public/`.\n\n---\n\n## Configuration Reference\n\nAll runtime settings live in `config.yaml` (gitignored). A fully annotated template is provided in `example.config.yaml`. Key sections:\n\n### `Runtime`\n\n| Key | Default | Description |\n|-----|---------|-------------|\n| `host` | `\"127.0.0.1\"` | Flask bind address. Set to `\"0.0.0.0\"` to accept remote connections. |\n| `port` | `5000` | TCP port. |\n| `debug` | `false` | Enable Flask debug mode (never use in production). |\n| `auto_install` | `false` | Auto-run `pip install -r requirements.txt` if modules are missing. |\n| `trace_esi` | `false` | Print every outbound ESI HTTP call to stdout. |\n| `secret_key` | random | Flask session secret. Set a persistent value so sessions survive restarts. |\n\n### `Python Console`\n\nControls what appears in the terminal/console output. The root `global_log_level` sets the console `StreamHandler` threshold; per-logger keys silence or amplify individual loggers.\n\n| Key | Default | Description |\n|-----|---------|-------------|\n| `global_log_level` | `INFO` | Threshold for all console output. |\n| `werkzeug_log_level` | `INFO` | Werkzeug request log level (`WARNING` to silence per-request lines). |\n| *(any logger name)* | — | Override the level of any named logger, e.g. `core.esi.rate: WARNING`. |\n\n### `Web Console`\n\nControls the in-browser live log in the admin panel.\n\n| Key | Default | Description |\n|-----|---------|-------------|\n| `admin_panel_global_log_level` | `DEBUG` | Minimum level captured into the admin panel live console. |\n\n### `Environment Variables`\n\n| Key | Default | Description |\n|-----|---------|-------------|\n| `LANGUAGE` | `\"en\"` | Primary language for SDE text fields. |\n| `SUPPORTED_LANGUAGES` | `\"en\"` | Comma-separated languages; SDE pruner keeps only these. |\n| `PUBLIC_DATA_FOLDER` | `\"_publicData\"` | Directory for DuckDB file and OAuth credentials. |\n| `EVE_PRIVATE_DATABASE_FOLDER` | `\"_privateData/\"` | Root directory for per-owner SQLite files. |\n| `SDE_PATH` | `\"_sde/\"` | Local SDE YAML file directory. |\n\n### `Auth`\n\n```yaml\nAuth:\n  default_roles:\n    - dashboard\n    - queue\n```\n\n### `SDE`\n\nToggle which SDE datasets load at startup. Set a section to `false` to skip it (will lazy-load on first use):\n\n```yaml\nSDE:\n  database_file: \"_publicData/public.duckdb\"\n  load_types: true        # type ID \u003c-\u003e name maps          (~26 MB, ~4s)\n  load_groups: true       # item group hierarchy            (~0.3 MB, fast)\n  load_categories: true   # item categories                 (~0.1 MB, fast)\n  load_market_groups: true # market group tree              (~0.3 MB, fast)\n  load_universe: true     # solar-system -\u003e region map      (~15s)\n  load_blueprints: true   # manufacturing blueprints        (~8s)\n  load_type_materials: true # reprocessing materials        (~3s)\n  load_type_dogma: true   # item dogma attributes           (~60s+)\n```\n\n### `Structures` and `Market`\n\nCooldown settings that prevent re-fetching recently inaccessible structures. See `example.config.yaml` for full details.\n\n---\n\n## Built-In Applications\n\n| Application | URL Prefix | Access | Description |\n|-------------|------------|--------|-------------|\n| `market_browser` | `/market` | Public (no login) | Browse live market orders by region and item type |\n| `dashboard` | `/dashboard` | Role: `dashboard` | Character overview and quick links to all tools |\n| `task_viewer` | `/tasks` | Role: `tasks` | Background task queue, live rate monitoring, scheduler (admin), and ESI explorer (admin) |\n| `db_viewer` | `/db` | Role: `database` | DB queue stats, read stats; admin-tier schema browser and query tool |\n| `admin_panel` | `/admin` | Admin only | User management — roles, admin promotion, user list |\n| `sde_browser` | `/sde` | Admin only | SDE table browser, lookup tool, and reload trigger |\n| `system` | `/system` | Admin only | Process metrics, git version, and system update |\n\n---\n\n## Background Scheduler\n\nThe scheduler (`core/tasks/engine.py`) runs a background thread that ticks every 30 seconds and fires registered jobs when their `next_run` is due. Job state (enabled, last run, next run, interval) is persisted in the DuckDB `scheduler_jobs` table so customizations survive restarts.\n\n### Default Jobs\n\n| Job | Default Interval | Worker |\n|-----|-----------------|--------|\n| Market Data Refresh | 1 hour | `analysis.market.regions.fetch_all_market_data` |\n| Structure Discovery | 24 hours | `analysis.structures.discover.discover_structures` |\n| Character Data Refresh | 24 hours | refreshes all owners via `analysis.character.populate.populate_all` |\n\nManage jobs through the **Scheduler** tab in the Task Manager (`/tasks/scheduler/`) or the `scheduler` adapter in code:\n\n```python\nfrom applications._api import scheduler\n\nscheduler.list_jobs()              # list[dict] — all registered jobs\nscheduler.set_enabled(\"market_refresh\", True)\nscheduler.run_now(\"structure_discovery\")  # returns task_id\n```\n\n---\n\n## Architecture Overview\n\n```\nmain.py                  # startup: load config, init DB, start Flask\nconfig.yaml              # runtime configuration (gitignored)\nexample.config.yaml      # annotated template — copy to config.yaml\nrequirements.txt\n\ncore/                    # infrastructure — never import from applications/\n  config.py              # load_config(), RuntimeSettings, ensure_dependencies()\n  auth/\n    __init__.py          # re-exports: require_login, require_admin, require_role, pick_token, fresh_token, get_token, resolve_default_owner_id\n    decorators.py        # require_login, require_admin, require_role — Flask route decorators\n    identity.py          # user/admin/role CRUD: link_public_user, get_user_roles, grant_user_roles, revoke_user_role, …\n    credentials.py       # CredentialManager — Fernet-encrypts/decrypts client_id/client_secret\n    tokens.py            # TokenDBManager, get_token(), pick_token(), fresh_token(), resolve_default_owner_id()\n    sso.py               # auth_bp — EVE SSO OAuth2 flow: /login, /callback, /logout, /add_toon, /switch_character\n  bus/\n    __init__.py          # re-exports topics, BusHandler, registry fns, websocket helpers\n    topics.py            # LOG_DB, LOG_ESI, LOG_SCHEDULER, ESI_RATE, DB_STATS, SYSTEM_PROCESS, QUEUE_TASKS, classify()\n    handler.py           # BusHandler(logging.Handler) — ring-buffer per-topic, publish(), subscribe()\n    registry.py          # TopicConfig, register_topic(), get_topic_config()\n    websocket.py         # /bus WebSocket endpoint, register_websock(), attach_all_websocks()\n    process_pub.py       # SYSTEM_PROCESS periodic publisher\n  db/\n    public.py            # DuckDB connect(), CRUD helpers, identity-table DDL\n    private.py           # SQLite per-owner: initialize_private_database(), get_private_session()\n    models/identity.py   # User, SiteAdmin (DuckDB ORM), Character (SQLite ORM)\n    sde.py               # in-memory SDE lookup caches\n    reader.py            # query_rows(), query_one(), query_scalar(), get_db_file_stats()\n    writer.py            # DuckDB write thread (serialises public DB writes)\n    stats.py             # get_table_stats(), start_db_stats_publisher()\n    market_buffer.py     # ephemeral in-process market buffer\n  esi/\n    request.py           # esi_request(), esi_get(), esi_post() — ALL ESI HTTP\n    rate.py              # ESIRateLimiter — floating-window token bucket, ETag caching, 429 backoff\n    cache.py             # DuckDB-backed response cache (esi_cache table)\n    registry.py          # ESI OpenAPI spec fetcher and DuckDB registry\n    generated/           # AUTO-GENERATED — do not edit by hand\n    personal/            # AUTO-GENERATED domain wrappers (character-scoped)\n    corp/                # AUTO-GENERATED domain wrappers (corporation-scoped)\n    public/              # AUTO-GENERATED domain wrappers (public)\n  tasks/\n    queue.py             # Task class, enqueue(), get_task(), cancel_task(), clear_tasks()\n    engine.py            # SchedulerEngine, get_engine()\n    jobs.py              # job catalog — add new scheduled jobs here\n    persist.py           # scheduler_jobs table DDL\n    output.py            # task log routing and IO\n    context.py           # thread-local task context\n    sde_loader.py        # SDE pipeline: download → unzip → prune → DuckDB warehouse\n  web/\n    __init__.py          # create_app() — Flask app factory\n    app.py               # start_webUI() entry point\n    context.py           # base_ctx() sidebar helper\n    home.py / setup.py   # home and setup wizard blueprints\n\nanalysis/                # empty — reserved for future use\ncollectors/              # data collection workers\n  character/populate.py  # per-owner ESI → private SQLite (skills, wallet, assets)\n  market/\n    regions.py           # NPC region market orders\n    structures.py        # player-structure market orders\n    history.py           # market order history (daily price snapshots)\n  structures/discover.py # discover + enrich public structures\n\napplications/            # user-facing web tools (auto-discovered)\n  _api.py                # single interface: BaseTool, ToolManifest, base_ctx, require_* and all adapters\n  dashboard/             # character overview — nav_section=\"overview\"\n  task_viewer/           # task queue + scheduler + ESI explorer — nav_section=\"tools\"\n  db_viewer/             # DB stats + query browser — nav_section=\"tools\"\n  admin_panel/           # user management — nav_section=\"admin\"\n  sde_browser/           # SDE table browser + update trigger — nav_section=\"admin\"\n  system/                # process metrics + system update — nav_section=\"admin\"\n  market_browser/        # live market order browser — nav_section=\"overview\"\n\nutils/build/             # ESI client code generation (run via build.py)\n```\n\n### Layering Rules\n\n- `core/` — infrastructure. No imports from `applications/` or `collectors/`.\n- `collectors/` — data collectors. Import from `core.*` only.\n- `applications/` — web applications. Import from `applications._api` **only** — never directly from `core.*`.\n\n---\n\n## Developer Guide — Creating Applications\n\nEach application lives in `applications/\u003cname\u003e/` and is auto-discovered by `pkgutil` on startup.\n\n### Required files\n\n```\napplications/my_tool/\n  __init__.py      # defines Tool = MyTool()\n  routes.py        # Flask blueprint\n  templates/       # Jinja2 templates\n  static/          # JavaScript, CSS (optional)\n```\n\n### `__init__.py`\n\n```python\nfrom applications._api import BaseTool, ToolManifest\nfrom applications.my_tool import routes\n\nclass MyTool(BaseTool):\n    manifest = ToolManifest(\n        id=\"my_tool\",\n        name=\"My Tool\",\n        icon=\"?\",\n        description=\"Does something useful.\",\n        url_prefix=\"/tools/my_tool\",\n        required_scopes=[],\n        nav_weight=50,\n        nav_section=\"apps\",\n        access_level=\"user\",\n        required_role=\"my_tool\",\n    )\n\n    def create_blueprint(self):\n        return routes.my_bp\n\nTool = MyTool()\n```\n\n`access_level` values: `\"public\"` | `\"user\"` | `\"admin\"` | `\"site_owner\"`\n\nSet `required_role=None` if no named role is required (access_level check only).\n\n`nav_section` values: `\"overview\"` | `\"tools\"` | `\"apps\"` | `\"admin\"` | `\"\"` (hidden)\n\n### `routes.py`\n\n```python\nfrom flask import Blueprint, render_template\nfrom applications._api import base_ctx, require_role, db, sde\n\nmy_bp = Blueprint(\"my_tool\", __name__,\n                  template_folder=\"templates\",\n                  static_folder=\"static\")\n\n@my_bp.route(\"/\")\n@require_role(\"my_tool\")\ndef index():\n    rows = db.query(\"SELECT type_id, name FROM sde_types LIMIT 10\")\n    return render_template(\"my_tool.html\", rows=rows, **base_ctx(\"my_tool\"))\n```\n\nUse `@require_role(\"role\")` for named-role access (admins bypass automatically).\nUse `@require_admin` for admin-only routes. Avoid bare `@require_login` on new routes.\n\n### Template\n\n```html\n{% extends \"base.html\" %}\n{% block title %}My Tool{% endblock %}\n{% block content %}\n  \u003cdiv class=\"pg-hd\"\u003e\u003ch1\u003eMy Tool\u003c/h1\u003e\u003c/div\u003e\n  \u003cdiv class=\"pg-body\"\u003e...\u003c/div\u003e\n{% endblock %}\n{% block scripts %}\n\u003cscript src=\"{{ url_for('my_tool.static', filename='my_tool.js') }}\"\u003e\u003c/script\u003e\n{% endblock %}\n```\n\n### Available adapters (`applications._api`)\n\n| Adapter | Type | Description |\n|---------|------|-------------|\n| `db` | `_LiveDBAdapter` | `query()`, `query_one()`, `scalar()`, `private_query()`, `market_price()` |\n| `sde` | `core.db.sde` module | All SDE lookup functions directly |\n| `raw_esi` | `_LiveRawESIAdapter` | `get()`, `post()`, `request()` — rate-limited HTTP |\n| `esi` | `_LiveESIAdapter` | `execute(op_id, ...)`, `fetch_pages(op_id, ...)` — typed client |\n| `tokens` | `_LiveTokenAdapter` | `get(owner_id)` — raw token map |\n| `token_resolution` | `_LiveTokenResolutionAdapter` | `pick_token()`, `fresh_token()` |\n| `tasks` | `_LiveTaskAdapter` | `enqueue(name, fn, *args, owner_id, queue)` |\n| `char_data` | `_LiveCharacterDataAdapter` | `get_character()`, `get_scopes()` |\n| `esi_registry` | `_LiveESIRegistryAdapter` | `get_status()` |\n| `db_admin` | `_LiveDBAdminAdapter` | Admin-level DB inspection helpers |\n| `esi_manifest` | `_LiveESIManifestAdapter` | `get_operations()`, `get_operation(op_id)`, `get_meta()` |\n| `queue_info` | `_LiveQueueInfoAdapter` | `get_all_tasks()`, `get_tasks_for_owner()`, `get_task()`, `cancel_task()`, `clear_tasks()`, `get_esi_rate_stats()` |\n| `scheduler` | `_LiveSchedulerAdapter` | `list_jobs()`, `set_enabled()`, `run_now()` |\n\n### Background workers from an application\n\nLong-running work (data fetches, calculations) should be enqueued rather than blocking a request:\n\n```python\nfrom applications._api import tasks\n\ndef my_worker(owner_id: int) -\u003e None:\n    # runs in background thread — logging is streamed to the browser\n    import logging\n    logger = logging.getLogger(__name__)\n    logger.info(\"Starting my_worker for owner %s\", owner_id)\n    # ... do work ...\n\n@my_bp.route(\"/run\")\n@require_role(\"my_tool\")\ndef run_task():\n    from flask import session\n    owner_id = session[\"owner_id\"]\n    task_id = tasks.enqueue(\"My Task\", my_worker, owner_id, owner_id=owner_id)\n    return {\"task_id\": task_id}\n```\n\n---\n\n## Developer Guide — Creating Analysis Collectors\n\nAnalysis collectors are data pipeline workers that populate DuckDB or per-owner SQLite. They live in `collectors/\u003cdomain\u003e/`.\n\n### Structure\n\n```\ncollectors/my_domain/\n  __init__.py       # re-exports entry points\n  worker.py         # table DDL + data collection functions\n```\n\n### Table Ownership Pattern\n\n```python\n# analysis/my_domain/worker.py\nimport logging\nimport core.db.public as db\n\nlogger = logging.getLogger(__name__)\n\ndef ensure_tables(con) -\u003e None:\n    \"\"\"Idempotent DDL — always call before any writes.\"\"\"\n    con.execute(\"\"\"\n        CREATE TABLE IF NOT EXISTS my_table (\n            id INTEGER PRIMARY KEY,\n            value DOUBLE,\n            fetched_at TIMESTAMP DEFAULT now()\n        )\n    \"\"\")\n\ndef fetch_my_data() -\u003e None:\n    \"\"\"Entry point called by the scheduler or manually.\"\"\"\n    con = db.connect()\n    try:\n        ensure_tables(con)\n        # ... collect data and write ...\n    finally:\n        con.close()\n```\n\n### Enrichment Pattern\n\nIf your collector adds columns to a table owned by another collector:\n\n```python\ndef ensure_columns(con) -\u003e None:\n    from collectors.structures.discover import ensure_tables as _ensure_structures\n    _ensure_structures(con)\n    con.execute(\"\"\"\n        ALTER TABLE structures ADD COLUMN IF NOT EXISTS my_col TIMESTAMP\n    \"\"\")\n```\n\n### Making ESI Requests\n\n**Never use `requests` directly.** Import `esi_get`/`esi_request` directly from `core.esi`:\n\n```python\nfrom core.esi import esi_get, esi_request\n\nresp = esi_get(\"https://esi.evetech.net/latest/markets/10000002/orders/\",\n               params={\"page\": 1, \"order_type\": \"all\"})\nif not resp.ok:\n    logger.warning(\"ESI %s: %s\", resp.status_code, resp.url)\n    return\n\ndata = resp.json()\ntotal_pages = int(resp.headers.get(\"X-Pages\", 1))\n```\n\n### Registering a Scheduled Job\n\nAdd an entry to `core/tasks/jobs.py`:\n\n```python\ntry:\n    from collectors.my_domain.worker import fetch_my_data\n    jobs.append({\n        \"job_id\": \"my_domain_refresh\",\n        \"label\": \"My Domain Data Refresh\",\n        \"fn\": fetch_my_data,\n        \"fn_path\": _path(fetch_my_data),\n        \"interval_s\": 3600,\n    })\nexcept Exception:\n    logger.warning(\"[SchedulerJobs] Could not import my_domain — skipping job\")\n```\n\nThe scheduler will auto-upsert the job on next startup.\n\n---\n\n## Developer Guide — Core Layer\n\nThe `core/` layer is the infrastructure backbone. Applications import from `applications._api` — they should not import from `core.*` directly. Analysis workers import from `core.*` directly.\n\n### Key modules\n\n| Module | Purpose |\n|--------|---------|\n| `core.config` | `load_config()`, `RuntimeSettings`, `get_runtime_settings()` |\n| `core.auth` | `require_login`, `require_admin`, `require_role`, SSO flow (`auth_bp`), user/role CRUD |\n| `core.auth.tokens` | `get_token()`, `pick_token()`, `fresh_token()`, `resolve_default_owner_id()` |\n| `core.auth.credentials` | `CredentialManager` — Fernet-encrypts/decrypts OAuth client credentials |\n| `core.db.public` | DuckDB connections and CRUD helpers |\n| `core.db.private` | Per-owner SQLite session factory |\n| `core.db.models` | `User`, `SiteAdmin` (DuckDB ORM), `Character` (SQLite ORM) |\n| `core.db.reader` | Read helpers — `query_rows()`, `query_one()`, `query_scalar()`, `get_db_file_stats()` |\n| `core.db.writer` | Serialised DuckDB write thread — `db_write()`, `db_executemany()` |\n| `core.db.sde` | SDE cache lookups — `name_from_type_id()`, `region_id_from_system_id()`, etc. |\n| `core.esi` | Re-exports `esi_get()`, `esi_post()`, `esi_request()` — all ESI HTTP |\n| `core.esi.rate` | `ESIRateLimiter` — floating-window token bucket, ETag caching, 429 backoff |\n| `core.esi.registry` | Fetch and parse ESI OpenAPI spec |\n| `core.esi.generated.client` | `execute_operation()`, `fetch_all_pages()` — typed ESI calls |\n| `core.tasks.queue` | `enqueue()`, `get_task()`, `cancel_task()`, `clear_tasks()`, task queue internals |\n| `core.tasks.engine` | `SchedulerEngine`, `get_engine()` — background job scheduler |\n| `core.bus` | Pub/sub event bus — `BusHandler`, topic ring buffers, WebSocket at `/bus` |\n\n### Adding a new private DB model\n\nSubclass `PrivateBase` in `core/db/models/identity.py` (or a new model file imported there):\n\n```python\nfrom core.db.models import PrivateBase\nfrom sqlalchemy import Column, Integer, String\n\nclass MyCharacterData(PrivateBase):\n    __tablename__ = \"my_character_data\"\n    id = Column(Integer, primary_key=True)\n    character_id = Column(Integer, nullable=False)\n    value = Column(String)\n```\n\n`initialize_private_database(owner_id)` will automatically create the table on first use.\n\n### DuckDB Thread Safety\n\nDuckDB connections are **not thread-safe**. Always get a fresh connection per operation:\n\n```python\nimport core.db.public as db\n\ncon = db.connect()\ntry:\n    rows = con.execute(\"SELECT * FROM sde_types WHERE type_id = ?\", [34]).fetchall()\nfinally:\n    con.close()\n```\n\nFor writes that must be serialised (to avoid DuckDB write contention), use the write thread:\n\n```python\nfrom core.db.writer import db_write, db_executemany\n\ndb_write(\"INSERT INTO my_table VALUES (?, ?)\", [1, \"value\"])\ndb_executemany(\"INSERT INTO my_table VALUES (?, ?)\", [(1, \"a\"), (2, \"b\")])\n```\n\n---\n\n## ESI Client \u0026 Code Generation\n\n### Typed Client\n\n```python\nfrom core.esi.generated.client import execute_operation, fetch_all_pages\n\n# Single page\nresult = execute_operation(\"GetMarketsRegionIdOrders\",\n                           path_params={\"region_id\": 10000002},\n                           query_params={\"order_type\": \"all\"})\n\n# All pages automatically\norders = fetch_all_pages(\"GetMarketsRegionIdOrders\",\n                          path_params={\"region_id\": 10000002},\n                          query_params={\"order_type\": \"all\"})\n```\n\n### Regenerating\n\n```powershell\npython build.py              # fetch spec + regenerate all generated packages\npython build.py --force      # force regenerate even if spec is current\npython build.py --spec-only  # only fetch the latest ESI spec\npython build.py --collectors # only regenerate core/esi/personal|corp|public/\n```\n\n\u003e **Do not hand-edit** `core/esi/generated/`, `core/esi/personal/`, `core/esi/corp/`, or `core/esi/public/` — every build run overwrites them.\n\n---\n\n## Security Notes\n\n| File/Path | Risk | Mitigation |\n|-----------|------|-----------|\n| `_publicData/key` | Fernet symmetric key — decrypts all OAuth tokens | **Never commit.** Listed in `.gitignore`. |\n| `_publicData/client_cred` | Encrypted `client_id`/`client_secret` | **Never commit.** Listed in `.gitignore`. |\n| `_privateData/` | Per-user SQLite databases with tokens and character data | **Never commit.** Listed in `.gitignore`. |\n| `config.yaml` | Contains host, port, session secret | **Never commit.** Listed in `.gitignore`. |\n\nAdditional protections:\n\n- **CSRF** — SSO callback validates the `state` parameter via a time-limited `OAuthStateCache` that consumes each token exactly once within a 5-minute window.\n- **SQL injection** — all DuckDB queries use parameterised `con.execute(\"... WHERE id = ?\", [val])`. No user input is interpolated into SQL strings.\n- **Token encryption** — OAuth tokens are Fernet-encrypted before being written to disk. The symmetric key never leaves the server.\n- **Rate limiting** — `esi_req.py` enforces ESI's floating-window token bucket and automatically backs off on 429 responses.\n\n---\n\n## Licensing\n\nThis project is released under the MIT License. See `LICENCE.md` for the full text.\n\nEVE Online and all associated materials are property of CCP hf. This project is not affiliated with or endorsed by CCP hf.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnolieravioli%2Feve_data_framework3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnolieravioli%2Feve_data_framework3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnolieravioli%2Feve_data_framework3/lists"}