{"id":28185824,"url":"https://github.com/johnforfar/pythia-v2","last_synced_at":"2025-05-16T06:12:49.174Z","repository":{"id":291996866,"uuid":"979470440","full_name":"johnforfar/pythia-v2","owner":"johnforfar","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-07T15:15:19.000Z","size":6676,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-07T16:29:14.118Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/johnforfar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-07T15:00:43.000Z","updated_at":"2025-05-07T15:15:24.000Z","dependencies_parsed_at":"2025-05-07T16:37:00.677Z","dependency_job_id":"ddcef971-c8d3-4b53-b4cf-a16fcf1735aa","html_url":"https://github.com/johnforfar/pythia-v2","commit_stats":null,"previous_names":["johnforfar/pythia-v2"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnforfar%2Fpythia-v2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnforfar%2Fpythia-v2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnforfar%2Fpythia-v2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnforfar%2Fpythia-v2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/johnforfar","download_url":"https://codeload.github.com/johnforfar/pythia-v2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478054,"owners_count":22077679,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-16T06:12:42.893Z","updated_at":"2025-05-16T06:12:49.168Z","avatar_url":"https://github.com/johnforfar.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pythia V2 - Open Source Query Engine for Web3\n\nPythia V2 aims to be an easy-to-use data analytics tool focused on providing natural language querying capabilities for on-chain and off-chain data. This version is a more modular and modernized iteration of the original Pythia concept.\n\n## Project Components\n\nThe project is structured as a monorepo containing the following key services:\n\n1.  **`openmesh-collectors`**:\n    *   A Python-based data collection system.\n    *   Responsible for gathering Web3 financial data (and potentially other data) from various on-chain sources (e.g., Ethereum) and off-chain exchanges.\n    *   Includes services for Apache Kafka and a specific `ethereum-collector` service.\n    *   The Ethereum collector uses the `runner.py` script to fetch on-chain data.\n    *   Data is published to Kafka topics (e.g., `ethereum_blocks`, `ethereum_transactions`).\n    *   The goal is to eventually feed processed data from Kafka into a PostgreSQL database.\n\n2.  **`pythia-backend`**:\n    *   A Node.js application built with the NestJS framework and TypeScript.\n    *   Provides the API endpoints for the frontend.\n    *   Handles natural language queries (NLQ) from users.\n    *   Translates NLQ into SQL queries using a Large Language Model (LLM).\n    *   Executes SQL queries against the PostgreSQL database.\n    *   Uses Prisma as its ORM.\n    *   **(For `openmesh-collectors`) Ethereum RPC Node URL:** If you intend to run the Ethereum collector, you'll need RPC URLs (HTTP and WebSocket) from a provider like Alchemy, Infura, QuickNode, etc. These are configured in `openmesh-collectors/keys/.env`.\n    *   **(For `pythia-backend`) LLM API Key/Configuration:** You'll need an API key for your chosen LLM (e.g., ASI-1, or configuration for a local LLM). Configured in `pythia-backend/.env`.\n\n3.  **`pythia-frontend`**:\n    *   A Next.js (React) application built with TypeScript.\n    *   Provides the user interface for interacting with Pythia.\n    *   Allows users to input natural language queries and view the results (tables, charts, etc.).\n\n4.  **PostgreSQL Database**:\n    *   The central data store for:\n        *   Application-specific data managed by `pythia-backend` (user accounts, chat history, etc.).\n        *   Processed on-chain and off-chain data ingested by `openmesh-collectors` (this part of the data pipeline is under development).\n\n5.  **Apache Kafka**:\n    *   Used by `openmesh-collectors` as a distributed streaming platform to ingest and process data before it lands in PostgreSQL.\n\n## Local Development Setup\n\nThis project uses Docker and Docker Compose, orchestrated by a root `Makefile`, to manage local development services.\n\n**Prerequisites:**\n\n*   **Docker Desktop:** Ensure Docker Desktop is installed and running. The `Makefile` uses `docker compose` (with a space).\n*   **Git:** For cloning the repository.\n*   **(For `openmesh-collectors`) Ethereum RPC Node URL:** If you intend to run the Ethereum collector, you'll need RPC URLs (HTTP and WebSocket) from a provider like Alchemy, Infura, QuickNode, etc. These are configured in `openmesh-collectors/keys/.env`.\n*   **(For `pythia-backend`) LLM API Key/Configuration:** You'll need an API key for your chosen LLM (e.g., ASI-1, or configuration for a local LLM). Configured in `pythia-backend/.env`.\n\n**Steps to Run Locally:**\n\n1.  **Clone the Repository:**\n    ```bash\n    # git clone \u003crepository-url\u003e\n    cd PYTHIA\n    ```\n\n2.  **Configure `openmesh-collectors`:**\n    *   Navigate to `openmesh-collectors/keys/`.\n    *   Create a `.env` file (if it doesn't exist) by copying from a template or manually.\n    *   Add Kafka connection details (these are for the local Docker Kafka instance run by `docker-compose.yml` in the same directory):\n        ```ini\n        KAFKA_BOOTSTRAP_SERVERS=broker01:9092,broker02:9092,broker03:9092\n        SCHEMA_REGISTRY_URL=http://schema-registry:8081\n        ```\n    *   If running the Ethereum collector, add your RPC URLs:\n        ```ini\n        ETHEREUM_NODE_HTTP_URL=\u003cyour_ethereum_rpc_http_url\u003e\n        ETHEREUM_NODE_WS_URL=\u003cyour_ethereum_rpc_ws_url\u003e\n        ```\n    *   **Note:** This `.env` file is copied into the `ethereum-collector` Docker image during the build process (see `openmesh-collectors/Dockerfile`) and read by the Python scripts directly.\n\n3.  **Configure `pythia-backend` (LLM API Key):**\n    *   The `Makefile` will create an initial `pythia-backend/.env` file.\n    *   After the first `make up` run (or by creating it manually beforehand), edit `pythia-backend/.env` and add the necessary environment variables for your chosen LLM. For example, for an OpenAI-API compatible endpoint (like ASI-1 or a local Ollama server):\n        ```env\n        # pythia-backend/.env\n        # ... (DATABASE_URL and PYTHIA_PORT will be auto-generated)\n        OPENAI_API_KEY=\"your_llm_api_key\"\n        OPENAI_API_BASE=\"your_llm_api_base_url_if_not_default_openai\" # e.g., https://api.asi1.ai/v1\n        LLM_MODEL_NAME=\"your_chosen_model_name\" # e.g., asi1-mini\n        ```\n\n4.  **Start All Services:**\n    From the root `PYTHIA` directory, run:\n    ```bash\n    make up\n    ```\n    This command will:\n    *   Start PostgreSQL.\n    *   Start Kafka, Zookeeper, and Schema Registry (defined in `openmesh-collectors/docker-compose.yml`).\n    *   Build and start the `ethereum-collector` service (also defined in `openmesh-collectors/docker-compose.yml`).\n    *   Build and start `pythia-backend`.\n    *   Build and start `pythia-frontend`.\n\n5.  **Accessing Services:**\n    *   **Pythia Frontend:** `http://localhost:3111` (or the port configured for the frontend)\n    *   **Pythia Backend API:** `http://localhost:3000` (or the port configured for the backend)\n    *   **PostgreSQL:** Connect via `localhost:5432` (User: `pythiauser`, Pass: `pythiapassword`, DB: `pythiadb`)\n    *   **Schema Registry (Kafka):** `http://localhost:8081`\n\n**Makefile Targets:**\n\n*   `make up`: Starts all defined services (including Kafka and the Ethereum collector).\n*   `make down`: Stops all defined services.\n*   `make logs`: Shows consolidated logs (usually not very readable for multiple services).\n*   `make logs-db`: Follows logs for PostgreSQL.\n*   `make logs-collectors`: Follows logs for Kafka services AND the `ethereum-collector`. To see only collector logs, run `docker compose logs -f ethereum-collector` from the `openmesh-collectors` directory.\n*   `make logs-backend`: Follows logs for `pythia-backend`.\n*   `make logs-frontend`: Follows logs for `pythia-frontend`.\n*   `make clean-db`: Stops PostgreSQL and **deletes its data volume** (use with caution).\n\n**Further Development:**\n\n*   **Data Ingestion:** The `make up` command now starts the `ethereum-collector` service automatically. Other collectors (if developed) would need similar service definitions or manual execution methods.\n*   **Kafka to PostgreSQL Pipeline:** The mechanism to move data collected by `openmesh-collectors` (currently Ethereum data in Kafka topics like `ethereum_blocks`) into queryable tables in PostgreSQL needs to be implemented. This might involve a separate Kafka consumer service or using a tool like Kafka Connect.\n*   **PostgreSQL Schema for Collected Data:** The `prisma/schema.prisma` in `pythia-backend` needs to be extended with table definitions for the Ethereum data (blocks, transactions, logs, transfers) collected by `openmesh-collectors`.\n\nThis project is under active development. ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnforfar%2Fpythia-v2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnforfar%2Fpythia-v2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnforfar%2Fpythia-v2/lists"}