{"id":31144085,"url":"https://github.com/meanderinghuman/panamapapers-llm-knowledgegraph","last_synced_at":"2026-08-12T06:31:16.298Z","repository":{"id":312814927,"uuid":"1048842909","full_name":"meanderinghuman/PanamaPapers-LLM-KnowledgeGraph","owner":"meanderinghuman","description":"Knowledge graph construction and analysis of the Panama Papers using Large Language Models (LLMs). This project integrates Natural Language Processing (NLP), Graph Databases, and AI-driven entity-relation extraction to uncover hidden connections and insights within complex offshore financial data.","archived":false,"fork":false,"pushed_at":"2025-09-02T05:45:50.000Z","size":582,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-18T14:41:23.761Z","etag":null,"topics":["graphdatabase","knowledge-graph","large-language-models","neo4j","panama-papers"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/meanderinghuman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-02T05:38:18.000Z","updated_at":"2025-09-02T05:56:10.000Z","dependencies_parsed_at":"2025-09-02T07:23:54.179Z","dependency_job_id":"bbf2c9eb-a2a5-49a8-8f28-7c5e997de8c4","html_url":"https://github.com/meanderinghuman/PanamaPapers-LLM-KnowledgeGraph","commit_stats":null,"previous_names":["meanderinghuman/panamapapers-llm-knowledgegraph"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/meanderinghuman/PanamaPapers-LLM-KnowledgeGraph","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meanderinghuman%2FPanamaPapers-LLM-KnowledgeGraph","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meanderinghuman%2FPanamaPapers-LLM-KnowledgeGraph/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meanderinghuman%2FPanamaPapers-LLM-KnowledgeGraph/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meanderinghuman%2FPanamaPapers-LLM-KnowledgeGraph/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/meanderinghuman","download_url":"https://codeload.github.com/meanderinghuman/PanamaPapers-LLM-KnowledgeGraph/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meanderinghuman%2FPanamaPapers-LLM-KnowledgeGraph/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36553024,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-08-06T04:43:03.162Z","status":"online","status_checked_at":"2026-08-12T02:00:07.134Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["graphdatabase","knowledge-graph","large-language-models","neo4j","panama-papers"],"created_at":"2025-09-18T14:10:41.925Z","updated_at":"2026-08-12T06:31:16.283Z","avatar_url":"https://github.com/meanderinghuman.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PanamaPapers-LLM-KnowledgeGraph — Extraction, Visualization \u0026 Querying\n\n\u003e End‑to‑end notebooks that build a Property Graph from unstructured text (Panama Papers) using **LlamaIndex** and query it with **embedding + LLM retrievers**, with interactive graph visualizations via **PyVis**.\n\n\n## ✨ What’s inside\n- **4 extraction strategies** with LlamaIndex property‑graph extractors: _Schema‑based_, _Free‑form_, _Dynamic LLM_, and _Implicit relations_.\n- **Stored indexes** for each strategy so you can reload without recomputation.\n- **Interactive HTML graph views** (PyVis / NetworkX export).\n- **Two concise notebooks**: one to **build** graphs, one to **query** them.\n\n\n## 📦 Repository structure\n```\nPanamaPapers-LLM-KnowledgeGraph/\n  README.md\n  config.py\n  requirements.txt\n  data/\n    panama_papers/\n      panama_papers.pdf\n  notebooks/\n    01_kg_extraction.ipynb\n    02_kg_querying.ipynb\n  outputs/\n    kg_dynamic_llm.html\n    kg_free_form.html\n    kg_implicit.html\n    kg_schema_llm.html\n```\n\n- **data/**: raw input documents (demo uses the Wikipedia article on the Panama Papers).  \n- **notebooks/**: Jupyter workflows for extraction and querying.  \n- **outputs/**: exported, interactive graph visualizations per strategy.  \n- **storage/** (created on first run): persisted LlamaIndex storage per extractor.\n\n\n## 🚀 Quickstart\n### 1) Environment\n- **Python**: 3.9–3.11 recommended\n- Install deps:\n```bash\npython -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\n```\n\u003e `requirements.txt` pins: `pyvis==0.3.2`, `llama_index==0.10.65`.\n\n### 2) Configure OpenAI\nThis repo reads your key from `config.py` and pushes it to the environment in the notebooks.\n\n- Option A — Edit `config.py`:\n```python\nOPENAI_API_KEY = \"sk-...\"\n```\n- Option B — Use environment variable (preferred):\n```bash\nexport OPENAI_API_KEY=sk-...\n# (Optionally leave config.py empty)\n```\n\n\u003e Models used in notebooks (editable):\n\u003e - `LLM_MODEL = \"gpt-4o-mini\"`\n\u003e - `EMBEDDING_MODEL = \"text-embedding-3-small\"`\n\n\n## 🏗️ 01_kg_extraction.ipynb — Build the graph\nOpen the notebook and run cells top‑to‑bottom. Key parameters:\n```python\npath_input_text = \"data/panama_papers\"   # directory with PDFs / text\npath_output_storage = \"storage\"          # per‑extractor storage\npath_output = \"outputs\"                  # exported HTML graphs\n```\n\n### Extraction strategies\nEach section creates/persists an index and exports an interactive graph (HTML) named by the extractor id.\n\n1. **Schema‑based extraction** — `SchemaLLMPathExtractor`\n   - Define a **validation schema** (node/edge types + properties) to constrain triples.\n   - Good when you know your domain entities/relations.\n   - Exports: `outputs/kg_schema_llm.html` and storage under `storage/schema_llm/` (or similar).\n\n2. **Free‑form extraction** — `SimpleLLMPathExtractor`\n   - Let the LLM propose entities/relations directly from text.\n   - Fast to prototype; may be noisy.\n   - Exports: `outputs/kg_free_form.html` + storage.\n\n3. **Dynamic LLM extraction** — `DynamicLLMPathExtractor`\n   - LLM adapts paths it extracts based on context; balances structure and recall.\n   - Used as the default **querying** graph in the second notebook.\n   - Exports: `outputs/kg_dynamic_llm.html` + storage `storage/dynamic_llm/`.\n\n4. **Implicit relation extraction** — `ImplicitPathExtractor`\n   - Surfaces relations that are implied rather than explicit in text.\n   - Exports: `outputs/kg_implicit.html` + storage.\n\n#### What gets saved\nEvery strategy:\n```python\n# persist index storage for reuse\nindex.storage_context.persist(persist_dir=f\"{path_output_storage}/{extractor_name}/\")\n\n# export a PyVis / NetworkX HTML for quick inspection\nindex.property_graph_store.save_networkx_graph(name=f\"{path_output}/kg_{extractor_name}.html\")\n```\n\n\u003e Tip: open the HTML exports directly in your browser to explore nodes/edges interactively.\n\n\n## 🔎 02_kg_querying.ipynb — Ask the graph\nPoint to the storage you want to query (defaults to dynamic LLM):\n```python\npath_storage = \"storage/dynamic_llm\"\nLLM_MODEL = \"gpt-4o-mini\"\nEMBEDDING_MODEL = \"text-embedding-3-small\"\nTEMPERATURE = 0.1\n```\n\n### Load index\n```python\nfrom llama_index.core import StorageContext, load_index_from_storage\nindex = load_index_from_storage(StorageContext.from_defaults(persist_dir=path_storage))\n```\n\n### Available retrievers (used in this repo)\n1. **VectorContextRetriever** — embedding search over graph‑aware chunks/paths.\n2. **LLMSynonymRetriever** — augments retrieval with synonym expansion via LLM.\n\nMinimal pattern (excerpt):\n```python\n# build a sub‑retriever (e.g., VectorContextRetriever) with your models\n# ... configure include_text=True, max_keywords, path_depth, etc.\n\nretriever = index.as_retriever(sub_retrievers=[sub_retriever])\nquery_engine = index.as_query_engine(sub_retrievers=[retriever])\n\nprint(query_engine.query(\n    \"Who were the main people involved in the Panama Papers scandal?\"\n).response)\n```\n\n\u003e The notebook also demonstrates a question on ICIJ involvement.\n\n\n## 🧪 Changing the dataset\nDrop your own PDFs / `.txt` files under `data/your_corpus/` and point `path_input_text` to that folder. The `SimpleDirectoryReader` in LlamaIndex will ingest PDFs and text out of the box.\n\n\n## ⚙️ Tuning \u0026 customization\n- **Schema design**: in the schema‑based section, enumerate node types (e.g., `Person`, `Organization`, `Location`) and allowed relations with validation rules.\n- **Models**: swap `LLM_MODEL` / `EMBEDDING_MODEL` to your preferred OpenAI (or other provider if you adapt the code).\n- **Depth \u0026 keywords**: in retrievers, adjust `path_depth`, `max_keywords`, and whether to `include_text`.\n- **Storage layout**: change `persist_dir` names to keep multiple runs side‑by‑side.\n\n\n## 📈 Outputs you should see\n- `outputs/kg_schema_llm.html`\n- `outputs/kg_free_form.html`\n- `outputs/kg_dynamic_llm.html`\n- `outputs/kg_implicit.html`\n\nOpen any of these in a browser to inspect nodes/edges, hover for details, and zoom/pan.\n\n\n## 🧰 Troubleshooting\n- **`openai.AuthenticationError` or empty results**: ensure `OPENAI_API_KEY` is set and has access to the specified models.\n- **`ModuleNotFoundError: llama_index`**: `pip install -r requirements.txt` inside an active virtualenv.\n- **Graphs don’t render**: some browsers block local JS; use a lightweight server:\n  ```bash\n  python -m http.server 8000\n  # then open http://localhost:8000/outputs/kg_dynamic_llm.html\n  ```\n- **Slow/expensive runs**: use smaller models, limit pages, or sample files in `data/`.\n\n\n## 🗺️ Roadmap ideas (optional)\n- Add a **Neo4j / Memgraph** sink and Cypher querying.\n- Try **KnowledgeGraphRAGRetriever** / **PGQueryEngine** in LlamaIndex for hybrid graph + text RAG.\n- Add **evaluation** (precision/recall of extracted triples) against a small hand‑labeled set.\n- Package the notebook logic into a reusable Python module + CLI.\n\n\n## 🙌 Acknowledgements\nBuilt with [LlamaIndex](https://github.com/run-llama/llama_index) property‑graph tooling and OpenAI models; visualized with PyVis/NetworkX.\n\n\n## 📄 License\nMIT License\n---\n\n### Badges (copy/paste if you want)\n```\n[![Python](https://img.shields.io/badge/python-3.10%2B-blue)]()\n[![LlamaIndex](https://img.shields.io/badge/LlamaIndex-0.10.65-9cf)]()\n[![PyVis](https://img.shields.io/badge/PyVis-0.3.2-lightgrey)]()\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeanderinghuman%2Fpanamapapers-llm-knowledgegraph","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeanderinghuman%2Fpanamapapers-llm-knowledgegraph","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeanderinghuman%2Fpanamapapers-llm-knowledgegraph/lists"}