{"id":46604562,"url":"https://github.com/ywatanabe1989/openalex-local","last_synced_at":"2026-03-07T17:08:32.553Z","repository":{"id":334738974,"uuid":"1133931782","full_name":"ywatanabe1989/openalex-local","owner":"ywatanabe1989","description":"Local OpenAlex database with 284M+ scholarly works, abstracts, and semantic search for RAG and research assistants","archived":false,"fork":false,"pushed_at":"2026-03-05T14:22:44.000Z","size":374,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-05T15:35:57.755Z","etag":null,"topics":["academic","database","literature-review","openalex","python","rag","research","semantic-search"],"latest_commit_sha":null,"homepage":"https://scitex.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ywatanabe1989.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-14T02:31:27.000Z","updated_at":"2026-02-06T03:24:28.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ywatanabe1989/openalex-local","commit_stats":null,"previous_names":["ywatanabe1989/openalex-local"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/ywatanabe1989/openalex-local","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fopenalex-local","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fopenalex-local/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fopenalex-local/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fopenalex-local/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ywatanabe1989","download_url":"https://codeload.github.com/ywatanabe1989/openalex-local/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fopenalex-local/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30222552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T17:00:40.062Z","status":"ssl_error","status_checked_at":"2026-03-07T17:00:39.026Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["academic","database","literature-review","openalex","python","rag","research","semantic-search"],"created_at":"2026-03-07T17:08:32.027Z","updated_at":"2026-03-07T17:08:32.540Z","avatar_url":"https://github.com/ywatanabe1989.png","language":"Python","funding_links":[],"categories":["Bibliography tools"],"sub_categories":["Online editors"],"readme":"# OpenAlex Local\n\nLocal OpenAlex database with 284M+ scholarly works, abstracts, and semantic search.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/scitex_if_validation.png\" alt=\"SciTeX IF vs JCR Validation\" width=\"600\"/\u003e\n  \u003cbr\u003e\n  \u003cem\u003eSciTeX Impact Factor (OpenAlex) validated against JCR 2024 (r = 0.96, 17,042 journals)\u003c/em\u003e\n\u003c/p\u003e\n\n[![PyPI version](https://badge.fury.io/py/openalex-local.svg)](https://badge.fury.io/py/openalex-local)\n[![Documentation](https://readthedocs.org/projects/openalex-local/badge/?version=latest)](https://openalex-local.readthedocs.io/en/latest/)\n[![Tests](https://github.com/ywatanabe1989/openalex-local/actions/workflows/test.yml/badge.svg)](https://github.com/ywatanabe1989/openalex-local/actions/workflows/test.yml)\n[![Python](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n[![License](https://img.shields.io/badge/license-AGPL--3.0-blue.svg)](LICENSE)\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eWhy OpenAlex Local?\u003c/strong\u003e\u003c/summary\u003e\n\n**Built for the LLM era** - features that matter for AI research assistants:\n\n| Feature | Benefit |\n|---------|---------|\n| **284M Works** | More coverage than CrossRef |\n| **Abstracts** | ~45-60% availability for semantic search |\n| **Concepts \u0026 Topics** | Built-in classification |\n| **Author Disambiguation** | Linked to institutions |\n| **Open Access Info** | OA status and URLs |\n\nPerfect for: RAG systems, research assistants, literature review automation.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eInstallation\u003c/strong\u003e\u003c/summary\u003e\n\n```bash\npip install openalex-local\n```\n\nFrom source:\n```bash\ngit clone https://github.com/ywatanabe1989/openalex-local\ncd openalex-local \u0026\u0026 make install\n```\n\nDatabase setup (~300 GB, ~1-2 days to build):\n```bash\n# Check system status\nmake status\n\n# 1. Download OpenAlex Works snapshot (~300GB)\nmake download-screen  # runs in background\n\n# 2. Build SQLite database\nmake build-db\n\n# 3. Build FTS5 index\nmake build-fts\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003ePython API\u003c/strong\u003e\u003c/summary\u003e\n\n```python\nfrom openalex_local import search, get, count\n\n# Full-text search (title + abstract)\nresults = search(\"machine learning neural networks\")\nfor work in results:\n    print(f\"{work.title} ({work.year})\")\n    print(f\"  Abstract: {work.abstract[:200]}...\")\n    print(f\"  Concepts: {[c['name'] for c in work.concepts]}\")\n\n# Get by OpenAlex ID or DOI\nwork = get(\"W2741809807\")\nwork = get(\"10.1038/nature12373\")\n\n# Count matches\nn = count(\"CRISPR\")\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eCLI\u003c/strong\u003e\u003c/summary\u003e\n\n```bash\nopenalex-local search \"CRISPR genome editing\" -n 5\nopenalex-local search-by-doi W2741809807\nopenalex-local search-by-doi 10.1038/nature12373\nopenalex-local status  # Configuration and database stats\n```\n\nWith abstracts (`-a` flag):\n```\n$ openalex-local search \"neural network\" -n 1 -a\n\nFound 1,523,847 matches in 45.2ms\n\n1. Deep learning for neural networks (2015)\n   OpenAlex ID: W2741809807\n   Abstract: This paper presents a comprehensive overview of deep learning\n   techniques for neural network architectures...\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eHTTP API\u003c/strong\u003e\u003c/summary\u003e\n\nStart the FastAPI server:\n```bash\nopenalex-local relay --host 0.0.0.0 --port 31292\n```\n\nEndpoints:\n```bash\n# Search works (FTS5)\ncurl \"http://localhost:31292/works?q=CRISPR\u0026limit=10\"\n\n# Get by ID or DOI\ncurl \"http://localhost:31292/works/W2741809807\"\ncurl \"http://localhost:31292/works/10.1038/nature12373\"\n\n# Batch lookup\ncurl -X POST \"http://localhost:31292/works/batch\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"ids\": [\"W2741809807\", \"10.1038/nature12373\"]}'\n\n# Database info\ncurl \"http://localhost:31292/info\"\n```\n\nHTTP mode (connect to running server):\n```bash\n# On local machine (if server is remote)\nssh -L 31292:127.0.0.1:31292 your-server\n\n# Python client\nfrom openalex_local import configure_http\nconfigure_http(\"http://localhost:31292\")\n\n# Or via CLI\nopenalex-local --http search \"CRISPR\"\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eMCP Server\u003c/strong\u003e\u003c/summary\u003e\n\nRun as MCP (Model Context Protocol) server:\n```bash\nopenalex-local mcp start\n```\n\nLocal MCP client configuration:\n```json\n{\n  \"mcpServers\": {\n    \"openalex-local\": {\n      \"command\": \"openalex-local\",\n      \"args\": [\"mcp\", \"start\"],\n      \"env\": {\n        \"OPENALEX_LOCAL_DB\": \"/path/to/openalex.db\"\n      }\n    }\n  }\n}\n```\n\nRemote MCP via HTTP:\n```bash\n# On server: start persistent MCP server\nopenalex-local mcp start -t http --host 0.0.0.0 --port 8083\n```\n```json\n{\n  \"mcpServers\": {\n    \"openalex-remote\": {\n      \"url\": \"http://your-server:8083/mcp\"\n    }\n  }\n}\n```\n\nDiagnose setup:\n```bash\nopenalex-local mcp doctor        # Check dependencies and database\nopenalex-local mcp list-tools    # Show available MCP tools\nopenalex-local mcp installation  # Show client config examples\n```\n\nAvailable tools:\n- `search` - Full-text search across 284M+ papers\n- `search_by_id` - Get paper by OpenAlex ID or DOI\n- `enrich_ids` - Batch lookup with metadata\n- `status` - Database statistics\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eSciTeX Impact Factor (OpenAlex)\u003c/strong\u003e\u003c/summary\u003e\n\nWe provide precomputed **SciTeX Impact Factors** calculated from OpenAlex citation data.\nThese follow the JCR formula but use OpenAlex as the data source.\n\n**Validation against JCR 2024** (17,042 matched journals):\n\n| Metric | Value |\n|--------|-------|\n| Pearson r | 0.96 |\n| Spearman ρ | 0.93 |\n| p-value | \u003c 1e-100 |\n\n**Export SciTeX IF:**\n```bash\n# Export all SciTeX IF values\nopenalex-local export-if -o scitex_if.csv\nopenalex-local export-if -o scitex_if.json\n\n# Top 1000\nopenalex-local export-if -o top1000.csv --limit 1000\n```\n\n**Use in search results:**\n```bash\nopenalex-local search \"machine learning\" --with-if\n```\n\n**Formula:**\n```\nSciTeX IF(Year) = Citations in Year to articles from (Year-1, Year-2)\n                  ─────────────────────────────────────────────────────\n                  Citable articles published in (Year-1, Year-2)\n```\n\nNote: \"SciTeX IF\" is our calculation using OpenAlex data.\nIt is not the trademarked \"Journal Impact Factor\" from Clarivate/JCR.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eRelated Projects\u003c/strong\u003e\u003c/summary\u003e\n\n**[crossref-local](https://github.com/ywatanabe1989/crossref-local)** - Sister project with CrossRef data:\n\n| Feature | crossref-local | openalex-local |\n|---------|----------------|----------------|\n| Works | 167M | 284M |\n| Abstracts | ~21% | ~45-60% |\n| Update frequency | Real-time | Monthly |\n| DOI authority | Yes (source) | Uses CrossRef |\n| Citations | Raw references | Linked works |\n| Concepts/Topics | No | Yes |\n| Author IDs | No | Yes |\n| Best for | DOI lookup, raw refs | Semantic search |\n\n**When to use CrossRef**: Real-time DOI updates, raw reference parsing, authoritative metadata.\n**When to use OpenAlex**: Semantic search, citation analysis, topic discovery.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eDocumentation\u003c/strong\u003e\u003c/summary\u003e\n\nFull documentation available at [openalex-local.readthedocs.io](https://openalex-local.readthedocs.io/en/latest/)\n\n- [Installation Guide](https://openalex-local.readthedocs.io/en/latest/installation.html)\n- [Quickstart](https://openalex-local.readthedocs.io/en/latest/quickstart.html)\n- [CLI Reference](https://openalex-local.readthedocs.io/en/latest/cli_reference.html)\n- [HTTP API Reference](https://openalex-local.readthedocs.io/en/latest/http_api.html)\n- [Python API](https://openalex-local.readthedocs.io/en/latest/api/openalex_local.html)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eData Source\u003c/strong\u003e\u003c/summary\u003e\n\nData from [OpenAlex](https://openalex.org/), an open catalog of scholarly works.\nUpdated monthly from their [snapshot](https://docs.openalex.org/download-all-data/openalex-snapshot).\n\n\u003c/details\u003e\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://scitex.ai\"\u003e\u003cimg src=\"docs/scitex-icon-navy-inverted.png\" alt=\"SciTeX\" width=\"40\"/\u003e\u003c/a\u003e\n  \u003cbr\u003e\n  AGPL-3.0\n\u003c/p\u003e\n\n\u003c!-- EOF --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fywatanabe1989%2Fopenalex-local","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fywatanabe1989%2Fopenalex-local","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fywatanabe1989%2Fopenalex-local/lists"}