{"id":51862571,"url":"https://github.com/mameshivaa/x-archive-rag","last_synced_at":"2026-07-24T10:01:35.366Z","repository":{"id":360346792,"uuid":"1249736836","full_name":"mameshivaa/x-archive-rag","owner":"mameshivaa","description":"Local-first MCP/RAG toolkit for turning your X/Twitter archive into a citation-grounded AI memory layer.","archived":false,"fork":false,"pushed_at":"2026-06-18T05:50:26.000Z","size":547,"stargazers_count":0,"open_issues_count":7,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-18T06:27:05.955Z","etag":null,"topics":["ai-memory","archive","claude","llm","local-first","mcp","model-context-protocol","personal-ai","personal-knowledge-management","rag","sqlite","twitter","twitter-archive","x","x-archive"],"latest_commit_sha":null,"homepage":"https://github.com/mameshivaa/x-archive-rag","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mameshivaa.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-26T02:01:07.000Z","updated_at":"2026-06-18T05:50:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mameshivaa/x-archive-rag","commit_stats":null,"previous_names":["mameshivaa/x-archive-rag"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mameshivaa/x-archive-rag","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mameshivaa%2Fx-archive-rag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mameshivaa%2Fx-archive-rag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mameshivaa%2Fx-archive-rag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mameshivaa%2Fx-archive-rag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mameshivaa","download_url":"https://codeload.github.com/mameshivaa/x-archive-rag/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mameshivaa%2Fx-archive-rag/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35837268,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-24T02:00:07.870Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-memory","archive","claude","llm","local-first","mcp","model-context-protocol","personal-ai","personal-knowledge-management","rag","sqlite","twitter","twitter-archive","x","x-archive"],"created_at":"2026-07-24T10:01:34.550Z","updated_at":"2026-07-24T10:01:35.353Z","avatar_url":"https://github.com/mameshivaa.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# x-archive-rag\n\nTurn your X/Twitter archive into a local AI memory layer.\n\n[![Release](https://img.shields.io/github/v/release/mameshivaa/x-archive-rag?label=release)](https://github.com/mameshivaa/x-archive-rag/releases)\n[![CI](https://github.com/mameshivaa/x-archive-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/mameshivaa/x-archive-rag/actions)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n[![GitHub stars](https://img.shields.io/github/stars/mameshivaa/x-archive-rag?style=social)](https://github.com/mameshivaa/x-archive-rag/stargazers)\n\nUnlike uploading a ZIP to ChatGPT, `x-archive-rag` converts your archive into reusable local infrastructure: SQLite storage, repeatable retrieval, citation IDs, persona profiling, grounded draft prompts, a local web UI, and MCP tools that AI clients can query without receiving your whole archive.\n\nThe default workflow keeps your archive local. You decide when to export data or send a small retrieved evidence set to an OpenAI-compatible model.\n\nIf this is useful for your personal AI or MCP workflow, starring the repository helps other people find it: \u003chttps://github.com/mameshivaa/x-archive-rag\u003e\n\n## What is technically different?\n\n`x-archive-rag` is not a bigger prompt. It is a small local data layer for your archive.\n\n| Capability | Why it matters |\n| --- | --- |\n| Local SQLite index | Import once, query repeatedly, inspect or back up the database. |\n| Keyword + semantic + hybrid retrieval | Ask narrow questions without sending the whole archive every time. |\n| Citation-grounded outputs | Search and draft prompts preserve tweet IDs such as `T1`. |\n| Persona profile as separate state | Writing patterns are computed once and reused with retrieved evidence. |\n| CLI, local web UI, export, and MCP | The same archive can serve humans, scripts, and AI clients. |\n| Read-only MCP tools | Claude or another MCP client can retrieve evidence without posting to X or calling a remote model. |\n\n## Local web UI\n\n![Local web UI showing archive stats, search, and draft prompt controls](docs/assets/web-ui.png)\n\nRun it locally after importing an archive:\n\n```bash\nx-archive-rag web --db ./memory.sqlite --open\n```\n\nTry the UI with the included sample archive before touching private data:\n\n```bash\ngit clone https://github.com/mameshivaa/x-archive-rag.git\ncd x-archive-rag\npython3 -m pip install -e .\nx-archive-rag demo --open\n```\n\n## Terminal demo\n\n![Terminal demo showing ingest, search, draft, and MCP usage](docs/assets/terminal-demo.svg)\n\n## What you can do\n\n1. Import your official X archive ZIP or extracted directory into a local SQLite database.\n2. Search it with keyword, semantic, or hybrid retrieval.\n3. Generate a persona profile from your own posts.\n4. Build citation-grounded draft prompts that show the tweets used as evidence.\n5. Expose the same local memory to AI clients through MCP tools.\n\n## Interface\n\nThe main interface is a small CLI with nine commands:\n\n- `ingest` turns an archive into a reusable local database.\n- `search` retrieves relevant posts with citation IDs.\n- `profile` summarizes recurring topics, tone, and writing patterns.\n- `draft` builds a grounded prompt from retrieved evidence.\n- `mcp` exposes the same memory to AI clients that support MCP.\n- `web` runs a local browser UI for search, profile inspection, and draft prompts.\n- `demo` creates a sample database and can open the local web UI.\n- `doctor` checks whether a database is ready for search, draft, and MCP use.\n- `export` writes imported tweets as JSONL or Markdown for inspection and reuse.\n\nText output is designed for humans. JSON and Markdown output are designed for scripts, agents, and repeatable workflows.\n\n## Status\n\nAlpha. The current release is intentionally small:\n\n- Import `tweets.js` / `tweet.js` from an extracted archive directory or archive ZIP, including common long-form tweet text fields and expanded URL entities.\n- Store normalized tweets in SQLite.\n- Search with SQLite FTS5, dependency-free semantic retrieval, or hybrid retrieval.\n- Generate a persona profile from your own posts.\n- Build grounded RAG prompts with cited tweets.\n- Expose archive search and draft prompts through a minimal MCP stdio server.\n- Optionally call OpenAI-compatible chat completions using `OPENAI_API_KEY`.\n\n## Why\n\nMost archive tools help you search old posts. This project focuses on a different workflow:\n\n\u003e \"Given what I have said before, draft something that sounds like me, while showing the evidence.\"\n\nThe assistant is designed for drafting and reflection, not impersonation or automated posting.\n\n## Why not just upload the ZIP to ChatGPT?\n\nUploading a ZIP is a one-off context dump. `x-archive-rag` is a reusable memory layer.\n\n| ZIP upload | `x-archive-rag` |\n| --- | --- |\n| Sends the archive to a remote chat session. | Imports the archive into a local SQLite database. |\n| Relies on one model pass over a large attachment. | Retrieves only relevant tweets per question. |\n| Hard to reproduce or inspect later. | Keeps repeatable search, export, and doctor checks. |\n| Evidence can disappear into a summary. | Preserves citation IDs such as `T1` in search and draft output. |\n| Works only inside that chat session. | Exposes CLI, JSON, Markdown export, and MCP tools. |\n\nFor a deeper technical comparison, see [docs/comparison.md](docs/comparison.md).\n\n## Install\n\nFrom a local checkout:\n\n```bash\npython3 -m pip install -e .\n```\n\nNo runtime dependencies are required.\n\n## Quick start\n\nWant to test without private archive data first?\n\n```bash\nx-archive-rag demo --open\n```\n\nThe shortest path is:\n\n```bash\nx-archive-rag ingest ./twitter-archive.zip --db ./memory.sqlite\nx-archive-rag profile --db ./memory.sqlite\nx-archive-rag search \"personal AI memory\" --db ./memory.sqlite --mode hybrid\nx-archive-rag draft \"What do I think about OSS personal AI?\" --db ./memory.sqlite\nx-archive-rag web --db ./memory.sqlite --open\n```\n\nUse the sections below when you need more control over import format, date filters, exports, OpenAI-compatible calls, or MCP.\n\nWant the Claude Desktop Extension artifact instead? Download the latest `.mcpb` from [Releases](https://github.com/mameshivaa/x-archive-rag/releases/latest) and follow [docs/mcpb.md](docs/mcpb.md).\n\nAI coding agents and MCP marketplace reviewers can start from [llms.txt](llms.txt) or [llms-install.md](llms-install.md).\n\nPrefer a browser surface? Run the local web UI and follow [docs/web.md](docs/web.md).\n\nImport an extracted X archive directory:\n\n```bash\nx-archive-rag ingest ./twitter-archive --db ./x-archive-rag.sqlite\n```\n\nRetweets are excluded by default so profiles and drafts focus on your own statements. Add `--include-retweets` if you want to index retweets too.\n\nOr import the original ZIP:\n\n```bash\nx-archive-rag ingest ./twitter-archive.zip --db ./x-archive-rag.sqlite\n```\n\nSearch your archive:\n\n```bash\nx-archive-rag search \"AI agents and personal tools\" --db ./x-archive-rag.sqlite\n```\n\nSearch and draft outputs include citation IDs such as `T1`, so generated text can refer back to specific retrieved posts.\n\nExample search output:\n\n```text\n[T1] 2024-01-01T12:00:00+00:00 https://x.com/i/web/status/1001\nPersonal AI should be local-first and cite the memories it uses.\n```\n\nJSON output is available for scripts:\n\n```bash\nx-archive-rag search \"AI agents and personal tools\" \\\n  --db ./x-archive-rag.sqlite \\\n  --json\n```\n\nSemantic and hybrid search are available without external services:\n\n```bash\nx-archive-rag search \"grounded memory for drafts\" \\\n  --db ./x-archive-rag.sqlite \\\n  --mode hybrid\n```\n\nLimit retrieval to a time range when older views should not influence a draft:\n\n```bash\nx-archive-rag search \"personal AI\" \\\n  --db ./x-archive-rag.sqlite \\\n  --since 2024-01-01 \\\n  --until 2024-12-31\n```\n\nBuild a persona profile:\n\n```bash\nx-archive-rag profile --db ./x-archive-rag.sqlite\n```\n\nProfiles include the number of tweets analyzed and the covered date range.\n\nDraft with retrieved context:\n\n```bash\nx-archive-rag draft \"What do I think about OSS personal AI?\" \\\n  --db ./x-archive-rag.sqlite \\\n  --mode hybrid \\\n  --output ./draft.md\n```\n\nThe draft command builds an evidence-grounded prompt. It does not call a remote model unless `--call-openai` is set.\n\n```text\nRetrieved tweets:\n[T1] 2024-01-01T12:00:00+00:00 https://x.com/i/web/status/1001\nPersonal AI should be local-first and cite the memories it uses.\n\nTask:\nDraft a response that is grounded in the retrieved tweets and consistent with the persona profile.\n```\n\nCall an OpenAI-compatible model:\n\n```bash\nexport OPENAI_API_KEY=\"...\"\nx-archive-rag draft \"What do I think about OSS personal AI?\" \\\n  --db ./x-archive-rag.sqlite \\\n  --call-openai \\\n  --model gpt-4.1-mini\n```\n\nUse an OpenAI-compatible endpoint:\n\n```bash\nx-archive-rag draft \"What do I think about local-first AI?\" \\\n  --db ./x-archive-rag.sqlite \\\n  --call-openai \\\n  --model local-model \\\n  --base-url http://localhost:8000/v1\n```\n\nRun as an MCP stdio server:\n\n```bash\nx-archive-rag mcp --db ./x-archive-rag.sqlite\n```\n\nAvailable MCP tools:\n\n- `search_archive`\n- `draft_from_archive`\n- `get_persona_profile`\n\nCheck database readiness:\n\n```bash\nx-archive-rag doctor --db ./x-archive-rag.sqlite\n```\n\nExport imported tweets for inspection or backup:\n\n```bash\nx-archive-rag export \\\n  --db ./x-archive-rag.sqlite \\\n  --format jsonl \\\n  --output ./tweets.jsonl\n```\n\nSee [docs/cli.md](docs/cli.md) for the full CLI reference.\n\n## Design\n\nThe pipeline is deliberately simple:\n\n```text\nX archive ZIP or directory\n  -\u003e tweet normalization\n  -\u003e persistent SQLite memory + FTS5 + lightweight semantic vectors\n  -\u003e per-prompt retrieval with citation IDs\n  -\u003e persona profile\n  -\u003e grounded draft prompt, optional LLM call, or MCP tool response\n```\n\nFuture releases can add external embedding providers, thread reconstruction, and local model providers without changing the core import/search model. The current release already includes CLI, MCP, export, and local web UI surfaces over the same SQLite memory layer.\n\n## Safety and ethics\n\nUse this with your own archive or with explicit permission. The project is meant to help a person understand and draft from their own history. It should not be used to impersonate someone, automate posting as someone else, or hide AI-generated output as human-authored speech.\n\nGenerated drafts should cite the retrieved posts they used. Treat the output as a draft that needs human review.\n\nSee [docs/privacy.md](docs/privacy.md) for local data flow and remote-call details.\n\n## Privacy Policy\n\n`x-archive-rag` is local-first. Archive ZIPs, extracted archive directories, SQLite databases, persona profiles, draft prompts, and exports stay on the user's machine unless the user explicitly exports them or uses an optional remote LLM call.\n\nMCP mode is read-only: it reads from the configured local SQLite database and returns retrieved tweets, the stored persona profile, or grounded draft prompts to the connected AI client. It does not post to X and does not call a remote LLM provider.\n\nThe full privacy policy is in [docs/privacy.md](docs/privacy.md).\n\n## Development\n\nRun the full local verification suite:\n\n```bash\nmake verify\n```\n\nRun the CLI without installing:\n\n```bash\nPYTHONPATH=src python3 -m x_archive_rag --help\nPYTHONPATH=src python3 -m x_archive_rag --version\n```\n\nRun the sample archive smoke test:\n\n```bash\nmake smoke\n```\n\nSee [docs/architecture.md](docs/architecture.md) for the module layout and planned extension points.\nSee [docs/cli.md](docs/cli.md) for the full CLI reference.\nSee [docs/mcp.md](docs/mcp.md) for MCP setup examples.\nSee [docs/mcpb.md](docs/mcpb.md) for the Claude Desktop Extension bundle.\nSee [docs/web.md](docs/web.md) for the local browser UI.\nSee [docs/privacy.md](docs/privacy.md) for local data flow and remote-call details.\nSee [docs/directory-submission.md](docs/directory-submission.md) for Claude and ChatGPT directory submission notes.\nSee [docs/launch.md](docs/launch.md) for public launch copy and sharing assets.\nSee [server.json](server.json) for official MCP Registry metadata.\nSee [docs/release/checklist.md](docs/release/checklist.md) before publishing a release.\nSee [docs/release/publishing.md](docs/release/publishing.md) for first public repository steps.\nSee [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md), [CONTRIBUTING.md](CONTRIBUTING.md), and [SECURITY.md](SECURITY.md) before opening issues or pull requests.\nSee [CITATION.cff](CITATION.cff) if you need citation metadata.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmameshivaa%2Fx-archive-rag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmameshivaa%2Fx-archive-rag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmameshivaa%2Fx-archive-rag/lists"}