{"id":35699784,"url":"https://github.com/queelius/ctk","last_synced_at":"2026-06-09T18:00:40.722Z","repository":{"id":291000034,"uuid":"976255641","full_name":"queelius/ctk","owner":"queelius","description":"Conversation Toolkit - Unified CLI for managing AI conversations from ChatGPT, Claude, Gemini, and more. Import, search, export, and chat with your conversation history.","archived":false,"fork":false,"pushed_at":"2026-05-22T00:32:36.000Z","size":28444,"stargazers_count":11,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-05-22T09:53:40.854Z","etag":null,"topics":["ai","chatgpt","claude","cli","conversation","llm","python","toolkit"],"latest_commit_sha":null,"homepage":"https://queelius.github.io/ctk/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/queelius.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":".zenodo.json","notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-01T19:26:37.000Z","updated_at":"2026-05-22T00:32:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"fe966edf-6f96-4385-935a-f8fd55245c36","html_url":"https://github.com/queelius/ctk","commit_stats":null,"previous_names":["queelius/ctk"],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/queelius/ctk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queelius%2Fctk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queelius%2Fctk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queelius%2Fctk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queelius%2Fctk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/queelius","download_url":"https://codeload.github.com/queelius/ctk/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/queelius%2Fctk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34118757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","chatgpt","claude","cli","conversation","llm","python","toolkit"],"created_at":"2026-01-06T01:52:33.006Z","updated_at":"2026-06-09T18:00:40.715Z","avatar_url":"https://github.com/queelius.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conversation Toolkit (CTK)\n\nA plugin-based system for managing AI conversations from multiple providers. Import, store, search, branch, and export your conversations in a unified tree format while preserving provider-specific details.\n\n## Quick Start\n\n```bash\n# Setup (one-time)\nmake install\n\n# Import conversations from any supported provider\nctk import chatgpt_export.json --db my_chats\nctk import claude_export.json   --db my_chats --format anthropic\n\n# Open the TUI on the database (primary interface)\nctk --db my_chats\n\n# Bulk-export for fine-tuning, archiving, or sharing\nctk export training.jsonl --db my_chats --format jsonl\n```\n\n`ctk` with no subcommand opens the full-screen Textual TUI on the database in `~/.ctk/config.json`'s `database.default_path` (or the path passed via `--db`). Everything an interactive user does (chat, search, branch trees, organize, edit) lives in the TUI. The CLI subcommands are kept small and scriptable: import, export, query, sql, and a handful of admin operations.\n\n## Key Features\n\n- **Universal Tree Format**: every conversation is a tree. Linear chats are single-path trees, branching conversations (e.g., ChatGPT regenerations) preserve all paths.\n- **Tree Primitive Algebra**: six primitives (delete, delete_subtree, prune_to, copy, copy_subtree, graft) compose into every higher-level operation: fork, branch, clone, snapshot, detach, promote.\n- **Plugin Architecture**: importers and exporters auto-discover via Python imports. Adding a new format is one file.\n- **Multiple LLM Backends**: named provider profiles for OpenAI, Azure, OpenRouter, vLLM, llama.cpp, LM Studio, Ollama, or any other OpenAI-compatible endpoint. Switch live via `/provider` in the TUI.\n- **Inline Images**: terminal image rendering via `textual-image` (Sixel, Kitty TGP, or Halfcell), automatic protocol detection.\n- **Tool Calling / MCP**: tools group under named virtual MCP providers (`ctk.builtin`, `ctk.network`). The LLM can search, list, find similar conversations, etc., directly during chat.\n- **MCP Server**: ctk also runs as a real MCP server (`python -m ctk.mcp_server`) so external clients can use the same surface.\n- **SQLite + FTS5**: local, fast, searchable. The \"database\" is a directory containing `conversations.db` and an associated `media/` folder for image attachments.\n\n## Installation\n\n```bash\ngit clone https://github.com/queelius/ctk.git\ncd ctk\nmake install\n```\n\nOr manually:\n\n```bash\npython3 -m venv .venv\nsource .venv/bin/activate\npip install -e .\n```\n\nPip install (when published):\n\n```bash\npip install conversation-tk\n```\n\n## CLI Surface\n\nThe full top-level command list is intentionally small:\n\n| Command | Purpose |\n|---|---|\n| `ctk` (no args) | Open the TUI on the configured DB |\n| `ctk tui` | Same, alias for muscle memory |\n| `ctk import` | Bulk import conversation exports |\n| `ctk export` | Bulk export to file |\n| `ctk query` | Filter / search with formatted output (table, json, csv) |\n| `ctk sql` | Read-only SQL on the DB |\n| `ctk db` | Maintenance: init, info, vacuum, backup, merge, diff, intersect, filter, split, dedupe, validate |\n| `ctk net` | Build embeddings + similarity graph (analytical queries are MCP tools, used from the TUI) |\n| `ctk auto-tag` | Bulk LLM-driven tagging |\n| `ctk llm` | Provider config: providers, models, test |\n| `ctk config` | Edit `~/.ctk/config.json` |\n\nPer-conversation, per-library, chat-REPL, and ad-hoc network analysis subcommands all moved into the TUI as bindings, slash commands, or MCP tool calls.\n\n## Importing Conversations\n\n### ChatGPT / OpenAI\n\nExport from [chatgpt.com/settings](https://chatgpt.com/settings) \u003e Data Controls \u003e Export.\n\n```bash\nctk import conversations.json --db chats\nctk import chatgpt_export.json --db chats --format openai --tags \"work,2024\"\n```\n\n### Claude / Anthropic\n\nExport from Claude's settings.\n\n```bash\nctk import claude_export.json --db chats --format anthropic\n```\n\n### GitHub Copilot\n\n```bash\nctk import ~/.vscode/workspaceStorage --db chats --format copilot\n```\n\n### Google Gemini\n\n```bash\nctk import gemini_export.json --db chats --format gemini\n```\n\n### Generic JSONL (for fine-tuning datasets, local LLMs, etc.)\n\n```bash\nctk import training_data.jsonl --db chats --format jsonl\nfor file in *.jsonl; do\n  ctk import \"$file\" --db chats --format jsonl\ndone\n```\n\n### Filesystem-Based Coding Agents\n\n```bash\n# Auto-detect coding agent data (Copilot, Claude Code, Cursor, etc.)\nctk import ~/.local/share/agent-storage --db chats --format filesystem_coding\n```\n\n## Querying from the Shell\n\n```bash\n# List conversations as a Rich table\nctk query --db chats\n\n# Filter and order\nctk query --db chats --starred --limit 10\nctk query --db chats --filter-source ChatGPT --filter-model gpt-4o\n\n# JSON or CSV for piping\nctk query --db chats --format json | jq '.[].title'\nctk query --db chats --format csv \u003e inventory.csv\n\n# Read-only SQL when the query language isn't enough\nctk sql \"SELECT source, COUNT(*) FROM conversations GROUP BY source\" --db chats\n```\n\n## The TUI\n\n```bash\nctk --db chats   # or just `ctk` if database.default_path is set\n```\n\nThe TUI has a tabbed sidebar (All / Starred / Pinned / Recent / Archived) plus a main pane with focusable message bubbles and a multi-line chat input.\n\n### Bindings\n\n| Key | Action |\n|---|---|\n| `q` | Quit |\n| `/` | Search overlay |\n| `Esc` | Cancel search / dismiss modal |\n| `Ctrl+R` | Refresh |\n| `Ctrl+N` | New conversation |\n| `Ctrl+H` | Help modal (bindings, slash commands, MCP providers) |\n| `Ctrl+S` | Toggle star |\n| `Ctrl+G` | System prompt modal |\n| `Ctrl+O` | Attach file modal |\n| `Ctrl+L` | Load more conversations into the sidebar |\n| `Tab` / `Shift+Tab` | Move focus between message bubbles |\n| `Ctrl+F` | Fork at focused message (truncate descendants and siblings) |\n| `Ctrl+B` | Branch at focused message (preserve full tree, new id) |\n| `Ctrl+D` | Delete subtree at focus (with confirm) |\n| `Ctrl+E` | Extract subtree at focus into a new conversation |\n| `Ctrl+P` | Promote focused path (drop sibling branches, with confirm) |\n| `[` / `]` | Switch to previous / next sibling at focused message |\n\n`Ctrl+H` opens a modal listing the live registry of bindings, slash commands, and MCP providers, so the canonical reference is always current.\n\n### Slash Commands\n\nType any of these in the chat input. They route through the dispatcher before reaching the LLM.\n\n| Command | Effect |\n|---|---|\n| `/help` | List all slash commands |\n| `/mcp` | List MCP tool providers and their tools |\n| `/model [name]` | Show or switch the chat model (also displays profile + base_url) |\n| `/provider [name]` | List provider profiles or switch to one (rebuilds the active provider) |\n| `/system [text]` | Show or set the system prompt |\n| `/title \u003ctext\u003e` | Rename the current conversation |\n| `/star`, `/pin`, `/archive` | Toggle the corresponding flag |\n| `/tag \u003cname\u003e...`, `/untag \u003cname\u003e` | Add or remove tags |\n| `/clone` | Duplicate the current conversation as a sibling |\n| `/snapshot` | Save a dated snapshot |\n| `/delete` | Delete the current conversation entirely (with confirm) |\n| `/delete-subtree` | Delete the focused subtree (with confirm) |\n| `/extract` | Copy focused subtree as a new conversation |\n| `/detach` | Move focused subtree out as a new conversation |\n| `/promote` | Make the focused message's path the only path |\n| `/graft \u003cconv-id-or-prefix\u003e` | Attach another conversation under the focused message |\n| `/fork`, `/branch` | Same as Ctrl+F / Ctrl+B but at the path tail |\n| `/attach \u003cpath\u003e` | Attach a file as a system message |\n| `/export \u003cpath\u003e [format]` | Export the current conversation |\n| `/sql \u003cquery\u003e` | Read-only SQL query |\n| `/clear` | Reset to a new empty conversation |\n| `/quit` | Exit the TUI |\n\n## Tree Operations: The Algebra\n\nEvery fork / branch / clone / snapshot / detach / promote operation in the TUI decomposes into one of six primitives. Five live on the in-memory `ConversationTree`, the sixth is DB-level.\n\n| # | Primitive | Effect |\n|---|---|---|\n| 1 | `db.delete_conversation(id)` | Remove the conversation from the database |\n| 2 | `tree.delete_subtree(n)` | Drop node `n` and all descendants |\n| 3 | `tree.prune_to(n)` | Keep only the ancestor chain of `n` |\n| 4 | `tree.copy()` | Full duplicate, new conversation id |\n| 5 | `tree.copy_subtree(n)` | New conversation rooted at `n`'s subtree |\n| 6 | `tree.graft(n, other)` | Attach a copy of `other` under `n`, fresh ids |\n\nDerived operations are compositions: fork = `copy().prune_to(n)`, detach = `copy_subtree(n)` then `delete_subtree(n)`, promote = `prune_to(leaf_of_path)`, snapshot = `copy()` plus a dated title.\n\nThis algebra is the design contract: when you need a new tree-shape operation, write it as a composition rather than reaching into `message_map` directly. Tests in `tests/unit/test_tree_primitives.py` prove the primitives behave correctly under every shape (single message, deep chains, branching, missing nodes, root, leaf).\n\n## LLM Backends and Provider Profiles\n\nCTK ships one provider implementation, `OpenAIProvider`, wrapping the official `openai` SDK. It speaks the OpenAI chat-completions protocol, so any compatible endpoint works:\n\n- The real OpenAI API\n- Azure, OpenRouter\n- Local: vLLM, llama.cpp server, LM Studio, Ollama (`http://localhost:11434/v1`)\n- Custom remote inference rigs\n\n### Named Profiles\n\nDefine multiple endpoints in `~/.ctk/config.json` and switch between them at startup or live in the TUI.\n\n```json\n{\n  \"providers\": {\n    \"default\": \"muse\",\n    \"openai\":  {\"base_url\": \"https://api.openai.com/v1\", \"default_model\": \"gpt-5\"},\n    \"muse\":    {\"base_url\": \"http://muse.lan:8000/v1\",   \"default_model\": \"qwen3-omni\"},\n    \"ollama\":  {\"base_url\": \"http://localhost:11434/v1\", \"default_model\": \"llama3.1:70b\"}\n  }\n}\n```\n\nIn the TUI:\n\n```\n/provider              # list profiles, marks the active one with *\n/provider muse         # switch live; rebuilds the provider, refreshes status\n/model                 # shows current model + profile + base_url\n/model gpt-4o          # switch model on the active profile\n```\n\nCLI:\n\n```bash\nctk --provider ollama --db chats         # open the TUI on a specific profile\nctk --provider muse --base-url http://x/v1   # ad-hoc override\nctk llm test --provider openai           # check connectivity\n```\n\nAPI keys read from `\u003cPROFILE\u003e_API_KEY` env var first (so `MUSE_API_KEY` works), falling back to `OPENAI_API_KEY`, then to the config file (with a warning).\n\n## Exporting\n\n### For Fine-Tuning (JSONL)\n\n```bash\nctk export training.jsonl --db chats --format jsonl\nctk export selected.jsonl --db chats --format jsonl --ids conv1 conv2\nctk export filtered.jsonl --db chats --format jsonl --filter-source ChatGPT\nctk export sanitized.jsonl --db chats --format jsonl --sanitize\n```\n\nThe `--sanitize` flag scrubs API keys, passwords, tokens, SSH keys, database URLs, and credit-card-like numbers before writing.\n\n### Markdown\n\n```bash\nctk export all.md       --db chats --format markdown   # single file\nctk export docs/        --db chats --format markdown   # one file per conversation\n```\n\n### Interactive HTML\n\n```bash\nctk export archive.html --db chats --format html\nctk export archive.html --db chats --format html --media-dir media\n```\n\nThe HTML export is a self-contained interactive app: branch navigation, search, image gallery, and tree-aware chat continuation against a local LLM endpoint (Ollama, LM Studio, etc.). Reply to any assistant message or continue at the end; new branches save to localStorage. Requires serving via HTTP for chat to work (`python -m http.server`).\n\n### Hugo (Static Site)\n\n```bash\nctk export content/conversations/ --db chats --format hugo\nctk export content/conversations/ --db chats --format hugo --starred\nctk export content/conversations/ --db chats --format hugo --hugo-organize tags\n```\n\nEach conversation becomes a Hugo page bundle with frontmatter and copied media files. `--hugo-organize` accepts `none`, `tags`, `source`, or `date`.\n\n### Path Selection\n\nFor a tree with multiple paths (regenerations, branches), pick which one to export:\n\n```bash\nctk export out.jsonl --db chats --path-selection longest   # default\nctk export out.jsonl --db chats --path-selection first\nctk export out.jsonl --db chats --path-selection last\n```\n\n## Tree Structure\n\nCTK stores all conversations as trees. The flexibility this enables (forking to explore alternatives, regenerations as siblings, grafting context from other conversations) is the core of the model.\n\nLinear conversation:\n\n```\nUser: \"What is Python?\"\n  Assistant: \"Python is a programming language...\"\n    User: \"How do I install it?\"\n      Assistant: \"You can install Python by...\"\n```\n\nBranching conversation (a regenerated assistant turn plus follow-ups on each):\n\n```\nUser: \"Write a poem\"\n  Assistant (v1): \"Roses are red...\"\n  Assistant (v2): \"In fields of gold...\"\n    User: \"Make it longer\"\n      Assistant: \"In fields of gold, where sunshine...\"\n```\n\nIn the TUI, branches are visible at every node with multiple children. `[` and `]` switch siblings.\n\n## Database Operations\n\n```bash\n# Combine databases\nctk db merge source1 source2 --output merged\n\n# Compare two databases\nctk db diff a b\n\n# Filter into a new database\nctk db filter all_chats --output work_only --tags \"work\"\nctk db filter all_chats --output starred --starred\n\n# Maintenance\nctk db init \u003cdir\u003e\nctk db info \u003cdir\u003e\nctk db vacuum \u003cdir\u003e\nctk db backup \u003cdir\u003e --output backup-2026-04.db\nctk db dedupe \u003cdir\u003e\nctk db validate \u003cdir\u003e\n```\n\n## Python API\n\n```python\nfrom ctk import ConversationDB, registry, Message, MessageContent, MessageRole\n\nwith ConversationDB(\"chats\") as db:\n    # Search\n    results = db.search_conversations(\"python async\", limit=20)\n\n    # Load a tree\n    conv = db.load_conversation(\"conv_id_123\")\n\n    # Tree primitives\n    paths = conv.get_all_paths()\n    longest = conv.get_longest_path()\n\n    cloned = conv.copy()\n    extracted = conv.copy_subtree(node_id=\"msg-abc\")\n    conv.delete_subtree(\"msg-xyz\")\n    conv.prune_to(\"msg-abc\")\n\n    # Add a new message\n    msg = Message(\n        role=MessageRole.USER,\n        content=MessageContent(text=\"Follow-up question\"),\n        parent_id=\"previous-msg-id\",\n    )\n    conv.add_message(msg)\n    db.save_conversation(conv)\n```\n\nA fluent API is also available for query chains:\n\n```python\nfrom ctk import CTK\n\nresults = (\n    CTK(\"chats\")\n    .search(\"python\")\n    .in_source(\"ChatGPT\")\n    .limit(10)\n    .get()\n)\n```\n\n## Plugin System\n\nImporters and exporters are auto-discovered via Python imports. To add a new format, drop a file in the right directory:\n\n```python\n# ctk/importers/my_format.py\nfrom ctk.core.plugin import ImporterPlugin\nfrom ctk.core.models import ConversationTree, Message, MessageContent, MessageRole\n\nclass MyFormatImporter(ImporterPlugin):\n    name = \"my_format\"\n    description = \"Import from My Custom Format\"\n    version = \"1.0.0\"\n\n    def validate(self, data):\n        return \"my_format_marker\" in str(data)\n\n    def import_data(self, data, **kwargs):\n        tree = ConversationTree(title=\"Imported Conversation\")\n        tree.add_message(Message(\n            role=MessageRole.USER,\n            content=MessageContent(text=\"Hello\"),\n        ))\n        return [tree]\n```\n\nThe plugin is picked up the next time `ctk` runs.\n\n### Built-in Importers\n\n`openai`, `anthropic`, `gemini`, `copilot`, `jsonl`, `filesystem_coding`.\n\n### Built-in Exporters\n\n`json`, `jsonl`, `markdown`, `html`, `hugo`, `csv`, `echo`.\n\n## Database Schema\n\nThe \"database\" is a directory:\n\n```\nmy_chats/\n  conversations.db       # SQLite file\n  media/                 # image attachments referenced by relative URL\n    \u003cuuid\u003e.webp\n    ...\n```\n\nTables (via SQLAlchemy ORM in `ctk/core/db_models.py`):\n\n- `conversations`: metadata, title, timestamps, source, model, slug, starred / pinned / archived flags\n- `messages`: content, role, parent / child relationships, namespaced ids (`\u003cconv-id\u003e::\u003cmsg-id\u003e`)\n- `tags`: searchable tags per conversation\n- `paths`: cached path traversals for fast retrieval\n- `embeddings`, `similarities`: TF-IDF vectors and pairwise similarity for `ctk net` analysis\n\nFTS5 full-text search across message content (with a LIKE fallback when FTS5 is unavailable).\n\n## Privacy\n\n- 100% local for storage and search. Nothing leaves your machine unless you point a provider profile at a remote endpoint.\n- No telemetry, no analytics.\n- Optional sanitization removes API keys, passwords, tokens, SSH keys, database URLs, and similar patterns before sharing or exporting (`--sanitize`).\n\n## Development\n\n```bash\nmake install            # editable install + dev deps\nmake test               # run all tests\nmake test-unit          # unit tests only\nmake test-integration   # integration tests only\nmake coverage           # coverage report (htmlcov/, term-missing)\nmake format             # black + isort\nmake lint               # flake8 + mypy\nmake clean              # remove build artifacts and caches\n```\n\nTest count: ~1700 unit tests as of 2.14.x. The Textual TUI has a Pilot-driven harness in `tests/unit/test_textual_tui.py` covering modal lifecycles, slash commands, and tree-op actions.\n\n## Citation\n\n```bibtex\n@software{towell_ctk_2026,\n  author    = {Towell, Alex},\n  title     = {{CTK}: Conversation Toolkit},\n  year      = 2026,\n  publisher = {GitHub},\n  url       = {https://github.com/queelius/ctk},\n  version   = {2.14.0}\n}\n```\n\nOr use [CITATION.cff](CITATION.cff) for automatic citation in GitHub.\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n\n## Contributing\n\nContributions welcome. To add a new provider import format:\n\n1. Create `ctk/importers/\u003cname\u003e.py` with an `ImporterPlugin` subclass.\n2. Implement `validate()` and `import_data()`.\n3. Add tests in `tests/unit/test_\u003cname\u003e_importer.py`.\n4. Submit a PR.\n\nThe same pattern applies to exporters in `ctk/exporters/`. New TUI bindings, slash commands, and tool providers also welcome (see `ctk/tui/slash.py` and `ctk/core/tools_registry.py`).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqueelius%2Fctk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqueelius%2Fctk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqueelius%2Fctk/lists"}