{"id":47737882,"url":"https://github.com/viperrcrypto/siftly","last_synced_at":"2026-04-02T23:02:27.163Z","repository":{"id":342634385,"uuid":"1172257670","full_name":"viperrcrypto/Siftly","owner":"viperrcrypto","description":"Local Twitter/X bookmark organizer with AI categorization and mindmap visualization","archived":false,"fork":false,"pushed_at":"2026-03-24T22:24:27.000Z","size":253,"stargazers_count":1976,"open_issues_count":23,"forks_count":183,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-03-25T03:00:42.594Z","etag":null,"topics":["ai","bookmarks","categorization","local-first","mindmap","nextjs","open-source","productivity","twitter","typescript"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/viperrcrypto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-04T05:22:52.000Z","updated_at":"2026-03-25T02:55:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/viperrcrypto/Siftly","commit_stats":null,"previous_names":["viperrcrypto/siftly"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/viperrcrypto/Siftly","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viperrcrypto%2FSiftly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viperrcrypto%2FSiftly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viperrcrypto%2FSiftly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viperrcrypto%2FSiftly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/viperrcrypto","download_url":"https://codeload.github.com/viperrcrypto/Siftly/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viperrcrypto%2FSiftly/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31318137,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T21:35:00.834Z","status":"ssl_error","status_checked_at":"2026-04-02T21:34:59.806Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","bookmarks","categorization","local-first","mindmap","nextjs","open-source","productivity","twitter","typescript"],"created_at":"2026-04-02T23:02:26.099Z","updated_at":"2026-04-02T23:02:27.149Z","avatar_url":"https://github.com/viperrcrypto.png","language":"TypeScript","readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"public/logo.svg\" alt=\"Siftly\" width=\"80\" height=\"80\" /\u003e\n\n  \u003ch1\u003eSiftly\u003c/h1\u003e\n\n  \u003cp\u003e\u003cstrong\u003eSelf-hosted Twitter/X bookmark manager with AI-powered organization\u003c/strong\u003e\u003c/p\u003e\n\n  \u003cp\u003eImport · Analyze · Categorize · Search · Explore\u003c/p\u003e\n\n  \u003cp\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Next.js-16-black?style=flat-square\u0026logo=next.js\" alt=\"Next.js 16\" /\u003e\n    \u003cimg src=\"https://img.shields.io/badge/TypeScript-5-blue?style=flat-square\u0026logo=typescript\" alt=\"TypeScript\" /\u003e\n    \u003cimg src=\"https://img.shields.io/badge/SQLite-local-green?style=flat-square\u0026logo=sqlite\" alt=\"SQLite\" /\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Tailwind-v4-38bdf8?style=flat-square\u0026logo=tailwindcss\" alt=\"Tailwind CSS\" /\u003e\n    \u003cimg src=\"https://img.shields.io/badge/license-MIT-yellow?style=flat-square\" alt=\"MIT License\" /\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n---\n\n## What is Siftly?\n\nSiftly turns your Twitter/X bookmarks into a **searchable, categorized, visual knowledge base** — running entirely on your machine. No cloud, no subscriptions, no browser extensions required. Everything stays local except the AI API calls you configure.\n\nIt runs a **4-stage AI pipeline** on your bookmarks:\n\n```\n📥 Import (built-in bookmarklet or console script — no extensions needed)\n    ↓\n🏷️  Entity Extraction   — mines hashtags, URLs, mentions, and 100+ known tools from raw tweet data (free, zero API calls)\n    ↓\n👁️  Vision Analysis      — reads text, objects, and context from every image/GIF/video thumbnail (30–40 visual tags per image)\n    ↓\n🧠 Semantic Tagging     — generates 25–35 searchable tags per bookmark for AI-powered search\n    ↓\n📂 Categorization       — assigns each bookmark to 1–3 categories with confidence scores\n```\n\nAfter the pipeline runs, you get:\n- **AI search** — find bookmarks by meaning, not just keywords (*\"funny meme about crypto crashing\"*)\n- **Interactive mindmap** — explore your entire bookmark graph visually\n- **Filtered browsing** — grid or list view, filter by category, media type, and date\n- **Export tools** — download media, export as CSV / JSON / ZIP\n\n---\n\n## Quick Start\n\n### Prerequisites\n\n- [Node.js 18+](https://nodejs.org)\n- npm (comes with Node.js)\n\n**That's it.** If you have [Claude Code CLI](https://claude.ai/code) installed and signed in, AI features work automatically — no API key needed.\n\n### Option A — One command (recommended)\n\n```bash\ngit clone https://github.com/viperrcrypto/Siftly.git\ncd Siftly\n./start.sh\n```\n\n`start.sh` installs dependencies, sets up the database, checks for Claude CLI auth, and opens [http://localhost:3000](http://localhost:3000) automatically.\n\n### Option B — Using Claude Code\n\nIf you're using [Claude Code](https://claude.ai/code) to set up the project, it will read `CLAUDE.md` and know exactly how to get started. Just open the project folder:\n\n```bash\ngit clone https://github.com/viperrcrypto/Siftly.git\nclaude Siftly/\n```\n\nClaude Code will handle setup and start the app using your existing Claude subscription — no extra configuration needed.\n\n### Option C — Manual setup\n\n```bash\ngit clone https://github.com/viperrcrypto/Siftly.git\ncd Siftly\nnpm install\nnpx prisma generate\nnpx prisma migrate dev --name init\nnpx next dev\n```\n\nOpen [http://localhost:3000](http://localhost:3000)\n\n---\n\n## AI Authentication\n\nSiftly automatically detects the best available auth method — no configuration needed in the most common case.\n\n### Priority order\n\n| # | Method | How |\n|---|--------|-----|\n| 1 | **Claude Code CLI** *(zero config)* | Already signed in? Siftly reads your session from the macOS keychain automatically |\n| 2 | **API key in Settings** | Open Settings in the app and paste your key |\n| 3 | **`ANTHROPIC_API_KEY` env var** | Set in `.env.local` or your shell environment |\n| 4 | **Local proxy** | Set `ANTHROPIC_BASE_URL` to any Anthropic-compatible endpoint |\n\n### Claude Code CLI (no API key needed)\n\nIf you use [Claude Code](https://claude.ai/code), you're already signed in. Siftly detects your session from the macOS keychain and uses your Claude subscription (Free/Pro/Max) automatically.\n\nThe Settings page shows a green **\"Claude CLI detected — no API key needed\"** badge with your subscription tier when this is active.\n\n\u003e **Note:** This works on macOS. On Linux/Windows, add an API key in Settings instead.\n\n### Getting an API key (if needed)\n\n1. Go to [console.anthropic.com](https://console.anthropic.com)\n2. Create a new API key\n3. Open Siftly → Settings → paste it in\n\nNew accounts include $5 free credit — enough for thousands of bookmarks at Haiku pricing (~$0.00025/bookmark).\n\n---\n\n## Importing Your Bookmarks\n\nSiftly has **built-in import tools** — no browser extensions required. Go to the **Import** page and choose either method:\n\n### Method A — Bookmarklet *(Recommended)*\n\n1. Go to **Import** in the Siftly sidebar\n2. Drag the **\"Export X Bookmarks\"** link to your browser's bookmark bar\n   *(or right-click the bookmark bar → Add Bookmark → paste the URL)*\n3. Go to [x.com/i/bookmarks](https://x.com/i/bookmarks) while logged in to X\n4. Click **\"Export X Bookmarks\"** in your bookmark bar — a purple button appears on the page\n5. Click **\"▶ Auto-scroll\"** — the tool scrolls through and captures all your bookmarks automatically\n6. When complete, click the purple **\"Export N bookmarks\"** button — `bookmarks.json` downloads\n7. Back in Siftly → **Import** → drop or upload the file\n\n### Method B — Browser Console Script\n\n1. Go to [x.com/i/bookmarks](https://x.com/i/bookmarks) while logged in to X\n2. Open DevTools: press `F12` (Windows/Linux) or `⌘⌥J` (Mac), then go to the **Console** tab\n3. Copy the console script from the Siftly Import page, paste it into the console, and press Enter\n4. Click **\"▶ Auto-scroll\"** and wait for all bookmarks to be captured\n5. Click the export button — `bookmarks.json` downloads automatically\n6. Back in Siftly → **Import** → upload the file\n\n### Re-importing\n\nRe-import anytime — Siftly automatically skips duplicates and only adds new bookmarks.\n\n---\n\n## AI Categorization\n\n**Categorization starts automatically as soon as you import.** You can also trigger it manually from:\n\n- The **Import** page (after upload)\n- The **Mindmap** page (when bookmarks are uncategorized)\n- The **Categorize** page in the sidebar\n\n### The 4-Stage Pipeline\n\n| Stage | What it does |\n|-------|-------------|\n| **Entity Extraction** | Mines hashtags, URLs, @mentions, and 100+ known tool/product names from stored tweet JSON — free, zero API calls |\n| **Vision Analysis** | Analyzes every image, GIF, and video thumbnail — OCR text, objects, scene, mood, meme templates, 30–40 visual tags per image |\n| **Semantic Tagging** | Generates 25–35 precise search tags per bookmark by combining tweet text + image context. Also extracts sentiment, people, and company names. |\n| **Categorization** | Assigns 1–3 categories per bookmark with confidence scores using all enriched data |\n\nThe pipeline is **incremental** — if interrupted, it picks up where it left off. Use **\"Re-run everything (force all)\"** to re-analyze bookmarks that were already processed.\n\n---\n\n## Features\n\n### 🔍 AI Search\n\nNatural language queries across all bookmark data:\n\n- *\"funny meme about crypto crashing\"*\n- *\"react hooks tutorial\"*\n- *\"bitcoin price chart\"*\n- *\"best AI coding tools\"*\n\nSearches tweet text, image OCR, visual tags, semantic tags, and categories simultaneously using a full-text search index (FTS5) + Claude semantic reranking. Results are ranked by relevance with AI-generated explanations for each match.\n\n### 🗺️ Mindmap\n\nInteractive force-directed graph showing all bookmarks organized by category:\n\n- Expand/collapse any category to reveal its bookmarks\n- Click a bookmark node to open the original tweet on X\n- Color-coded legend by category\n- If bookmarks aren't categorized yet, an inline **AI Categorize** button starts the pipeline without leaving the page\n\n### 📚 Browse \u0026 Filter\n\n- **Grid view** (masonry layout) or **List view**\n- Filter by category, media type (photo / video), or search text\n- Sort by newest or oldest\n- Pagination with 24 items per page\n- Active filter chips — removable individually or all at once\n- Hover any card to download media or jump to the original tweet\n\n### ⚙️ Categories\n\n8 default categories pre-seeded with AI-readable descriptions:\n\n| Category | Color |\n|----------|-------|\n| Funny Memes | Amber |\n| AI Resources | Violet |\n| Dev Tools | Cyan |\n| Design | Pink |\n| Finance \u0026 Crypto | Green |\n| Productivity | Orange |\n| News | Indigo |\n| General | Slate |\n\nCreate custom categories with a name, color, and optional description. The description is passed directly to the AI during categorization — the more specific, the more accurate the results.\n\n### 📤 Export\n\n- **CSV** — spreadsheet-compatible with all fields\n- **JSON** — full structured data export\n- **ZIP** — exports a category's bookmarks + all media files with a `manifest.csv`\n\n### ⌨️ Command Palette\n\nPress `Cmd+K` (Mac) or `Ctrl+K` (Windows/Linux) to search across all bookmarks from anywhere in the app.\n\n---\n\n## Configuration\n\nAll settings are manageable in the **Settings** page at `/settings` or via environment variables:\n\n| Setting | Env Var | Description |\n|---------|---------|-------------|\n| Anthropic API Key | `ANTHROPIC_API_KEY` | Optional if Claude CLI is signed in — otherwise required for AI features |\n| API Base URL | `ANTHROPIC_BASE_URL` | Custom endpoint for proxies or local Anthropic-compatible models |\n| AI Model | Settings page only | Haiku 4.5 (default, fastest/cheapest), Sonnet 4.6, Opus 4.6 |\n| OpenAI Key | Settings page only | Alternative provider if no Anthropic key is set |\n| Database | `DATABASE_URL` | SQLite file path (default: `file:./prisma/dev.db`) |\n\n### Custom API Endpoint\n\nPoint Siftly at any Anthropic-compatible server:\n\n```env\nANTHROPIC_BASE_URL=http://localhost:8080\n```\n\n---\n\n## Architecture\n\n```\nsiftly/\n├── app/\n│   ├── api/\n│   │   ├── analyze/images/   # Batch image vision analysis (GET progress, POST run)\n│   │   ├── bookmarks/        # List, filter, paginate, delete\n│   │   │   └── [id]/categories/ # Per-bookmark category management\n│   │   ├── categories/       # Category CRUD\n│   │   │   └── [slug]/       # Individual category operations\n│   │   ├── categorize/       # 4-stage AI pipeline (start, status, stop)\n│   │   ├── export/           # CSV, JSON, ZIP export\n│   │   ├── import/           # JSON file import with dedup + auto-pipeline trigger\n│   │   │   ├── bookmarklet/  # Bookmarklet-specific import endpoint\n│   │   │   └── twitter/      # Twitter-specific import endpoint\n│   │   ├── link-preview/     # Server-side OG metadata scraper\n│   │   ├── media/            # Media proxy/download endpoint\n│   │   ├── mindmap/          # Graph nodes + edges for visualization\n│   │   ├── search/ai/        # Natural language semantic search (FTS5 + Claude)\n│   │   ├── settings/         # API key + model config\n│   │   │   ├── cli-status/   # Claude CLI auth detection endpoint\n│   │   │   └── test/         # API key validation endpoint\n│   │   └── stats/            # Dashboard stats\n│   ├── ai-search/            # AI search page\n│   ├── bookmarks/            # Browse, filter, paginate\n│   ├── categories/           # Category management\n│   │   └── [slug]/           # Category detail page\n│   ├── categorize/           # Pipeline monitor with live progress\n│   ├── import/               # 3-step import flow (instructions → upload → categorize)\n│   ├── mindmap/              # Interactive graph\n│   ├── settings/             # Configuration\n│   └── page.tsx              # Dashboard\n│\n├── components/\n│   ├── mindmap/              # Mindmap canvas, nodes, edges\n│   │   ├── mindmap-canvas.tsx\n│   │   ├── category-node.tsx\n│   │   ├── tweet-node.tsx\n│   │   ├── root-node.tsx\n│   │   ├── chain-edge.tsx\n│   │   └── mindmap-context.ts\n│   ├── command-palette.tsx   # Cmd+K global search\n│   ├── nav.tsx               # Sidebar navigation\n│   └── theme-toggle.tsx      # Light/dark mode\n│\n├── lib/\n│   ├── categorizer.ts        # AI categorization logic + default categories\n│   ├── claude-cli-auth.ts    # Claude CLI OAuth session detection (macOS keychain)\n│   ├── vision-analyzer.ts    # Image analysis + batch semantic tagging\n│   ├── image-context.ts      # Shared image context builder\n│   ├── fts.ts                # SQLite FTS5 full-text search index\n│   ├── rawjson-extractor.ts  # Entity extraction from raw tweet JSON\n│   ├── parser.ts             # Multi-format JSON parser\n│   ├── exporter.ts           # CSV, JSON, ZIP export\n│   ├── types.ts              # Shared TypeScript types\n│   └── db.ts                 # Prisma client singleton\n│\n├── prisma/\n│   └── schema.prisma         # SQLite schema\n│\n├── start.sh                  # One-command launcher (install + DB setup + open browser)\n└── CLAUDE.md                 # Instructions for Claude Code AI assistant\n```\n\n### Database Schema\n\n```\nBookmark          — tweet text, author, date, raw JSON, semantic tags, enrichment metadata\n  ├── MediaItem   — images / videos / GIFs with AI-generated image tags\n  └── BookmarkCategory — category assignments with confidence scores (0–1)\n\nCategory          — name, slug, hex color, AI-readable description\nSetting           — key-value store (API keys, model preferences)\nImportJob         — tracks import file status and progress\n```\n\n### Prisma + SQLite + FTS5\n\nSiftly uses Prisma migrations for relational schema changes.\nIn development, run `npx prisma migrate dev --name \u003cchange-name\u003e` when schema changes.\nFor runtime/prod-style startup, apply committed migrations with `npx prisma migrate deploy`.\nFTS5 (`bookmark_fts`) is managed at runtime in [`lib/fts.ts`](./lib/fts.ts), not in `schema.prisma`.\nThis is intentional for now because Prisma does not model SQLite virtual table definitions directly.\n\nFor Prisma command and workflow details, see:\n- https://www.prisma.io/docs/orm/prisma-migrate/workflows/development-and-production\n- https://www.prisma.io/docs/orm/prisma-client/setup-and-configuration/generating-prisma-client\n\n---\n\n## Tech Stack\n\n| Technology | Version | Role |\n|------------|---------|------|\n| [Next.js](https://nextjs.org) | 16 | Full-stack framework (App Router) |\n| [TypeScript](https://www.typescriptlang.org) | 5 | Type safety throughout |\n| [Prisma](https://www.prisma.io) | 7 | ORM + migrations |\n| [SQLite](https://sqlite.org) | — | Local database — zero setup, includes FTS5 |\n| [Tailwind CSS](https://tailwindcss.com) | v4 | Styling |\n| [Anthropic SDK](https://docs.anthropic.com) | — | Vision, semantic tagging, categorization, search |\n| [@xyflow/react](https://xyflow.com) | 12 | Interactive mindmap graph |\n| [Framer Motion](https://www.framer.com/motion/) | 12 | Animations |\n| [Radix UI](https://www.radix-ui.com) | — | Accessible UI primitives |\n| [JSZip](https://stuk.github.io/jszip/) | — | Category ZIP export |\n| [Lucide React](https://lucide.dev) | — | Icons |\n\n---\n\n## Development\n\n```bash\n# One-command start (installs, sets up DB, opens browser)\n./start.sh\n\n# Or manually:\nnpm install\nnpx prisma generate\nnpx prisma migrate dev --name init\nnpx next dev\n\n# Type check\nnpx tsc --noEmit\n\n# Open database GUI\nnpx prisma studio\n\n# Build for production\nnpm run build \u0026\u0026 npm start\n```\n\n### Customizing Categories\n\nEdit `DEFAULT_CATEGORIES` in `lib/categorizer.ts`. Each entry needs:\n\n```ts\n{\n  name: 'My Category',       // Display name\n  slug: 'my-category',       // URL-safe identifier (must be unique)\n  color: '#6366f1',          // Hex color shown in UI\n  description: '...',        // Natural language description — used verbatim in AI prompts\n}\n```\n\nThe `description` field directly shapes how the AI classifies bookmarks. Be specific.\n\n### Adding Known Tools\n\nAdd domain strings to `KNOWN_TOOL_DOMAINS` in `lib/rawjson-extractor.ts` to have the entity extractor automatically recognize links to those tools in tweet data.\n\n---\n\n## Privacy\n\n- All data is stored **locally** in a SQLite file on your machine\n- The only external calls are to the AI provider you configure (tweet text + image data)\n- No telemetry, no tracking, no accounts required\n- Your bookmarks never touch any third-party server except your configured AI endpoint\n\n---\n\n## Support Development\n\nIf Siftly saves you time, consider leaving a tip ☕\n\n---\n\n## License\n\nMIT — see [LICENSE](LICENSE)\n\n---\n\n\u003cdiv align=\"center\"\u003e\n  \u003cp\u003eBuilt by \u003ca href=\"https://x.com/viperr\"\u003e@viperr\u003c/a\u003e · Self-hosted · No extensions · No cloud\u003c/p\u003e\n\u003c/div\u003e\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviperrcrypto%2Fsiftly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fviperrcrypto%2Fsiftly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviperrcrypto%2Fsiftly/lists"}