{"id":34642275,"url":"https://github.com/mbailey/sqldown","last_synced_at":"2026-05-06T00:02:24.946Z","repository":{"id":324388799,"uuid":"1095678160","full_name":"mbailey/sqldown","owner":"mbailey","description":"SQLite is the new markdown. Bidirectional markdown-sqlite converter - query your docs with SQL, export results as markdown.","archived":false,"fork":false,"pushed_at":"2025-11-18T09:25:16.000Z","size":134,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-04-15T19:42:41.864Z","etag":null,"topics":["cli","markdown","python","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mbailey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-13T11:23:07.000Z","updated_at":"2026-04-10T00:53:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mbailey/sqldown","commit_stats":null,"previous_names":["mbailey/sqldown"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mbailey/sqldown","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbailey%2Fsqldown","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbailey%2Fsqldown/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbailey%2Fsqldown/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbailey%2Fsqldown/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mbailey","download_url":"https://codeload.github.com/mbailey/sqldown/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbailey%2Fsqldown/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32672682,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-05T11:29:49.557Z","status":"ssl_error","status_checked_at":"2026-05-05T11:29:48.587Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","markdown","python","sqlite"],"created_at":"2025-12-24T17:25:16.868Z","updated_at":"2026-05-06T00:02:24.933Z","avatar_url":"https://github.com/mbailey.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ☠️ SQLDown - pre-alpha - USE AT OWN RISK\n\n[![PyPI version](https://badge.fury.io/py/sqldown.svg)](https://pypi.org/project/sqldown/)\n[![Python versions](https://img.shields.io/pypi/pyversions/sqldown.svg)](https://pypi.org/project/sqldown/)\n[![CI/CD](https://github.com/mbailey/sqldown/actions/workflows/ci.yml/badge.svg)](https://github.com/mbailey/sqldown/actions)\n\n**Bidirectional markdown ↔ SQLite conversion** - Load markdown files into SQLite, query with SQL, and export back to markdown.\n\n## Features\n\n☠️ **Pre-Alpha Software (v0.1.0) - USE AT YOUR OWN RISK:**\n- Dynamic schema generation from YAML frontmatter and markdown structure\n- Column limit protection with intelligent section extraction\n- Import markdown collections into queryable SQLite databases\n- Export database rows back to markdown files\n- Watch mode for auto-refresh on file changes\n- Gitignore-aware file filtering\n- Smart change detection (skip unchanged files)\n\n## Installation\n\n```bash\n# Install from PyPI\npip install sqldown\n\n# Or use uv for faster installation\nuv pip install sqldown\n```\n\n## Quick Start\n\n```bash\n# Load markdown files into SQLite\nsqldown load ~/tasks\n\n# Query with sqlite3\nsqlite3 sqldown.db \"SELECT * FROM docs WHERE status='active'\"\n\n# Export back to markdown\nsqldown dump -d sqldown.db -o ~/restored\n\n# Get database info\nsqldown info\n\n# Show table details\nsqldown info -t docs\n```\n\n## Load Command\n\n```bash\nsqldown load PATH [OPTIONS]\n```\n\n**What it does:**\n- Scans markdown files recursively\n- Parses YAML frontmatter → database columns\n- Extracts H2 sections → `section_*` columns\n- Creates schema dynamically based on discovered fields\n- Upserts into SQLite (idempotent - safe to run multiple times)\n- Respects `.gitignore` patterns automatically\n\n**Options:**\n- `-d, --db PATH` - Database file (default: `sqldown.db`)\n- `-t, --table NAME` - Table name (default: `docs`)\n- `-p, --pattern GLOB` - File pattern (default: `**/*.md`)\n- `-m, --max-columns N` - Maximum allowed columns (default: 1800, SQLite limit: 2000)\n- `-n, --top-sections N` - Extract only top N most common sections (default: 20, 0=all)\n- `-w, --watch` - Watch for file changes and auto-update\n- `-v, --verbose` - Show detailed progress\n\n## Dump Command\n\n```bash\nsqldown dump -d DATABASE -o OUTPUT_DIR [OPTIONS]\n```\n\n**What it does:**\n- Exports database rows back to markdown files\n- Reconstructs original markdown structure with frontmatter\n- Preserves file paths from original import\n- Skips unchanged files (smart change detection)\n- Supports SQL filtering to export subsets\n\n**Options:**\n- `-d, --db PATH` - Database file (required)\n- `-t, --table NAME` - Table name (default: `docs`)\n- `-o, --output PATH` - Output directory (required)\n- `-f, --filter WHERE` - SQL WHERE clause to filter rows\n- `--force` - Always write files, even if unchanged\n- `-n, --dry-run` - Preview what would be exported without writing\n- `-v, --verbose` - Show detailed progress\n\n**Examples:**\n```bash\n# Export all documents\nsqldown dump -d cache.db -o ~/restored\n\n# Export only active tasks\nsqldown dump -d cache.db -t tasks -o ~/active --filter \"status='active'\"\n\n# Preview export without writing files\nsqldown dump -d cache.db -o ~/export --dry-run\n```\n\n## Info Command\n\n```bash\nsqldown info [OPTIONS]\n```\n\n**What it does:**\n- Shows database statistics and table information\n- Lists all tables with document counts\n- Displays column breakdown (frontmatter vs sections)\n- Provides schema details for specific tables\n\n**Options:**\n- `-d, --db PATH` - Database file (default: `sqldown.db` if exists)\n- `-t, --table NAME` - Show detailed info for specific table\n\n**Examples:**\n```bash\n# Show database overview (uses sqldown.db if present)\nsqldown info\n\n# Show info for specific database\nsqldown info -d cache.db\n\n# Show detailed table information\nsqldown info -t tasks\n```\n\n## Using SQLite3\n\nOnce imported, use sqlite3 directly for all queries:\n\n```bash\n# List tables\nsqlite3 cache.db \".tables\"\n\n# Show schema\nsqlite3 cache.db \".schema tasks\"\n\n# Query\nsqlite3 cache.db \"SELECT title, status FROM tasks WHERE status='active' LIMIT 10\"\n\n# Aggregate\nsqlite3 cache.db \"SELECT status, COUNT(*) FROM tasks GROUP BY status\"\n\n# Complex queries\nsqlite3 cache.db \"\n  SELECT project, COUNT(*) as active_count\n  FROM tasks\n  WHERE status='active'\n  GROUP BY project\n  ORDER BY active_count DESC\n\"\n\n# Export to CSV\nsqlite3 -csv cache.db \"SELECT * FROM tasks WHERE status='active'\" \u003e active.csv\n\n# Interactive mode\nsqlite3 cache.db\n```\n\n## Dynamic Schema Example\n\nFrom this markdown:\n```markdown\n---\nstatus: active\nproject: agents\npriority: high\n---\n\n# Add SQLite caching\n\n## Objective\nCreate a cache layer...\n\n## Implementation Plan\n1. Parser\n2. Schema\n```\n\nCreates these columns automatically:\n- Core: `_id`, `_path`, `_sections`, `title`, `body`, `lead`, `file_modified`\n- Frontmatter: `status`, `project`, `priority`\n- Sections: `section_objective`, `section_implementation_plan`\n\n**Real example:** 87 tasks → 181 columns (no schema design needed!)\n\n## Column Limit Protection\n\nSQLite has a hard limit of 2000 columns per table. With diverse markdown documents, you can easily hit this limit:\n\n**Problem:** 5,225 tasks with diverse sections = 6,694 columns (💥 exceeds limit!)\n\n**Solution:** Use `--top-sections` to extract only the most common sections:\n\n```bash\n# Extract only top 20 most common sections (default)\nsqldown load ~/tasks -d cache.db --top-sections 20\n\n# Result: 5,225 tasks → 116 columns ✅\n# - 7 base columns (_id, _path, title, body, lead, _sections, file_modified)\n# - 89 frontmatter columns (status, project, type, priority, etc.)\n# - 20 section columns (overview, usage, objective, notes, next_steps, etc.)\n```\n\n**What happens to other sections?**\n- All content is preserved in the `body` field\n- You can still search across all sections using SQLite FTS5\n- Only the top N sections become queryable columns\n\n**Column limit validation:**\n```bash\n# Check if your documents will fit before importing\nsqldown load ~/docs -d test.db --verbose\n\n# Output shows breakdown:\n# 📊 Column breakdown:\n#   - Base columns: 7\n#   - Frontmatter columns: 89\n#   - Section columns: 20\n#   - Total: 116 (limit: 1800)\n```\n\n**When approaching limit (\u003e90%):**\n- Shows warning but continues import\n- Consider: reducing --top-sections or increasing --max-columns\n\n**When exceeding limit:**\n- Stops before import to prevent database corruption\n- Shows breakdown and suggestions\n\n## Multiple Collections\n\nOne database, multiple tables:\n\n```bash\n# Import different document types\nsqldown load ~/tasks -d cache.db -t tasks\nsqldown load ~/notes -d cache.db -t notes\nsqldown load ~/.claude/skills -d cache.db -t skills\n\n# Query them\nsqlite3 cache.db \"SELECT * FROM tasks WHERE status='active'\"\nsqlite3 cache.db \"SELECT * FROM notes WHERE tags LIKE '%sqlite%'\"\n\n# Join across tables\nsqlite3 cache.db \"\n  SELECT t.title as task, n.title as note\n  FROM tasks t\n  JOIN notes n ON n.tags LIKE '%' || t.project || '%'\n  WHERE t.status='active'\n\"\n```\n\n## Refresh Strategy\n\n**One-time import:**\n\nImport is idempotent - just run it again:\n\n```bash\n# Add this to cron or a git hook\nsqldown load ~/tasks -d cache.db -t tasks\n```\n\n**Watch mode (auto-refresh):**\n\nUse the `--watch` / `-w` flag to automatically update the cache when files change:\n\n```bash\n# Watch mode: import once, then auto-update on file changes\nsqldown load ~/tasks -d cache.db -t tasks --watch\n\n# Output:\n# ✅ Imported 87 documents into cache.db:tasks\n# 📋 Schema has 181 columns\n#\n# 👀 Watching /Users/admin/tasks for changes... (Ctrl-C to stop)\n# [2025-01-15 10:23:45] Updated: AG-22_feat_add-configuration/README.md\n# [2025-01-15 10:24:12] Added: AG-31_feat_new-feature/README.md\n```\n\nWatch mode is ideal for development workflows where you want the cache to stay in sync with your files.\n\n## Common Queries\n\n```bash\n# Active tasks\nsqlite3 cache.db \"SELECT title FROM tasks WHERE status='active'\"\n\n# Recent updates\nsqlite3 cache.db \"SELECT title, updated FROM tasks ORDER BY updated DESC LIMIT 10\"\n\n# By project\nsqlite3 cache.db \"SELECT project, COUNT(*) FROM tasks GROUP BY project\"\n\n# Search content\nsqlite3 cache.db \"SELECT title FROM tasks WHERE body LIKE '%cache%'\"\n\n# High priority incomplete\nsqlite3 cache.db \"SELECT title FROM tasks WHERE priority='high' AND status != 'completed'\"\n```\n\n## Philosophy\n\nSQLDown follows the Unix philosophy: do one thing well.\n\n- **Load**: SQLDown handles the complex markdown → SQLite conversion\n- **Query**: sqlite3 provides perfect SQL interface (no wrapper needed)\n- **Dump**: SQLDown reconstructs markdown from database rows\n\nWhy not wrap sqlite3? Because it's already perfect for queries:\n- Full SQL power without wrapper limitations\n- Standard tool with excellent documentation\n- Multiple output formats (CSV, JSON, column, etc.)\n- Interactive shell with history and completion\n- Zero overhead for read operations\n\n## Requirements\n\n- Python 3.8+ (includes sqlite3 module)\n- sqlite3 CLI (built-in on macOS/Linux)\n\n**Python Dependencies** (installed automatically):\n- click \u003e= 8.0 - CLI framework\n- sqlite-utils \u003e= 3.30 - SQLite schema management\n- PyYAML \u003e= 6.0 - YAML frontmatter parsing\n- pathspec \u003e= 0.11 - Gitignore pattern support\n- watchdog \u003e= 3.0 - File system monitoring\n\n## Integration\n\nSQLDown is designed for both human and AI use:\n\n**For Developers:**\n- Simple CLI with sensible defaults\n- Watch mode for development workflows\n- Smart change detection saves time\n- Direct sqlite3 access for queries\n\n**For AI Assistants:**\n- Efficient token usage via SQL queries\n- Progressive disclosure pattern\n- Query metadata first, read files only when needed\n- Structured data extraction from markdown\n\n## Contributing\n\nContributions welcome! Please check out the [issues](https://github.com/mbailey/sqldown/issues) on GitHub.\n\n## License\n\nMIT\n\n## Changelog\n\n### v0.1.0 (2025-01-14)\n\n**Initial PyPI Release** 🎉\n\n- Full bidirectional markdown ↔ SQLite conversion\n- Dynamic schema generation from YAML frontmatter\n- Intelligent column limit protection (SQLite 2000 column limit)\n- Top-N section extraction for diverse document collections\n- Watch mode for automatic file sync\n- Smart change detection on export\n- Comprehensive CLI with `load`, `dump`, and `info` commands\n- Python 3.8+ support\n\n### Development History\n\nSee [SPECIFICATION.md](SPECIFICATION.md) for the complete design.\nSee [REVIEW.md](REVIEW.md) for architectural decisions and trade-offs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmbailey%2Fsqldown","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmbailey%2Fsqldown","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmbailey%2Fsqldown/lists"}