{"id":36954912,"url":"https://github.com/aucontraire/echomine","last_synced_at":"2026-01-13T13:01:05.116Z","repository":{"id":327452600,"uuid":"1105640106","full_name":"aucontraire/echomine","owner":"aucontraire","description":"Library-first tool for parsing AI conversation exports with search, filtering, and markdown export","archived":false,"fork":false,"pushed_at":"2025-12-15T06:42:35.000Z","size":1884,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-12-18T03:32:21.655Z","etag":null,"topics":["ai","ai-tools","bm25","chat-export","chatgpt","cli","data-export","llm","markdown","openai","python","search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aucontraire.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-27T23:52:40.000Z","updated_at":"2025-12-15T06:40:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/aucontraire/echomine","commit_stats":null,"previous_names":["aucontraire/echomine"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/aucontraire/echomine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aucontraire%2Fechomine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aucontraire%2Fechomine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aucontraire%2Fechomine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aucontraire%2Fechomine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aucontraire","download_url":"https://codeload.github.com/aucontraire/echomine/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aucontraire%2Fechomine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28385798,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T12:01:30.995Z","status":"ssl_error","status_checked_at":"2026-01-13T12:00:09.625Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-tools","bm25","chat-export","chatgpt","cli","data-export","llm","markdown","openai","python","search"],"created_at":"2026-01-13T13:00:31.408Z","updated_at":"2026-01-13T13:01:05.090Z","avatar_url":"https://github.com/aucontraire.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Echomine\n\n**Library-first tool for parsing AI conversation exports with search, filtering, and markdown export**\n\n![Beta](https://img.shields.io/badge/status-beta-yellow?style=flat-square)\n[![PyPI Downloads](https://img.shields.io/pepy/dt/echomine)](https://pepy.tech/project/echomine)\n[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)\n[![Type Checked](https://img.shields.io/badge/mypy-strict-blue.svg)](https://mypy.readthedocs.io/)\n[![Code Style: Ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)\n[![codecov](https://codecov.io/gh/aucontraire/echomine/graph/badge.svg)](https://codecov.io/gh/aucontraire/echomine)\n[![Documentation](https://img.shields.io/badge/docs-mkdocs-blue.svg)](https://aucontraire.github.io/echomine/)\n\n## Overview\n\nEchomine is a Python library and CLI tool for parsing, searching, and exporting AI conversation exports. Built with a multi-provider adapter pattern, it currently supports OpenAI ChatGPT and Anthropic Claude exports, with extensibility for future AI platforms (Gemini, etc.).\n\n### Key Features\n\n- **Memory Efficient**: Stream-based parsing handles 1GB+ files with constant memory usage\n- **Advanced Search**: BM25 relevance ranking with exact phrase matching, boolean logic, role filtering, and keyword exclusion\n- **Message Snippets**: Automatic preview generation for search results with match context\n- **Statistics \u0026 Analytics**: Calculate export statistics, conversation metrics, and temporal patterns\n- **Rich CLI Output**: Color-coded terminal formatting, tables, progress bars, and syntax highlighting\n- **Multiple Export Formats**: Export to Markdown (with YAML frontmatter), JSON, or CSV\n- **Type Safe**: Strict typing with Pydantic v2 and mypy --strict compliance\n- **Library First**: All CLI capabilities available as importable Python library\n- **Multi-Provider Support**: OpenAI ChatGPT and Anthropic Claude exports with auto-detection\n\n### Design Principles\n\n1. **Library-First Architecture**: CLI built on top of library, not vice versa\n2. **Strict Type Safety**: mypy --strict, no `Any` types in public API\n3. **Memory Efficiency**: Stream-based parsing, never load entire file into memory\n4. **Test-Driven Development**: All features test-first validated\n5. **YAGNI**: Simple solutions, no speculative features\n\nSee [Constitution](.specify/memory/constitution.md) for complete design principles.\n\n## Installation\n\n### From Source\n\n```bash\n# Clone repository\ngit clone https://github.com/echomine/echomine.git\ncd echomine\n\n# Install with development dependencies\npip install -e \".[dev]\"\n\n# Install pre-commit hooks (optional)\npre-commit install\n```\n\n### From PyPI (when published)\n\n```bash\npip install echomine\n```\n\n## Quick Start\n\n### Library API (Primary Interface)\n\n```python\nfrom echomine import OpenAIAdapter, ClaudeAdapter, SearchQuery\nfrom pathlib import Path\n\n# Initialize adapter for your provider (stateless, reusable)\nadapter = OpenAIAdapter()  # For ChatGPT exports\n# adapter = ClaudeAdapter()  # For Claude exports\nexport_file = Path(\"conversations.json\")\n\n# 1. List all conversations (discovery)\nfor conversation in adapter.stream_conversations(export_file):\n    print(f\"[{conversation.created_at.date()}] {conversation.title}\")\n    print(f\"  Messages: {len(conversation.messages)}\")\n\n# 2. Search with keywords (BM25 ranking)\nquery = SearchQuery(keywords=[\"algorithm\", \"design\"], limit=10)\nfor result in adapter.search(export_file, query):\n    print(f\"{result.conversation.title} (score: {result.score:.2f})\")\n    print(f\"  Preview: {result.snippet}\")  # v1.1.0: automatic snippets\n\n# 3. Advanced search with filters (v1.1.0+)\nfrom datetime import date\nquery = SearchQuery(\n    keywords=[\"refactor\"],\n    phrases=[\"algo-insights\"],  # Exact phrase matching\n    match_mode=\"all\",  # Require ALL keywords (AND logic)\n    exclude_keywords=[\"test\"],  # Filter out unwanted results\n    role_filter=\"user\",  # Search only user messages\n    from_date=date(2024, 1, 1),\n    to_date=date(2024, 3, 31),\n    limit=5\n)\nfor result in adapter.search(export_file, query):\n    print(f\"[{result.score:.2f}] {result.conversation.title}\")\n    print(f\"  Snippet: {result.snippet}\")\n\n# 4. Calculate statistics (v1.2.0+)\nfrom echomine import calculate_statistics\n\nstats = calculate_statistics(export_file)\nprint(f\"Total conversations: {stats.total_conversations}\")\nprint(f\"Total messages: {stats.total_messages}\")\nprint(f\"Average messages: {stats.average_messages:.1f}\")\n\n# 5. Get specific conversation by ID\nconversation = adapter.get_conversation_by_id(export_file, \"conv-abc123\")\nif conversation:\n    print(f\"Found: {conversation.title}\")\n```\n\n### CLI Usage (Built on Library)\n\n```bash\n# Auto-detect provider (default - works for both OpenAI and Claude)\nechomine list export.json\n\n# Explicit provider selection (v1.3.0+)\nechomine list export.json --provider claude\nechomine list export.json --provider openai\n\n# Search by keywords\nechomine search export.json --keywords \"algorithm,design\" --limit 10\n\n# Search by exact phrase (v1.1.0+)\nechomine search export.json --phrase \"algo-insights\"\n\n# Boolean match mode: require ALL keywords (v1.1.0+)\nechomine search export.json -k \"python\" -k \"async\" --match-mode all\n\n# Exclude unwanted results (v1.1.0+)\nechomine search export.json -k \"python\" --exclude \"django\" --exclude \"flask\"\n\n# Role filtering: search only user/assistant messages (v1.1.0+)\nechomine search export.json -k \"refactor\" --role user\n\n# Combine all filters (v1.1.0+)\nechomine search export.json --phrase \"api\" -k \"python\" --exclude \"test\" --role user --match-mode all\n\n# Search by title (fast, metadata-only)\nechomine search export.json --title \"Project\"\n\n# Filter by date range\nechomine search export.json --from-date \"2024-01-01\" --to-date \"2024-03-31\"\n\n# View export statistics (v1.2.0+)\nechomine stats export.json\n\n# Get conversation by ID (v1.2.0+)\nechomine get export.json conv-abc123\n\n# Export conversation to markdown with YAML frontmatter (v1.2.0+)\nechomine export export.json conv-abc123 --output algo.md\n\n# Export as JSON\nechomine export export.json conv-abc123 --format json --output algo.json\n\n# Export as CSV (v1.2.0+)\nechomine export export.json conv-abc123 --format csv --output algo.csv\n\n# JSON output for search results\nechomine search export.json --keywords \"python\" --json | jq '.results[].title'\n\n# Version info\nechomine --version\n```\n\n**Search Filter Logic:** Content matching (phrases OR keywords) happens first, then post-filtering (--exclude, --role, --title, dates) is applied. See [CLI Usage](https://aucontraire.github.io/echomine/cli-usage/#how-search-filters-combine) for details.\n\nSee [Quickstart Guide](docs/quickstart.md) for detailed examples.\n\n## Development\n\n### Prerequisites\n\n- Python 3.12 or higher\n- Git\n\n### Setup Development Environment\n\n```bash\n# Clone repository\ngit clone https://github.com/echomine/echomine.git\ncd echomine\n\n# Install with development dependencies\npip install -e \".[dev]\"\n\n# Install pre-commit hooks\npre-commit install\n```\n\n### Running Tests\n\n```bash\n# Run all tests\npytest\n\n# Run with coverage\npytest --cov=echomine --cov-report=html\n\n# Run specific test categories\npytest -m unit           # Unit tests only\npytest -m integration    # Integration tests only\npytest -m contract       # Contract tests only\npytest -m performance    # Performance benchmarks\n```\n\n### Code Quality\n\n```bash\n# Type checking (strict mode)\nmypy src/\n\n# Linting and formatting\nruff check .\nruff format .\n\n# Run pre-commit hooks manually\npre-commit run --all-files\n```\n\n### Project Structure\n\n```\nechomine/\n├── src/echomine/           # Library source code\n│   ├── models/             # Pydantic data models\n│   ├── adapters/           # Provider adapters (OpenAI, etc.)\n│   ├── parsers/            # Streaming JSON parsers\n│   ├── search/             # Search and ranking logic\n│   ├── exporters/          # Export formatters (markdown, JSON)\n│   └── cli/                # CLI commands\n├── tests/                  # Test suite\n│   ├── unit/               # Unit tests\n│   ├── integration/        # Integration tests\n│   ├── contract/           # Protocol contract tests\n│   └── performance/        # Performance benchmarks\n└── specs/                  # Design documents\n    └── 001-ai-chat-parser/ # Feature specification\n```\n\n## Documentation\n\n**[Full Documentation](https://aucontraire.github.io/echomine/)** - Comprehensive guides, API reference, and examples\n\n### Quick Links\n\n- [Getting Started](https://aucontraire.github.io/echomine/quickstart/)\n- [Library Usage](https://aucontraire.github.io/echomine/library-usage/)\n- [CLI Reference](https://aucontraire.github.io/echomine/cli-usage/)\n- [API Reference](https://aucontraire.github.io/echomine/api/)\n\n### Spec Documents\n\n- [Feature Specification](specs/001-ai-chat-parser/spec.md)\n- [Implementation Plan](specs/001-ai-chat-parser/plan.md)\n- [CLI Interface Contract](specs/001-ai-chat-parser/contracts/cli_spec.md)\n- [Data Model](specs/001-ai-chat-parser/data-model.md)\n\n## Performance\n\nEchomine is designed for memory efficiency and speed:\n\n- **Memory**: O(1) memory usage regardless of file size (streaming-based)\n- **Search**: \u003c30 seconds for 1.6GB files (10K conversations, 50K messages)\n- **Listing**: \u003c5 seconds for 10K conversations\n\nSee [Performance Requirements](specs/001-ai-chat-parser/spec.md#performance-requirements) for benchmarks.\n\n## Contributing\n\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for:\n\n- Development setup and prerequisites\n- TDD workflow (RED-GREEN-REFACTOR cycle mandatory)\n- Testing guidelines (pytest, mypy --strict, ruff)\n- Code quality standards and conventions\n- Commit message format (conventional commits)\n- Pull request process\n\n## License\n\nAGPL-3.0 License - See [LICENSE](LICENSE) file for details\n\n## Acknowledgments\n\nBuilt with:\n- [Pydantic](https://docs.pydantic.dev/) - Data validation and type safety\n- [ijson](https://github.com/ICRAR/ijson) - Streaming JSON parser\n- [Typer](https://typer.tiangolo.com/) - CLI framework\n- [Rich](https://rich.readthedocs.io/) - Terminal formatting\n- [structlog](https://www.structlog.org/) - Structured logging\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faucontraire%2Fechomine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faucontraire%2Fechomine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faucontraire%2Fechomine/lists"}