{"id":30354145,"url":"https://github.com/estevaom/markdown-journal-rust","last_synced_at":"2025-08-19T02:05:56.933Z","repository":{"id":301705736,"uuid":"976492719","full_name":"estevaom/markdown-journal-rust","owner":"estevaom","description":"RAG to index md files accessible via Rust scripts","archived":false,"fork":false,"pushed_at":"2025-08-15T03:28:14.000Z","size":145,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-18T05:48:30.210Z","etag":null,"topics":["journal","journaling","private","rust","rust-lang","rustlang"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/estevaom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-02T07:49:31.000Z","updated_at":"2025-08-15T07:12:36.000Z","dependencies_parsed_at":"2025-06-28T10:32:47.451Z","dependency_job_id":"da7145f6-5922-4508-a82f-2b0551d5b597","html_url":"https://github.com/estevaom/markdown-journal-rust","commit_stats":null,"previous_names":["estevaom/md-rag-mcp","estevaom/markdown-journal-rust"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/estevaom/markdown-journal-rust","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estevaom%2Fmarkdown-journal-rust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estevaom%2Fmarkdown-journal-rust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estevaom%2Fmarkdown-journal-rust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estevaom%2Fmarkdown-journal-rust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/estevaom","download_url":"https://codeload.github.com/estevaom/markdown-journal-rust/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estevaom%2Fmarkdown-journal-rust/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271088140,"owners_count":24697081,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-19T02:00:09.176Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["journal","journaling","private","rust","rust-lang","rustlang"],"created_at":"2025-08-19T02:04:31.030Z","updated_at":"2025-08-19T02:05:56.910Z","avatar_url":"https://github.com/estevaom.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Journal RAG System\n\nA powerful, privacy-focused journal system with AI-powered semantic search and metadata analysis. This system allows you to query your personal journal entries using natural language and analyze patterns in your mood, habits, and thoughts over time.\n\n## ✨ Features\n\n- **🔍 Semantic Search**: Query your journal entries using natural language with Rust-powered performance\n- **📊 Frontmatter Analysis**: Track mood, anxiety, weight, and custom metrics over time\n- **🦀 Rust Implementation**: High-performance indexing and search using native Rust\n- **🏠 Local Processing**: Journal files and search indexing stay on your machine\n- **⚡ Fast Embeddings**: Uses fastembed for efficient vector generation\n- **📱 Cross-Platform**: Works on macOS, Linux, and Windows (via WSL)\n\n## 🚀 Quick Start\n\n### 1. Automated Setup\n\nChoose your platform and run the setup script:\n\n**macOS:**\n```bash\n./setup_mac_environment.sh\n```\n\n**Ubuntu/Debian (including WSL):**\n```bash\n./setup_ubuntu_environment.sh\n```\n\n**Arch Linux:**\n```bash\n./setup_arch_environment.sh\n```\n\nThese scripts will:\n- Install Rust (if needed)\n- Install system dependencies\n- Build all tools\n- Set up convenience scripts\n\n### 2. Manual Setup (Optional)\n\nIf you prefer manual setup:\n\n**Install Rust:**\n```bash\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\nsource $HOME/.cargo/env\n```\n\n**Build the tools:**\n```bash\n# Build the RAG search tools\ncd .tech/code/rust_scripts/rag_search\ncargo build --release\n\n# Build the frontmatter query tool\ncd ../frontmatter_query\ncargo build --release\n```\n\n### 3. Start Journaling\n\nAdd entries to the `journal/` directory using the provided templates:\n\n**Daily Entry** (`template/daily.md`):\n```\njournal/2025/01/01.md\njournal/2025/01/02.md\n```\n\n**Weekly Retrospective** (`template/weekly_retro.md`):\n```\njournal/2025/week_01_retro.md\n```\n\nExample frontmatter for tracking:\n```yaml\n---\ndate: 2025-01-01\nmood: 7\nenergy: 8\nsleep_hours: 7.5\ntags: [\"work\", \"learning\", \"reflection\"]\n---\n\n# Today's thoughts\nYour journal content here...\n```\n\nTemplates are provided in the `template/` directory:\n- `daily.md` - Daily journal template with sections for reflection\n- `weekly_retro.md` - Weekly retrospective for reviewing progress\n- `prompt.md` - Guide for configuring your AI assistant persona\n\n## 🎯 Usage\n\n### Convenient Shell Scripts\n\nThe project includes convenient shell scripts for common operations:\n\n```bash\n# Reindex journal entries (run after adding new entries)\n./reindex-rag.sh\n\n# Search your journal\n./search-rag.sh \"anxiety and sleep patterns\"\n./search-rag.sh \"rust learning journey\" -n 10\n./search-rag.sh \"relationship insights\" --after 2025-06-01\n./search-rag.sh \"debugging professional\" --files-only\n\n# Query frontmatter metadata\n./query-frontmatter.sh --fields mood anxiety weight_kg\n./query-frontmatter.sh --fields mood --start-date 2025-07-01 --stats\n./query-frontmatter.sh --fields weight_kg --format csv \u003e weight_data.csv\n```\n\n### Direct Command Usage\n\nYou can also use the Rust binaries directly:\n\n#### Index Your Journal Entries\n\n```bash\n# Index all journal files\n.tech/code/rust_scripts/rag_search/target/release/rag-index \\\n  --journal-dir journal \\\n  --lance-dir .tech/data/lancedb\n\n# Force rebuild the entire index\n.tech/code/rust_scripts/rag_search/target/release/rag-index --rebuild\n\n# Index only recent files (since a specific date)\n.tech/code/rust_scripts/rag_search/target/release/rag-index --since 2025-01-01\n```\n\n#### Semantic Search\n\n```bash\n# Basic search\n.tech/code/rust_scripts/rag_search/target/release/rag-search \"productivity tips\"\n\n# Search with date filters\n.tech/code/rust_scripts/rag_search/target/release/rag-search \"anxiety\" \\\n  --after 2025-01-01 \\\n  --before 2025-01-31\n\n# Get more results\n.tech/code/rust_scripts/rag_search/target/release/rag-search \"meditation\" --num-results 20\n\n# Output as JSON\n.tech/code/rust_scripts/rag_search/target/release/rag-search \"goals\" --format json\n```\n\n#### Frontmatter Analysis\n\n```bash\n# Query specific fields\n.tech/code/rust_scripts/frontmatter_query/target/release/frontmatter-query \\\n  --fields mood anxiety weight_kg\n\n# Calculate statistics\n.tech/code/rust_scripts/frontmatter_query/target/release/frontmatter-query \\\n  --fields mood anxiety \\\n  --stats\n\n# Export as CSV\n.tech/code/rust_scripts/frontmatter_query/target/release/frontmatter-query \\\n  --fields mood weight_kg \\\n  --format csv \u003e mood_weight.csv\n```\n\n## 📁 Directory Structure\n\n```\nmarkdown-journal-rust/\n├── README.md                           # This file\n├── CLAUDE.md                          # Instructions for Claude Code\n├── setup_mac_environment.sh           # macOS setup script\n├── setup_ubuntu_environment.sh        # Ubuntu/Debian setup script\n├── setup_arch_environment.sh          # Arch Linux setup script\n├── search-rag.sh                      # Convenient search script\n├── reindex-rag.sh                     # Reindex journal script\n├── query-frontmatter.sh               # Query metadata script\n├── .claude/                           # Claude Code integration\n│   ├── agents/                        # Custom AI agents\n│   ├── commands/                      # Slash commands\n│   └── settings.json                  # Claude Code settings\n├── template/                          # Journal templates\n│   ├── daily.md                      # Daily entry template\n│   ├── weekly_retro.md               # Weekly review template\n│   └── prompt.md                     # AI assistant configuration\n├── journal/                           # Your journal entries\n│   ├── 2025/01/01.md                 # Daily entries\n│   └── topics/                       # Topic-based entries\n└── .tech/                            # Technical components\n    ├── code/\n    │   ├── rust_scripts/             # Rust implementations (PRIMARY)\n    │   │   ├── rag_search/           # RAG search workspace\n    │   │   │   ├── rag-index/        # Indexing tool\n    │   │   │   └── rag-search/       # Search tool\n    │   │   └── frontmatter_query/    # Metadata analysis tool\n    │   └── python_legacy/            # Python implementation (DEPRECATED)\n    │       ├── mcp/                  # MCP servers\n    │       ├── scripts/              # Python scripts\n    │       └── LEGACY_README.md      # Legacy documentation\n    └── data/                         # Generated data\n        ├── lancedb/                  # Vector database (Rust)\n        └── chroma_db/                # Vector database (Python legacy)\n```\n\n## 🔧 System Requirements\n\n### Minimum Requirements\n- **Rust 1.70+** (install via rustup)\n- **4GB RAM** \n- **2GB free disk space**\n\n### Recommended\n- **8GB+ RAM** for better performance\n- **SSD storage** for faster indexing\n\n## 📊 Frontmatter Fields\n\nTrack various metrics in your journal entries:\n\n```yaml\n---\ndate: 2025-01-01          # Entry date (required)\nmood: 7                   # 1-10 scale\nanxiety: 3                # 1-10 scale  \nenergy: 8                 # 1-10 scale\nsleep_hours: 7.5          # Hours of sleep\nweight_kg: 70             # Weight in kg\nexercise_minutes: 30      # Exercise duration\nweather: \"sunny\"          # Weather description\ntags: [\"work\", \"family\"]  # Custom tags\n---\n```\n\nYou can customize these fields based on what you want to track.\n\n## 🤖 Claude Code Integration\n\nThis project includes enhanced Claude Code integration for a superior development experience:\n\n### Features\n\n- **Custom Agents** (`.claude/agents/`):\n  - `journal-rag-search`: Automatically searches your journal when context is needed\n  - `journal-field-completer`: Helps complete journal frontmatter fields\n  - `weekly-retro-analyzer`: Analyzes weekly retrospectives\n\n- **Slash Commands** (`.claude/commands/`):\n  - `/start`: Initialize your journaling session\n  - `/commit`: Smart git commits with context\n\n- **Status Line**: Real-time token usage tracking (requires `bunx ccusage`)\n\n- **Automatic Timestamps**: Each prompt includes current timestamp for context\n\n### Setup (Optional)\n\n1. Claude Code will automatically detect the `.claude` directory\n2. For status line, install ccusage: `npm install -g ccusage`\n3. Agents and commands are available immediately\n\n### Usage Examples\n\nWith Claude Code:\n- Ask about past events - the RAG agent will search automatically\n- Use `/start` to begin a new session with context\n- Use `/commit` for intelligent commit messages\n\nThese integrations are optional but recommended for the best experience.\n\n## 🔍 Advanced Features\n\n### Backup and Sync\n\nYour journal is just markdown files - sync with any service:\n- Git repositories\n- Cloud storage (Dropbox, iCloud, etc.)\n- Network drives\n\nThe LanceDB index (`.tech/data/lancedb/`) can be rebuilt anytime from your journal files.\n\n## 📚 Documentation\n\n- **Setup Scripts**: `setup_mac_environment.sh`, `setup_ubuntu_environment.sh`, `setup_arch_environment.sh`\n- **Claude Code Guide**: `CLAUDE.md`\n\n## 🆘 Support\n\n### Common Issues\n\n**\"Command not found\" errors:**\n- Ensure Rust is installed: `rustc --version`\n- Check if tools are built: `ls .tech/code/rust_scripts/*/target/release/`\n- Use full paths or create aliases\n\n**Indexing issues:**\n- Ensure journal directory exists and contains .md files\n- Check file permissions\n- Try rebuilding with `--rebuild` flag\n\n**Performance issues:**\n- The first indexing might be slow as models are downloaded\n- Subsequent runs use cached models and are much faster\n- Consider using fewer results with `--num-results`\n\n### Getting Help\n\n1. Check the documentation files\n2. Review the `--help` output for each tool\n3. Open an issue on GitHub with system details\n\n---\n\n## 🔒 Privacy \u0026 Data Considerations\n\n**What stays private (always local):**\n- Your journal files and content\n- Vector embeddings and search indexing\n- Frontmatter analysis and statistics\n- All file processing and storage\n\n**Performance Note:** This Rust implementation offers superior search quality and blazing-fast search speeds (\u003c20ms) compared to the Python version. While indexing is slower than the GPU-accelerated Python version, the search results are more accurate and retrieval is nearly instantaneous.\n\n---\n\n## 📦 Legacy Python Implementation\n\nA Python implementation is available in `.tech/code/python_legacy/` for compatibility and educational purposes. This includes:\n\n- Original MCP servers for AI agent integration\n- ChromaDB-based semantic search\n- Python-based frontmatter analysis\n\n**Note:** The Python implementation is deprecated and not actively maintained. It's provided for:\n- Users who need MCP protocol compatibility\n- Educational comparison between implementations\n- Fallback if Rust setup is problematic\n\nFor setup instructions, see [`.tech/code/python_legacy/LEGACY_README.md`](.tech/code/python_legacy/LEGACY_README.md).\n\n**We strongly recommend using the Rust implementation for all new projects.**","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festevaom%2Fmarkdown-journal-rust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Festevaom%2Fmarkdown-journal-rust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festevaom%2Fmarkdown-journal-rust/lists"}