{"id":29007351,"url":"https://github.com/deeptanshu-sankhwar/semantic-code-navigator","last_synced_at":"2025-08-11T17:08:16.971Z","repository":{"id":298507970,"uuid":"999969076","full_name":"Deeptanshu-sankhwar/semantic-code-navigator","owner":"Deeptanshu-sankhwar","description":"Stress testing MindsDB Knowledge Base with a semantic codebase navigator CLI","archived":false,"fork":false,"pushed_at":"2025-06-29T11:22:17.000Z","size":185,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-17T03:25:37.282Z","etag":null,"topics":["mindsdb","mindsdb-python-sdk","semantic-search"],"latest_commit_sha":null,"homepage":"https://medium.com/@deeptanshu.sankhwar/i-built-a-cli-that-reads-your-entire-codebase-like-a-human-developer-and-its-scary-good-a9ec79c4fd6d","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Deeptanshu-sankhwar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-11T04:32:27.000Z","updated_at":"2025-07-10T03:23:33.000Z","dependencies_parsed_at":"2025-06-11T13:48:09.058Z","dependency_job_id":"c20c6a0b-450c-4eaa-84b2-1df89291f5be","html_url":"https://github.com/Deeptanshu-sankhwar/semantic-code-navigator","commit_stats":null,"previous_names":["deeptanshu-sankhwar/semantic-code-navigator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Deeptanshu-sankhwar/semantic-code-navigator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deeptanshu-sankhwar%2Fsemantic-code-navigator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deeptanshu-sankhwar%2Fsemantic-code-navigator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deeptanshu-sankhwar%2Fsemantic-code-navigator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deeptanshu-sankhwar%2Fsemantic-code-navigator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Deeptanshu-sankhwar","download_url":"https://codeload.github.com/Deeptanshu-sankhwar/semantic-code-navigator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deeptanshu-sankhwar%2Fsemantic-code-navigator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269922906,"owners_count":24496999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mindsdb","mindsdb-python-sdk","semantic-search"],"created_at":"2025-06-25T13:08:41.535Z","updated_at":"2025-08-11T17:08:16.945Z","avatar_url":"https://github.com/Deeptanshu-sankhwar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Semantic Code Navigator\n\nA CLI tool for stress testing MindsDB's Knowledge Base feature through semantic codebase navigation. Ingests codebases and enables natural language search using MindsDB's embedding and reranking capabilities.\n\n## Overview\n\nThe Semantic Code Navigator is a CLI application that transforms codebase navigation through semantic search. It clones GitHub repositories, extracts functions and classes from multiple programming languages, and ingests them into MindsDB Knowledge Bases with rich metadata. Users can perform natural language queries with advanced filtering capabilities.\n\nThe application demonstrates MindsDB Knowledge Base capabilities including CREATE KNOWLEDGE_BASE with OpenAI embedding models, batch INSERT operations, complex SELECT queries with metadata filtering, and CREATE INDEX for performance optimization.\n\n## Features\n\n### Core Functionality\n- Semantic code search with natural language queries\n- Metadata filtering by language, file path, function name, repository\n- Batch processing for large codebases\n- Multiple output formats (table, JSON, compact)\n- Progress tracking with rich CLI interface\n\n### AI-Enhanced Analysis\n- Code purpose classification\n- Natural language explanations\n- Automated docstring generation\n- Test case suggestions\n- Search result rationale\n\n### Stress Testing Capabilities\n- Concurrent query testing\n- Performance benchmarking\n- Scalability analysis\n- Error rate monitoring\n\n## Prerequisites\n\n1. **MindsDB**: Install and run MindsDB locally or use MindsDB Cloud\n   ```bash\n   # Local installation\n   pip install mindsdb\n   \n   # Docker\n   docker-compose up\n   ```\n\n2. **OpenAI API Key**: Required for embeddings and reranking\n   - Get your API key from [OpenAI Platform](https://platform.openai.com/api-keys)\n\n## Installation\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/Deeptanshu-sankhwar/semantic-code-navigator.git\n   cd semantic-code-navigator\n   ```\n\n2. Install dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. Configure environment:\n   ```bash\n   cp env.example .env\n   # Edit .env with your configuration\n   ```\n\n4. Set environment variables:\n   ```bash\n   # Required\n   OPENAI_API_KEY=sk-your-openai-api-key-here\n   \n   # Optional (for local MindsDB)\n   MINDSDB_HOST=127.0.0.1\n   MINDSDB_PORT=47334\n   \n   # Optional (for MindsDB Cloud)\n   MINDSDB_USER=your-email@example.com\n   MINDSDB_PASSWORD=your-password\n   ```\n\n## Usage\n\n### Initialize Knowledge Base\n\n```bash\npython main.py kb:init --validate-config\n```\n\nOptions:\n- `--force`: Recreate knowledge base if exists\n- `--validate-config`: Validate configuration before creation\n\n### Semantic Search\n\n```bash\n# Basic search\npython main.py kb:query \"authentication middleware\"\n\n# With filters\npython main.py kb:query \"database connection\" --language python --limit 20\n\n# With AI analysis\npython main.py kb:query \"error handling\" --ai-all\n\n# Different output formats\npython main.py kb:query \"JWT validation\" --output-format json\n```\n\nSearch Options:\n- `--language, -l`: Filter by programming language\n- `--filepath, -f`: Filter by file path pattern\n- `--function`: Filter by function name\n- `--repo, -r`: Filter by repository name\n- `--limit`: Maximum number of results (default: 10)\n- `--relevance-threshold`: Minimum relevance score (0.0-1.0)\n- `--output-format`: Output format (table, json, compact)\n- `--ai-purpose`: Add AI purpose classification\n- `--ai-explain`: Add AI code explanations\n- `--ai-docstring`: Add AI-generated docstrings\n- `--ai-tests`: Add AI test case suggestions\n- `--ai-all`: Add all AI analysis\n\n### Repository Ingestion\n\nThe `kb:ingest` command clones Git repositories, parses code files to extract functions and classes, and inserts the resulting code chunks into the knowledge base with comprehensive metadata.\n\n```bash\n# Basic ingestion\npython main.py kb:ingest https://github.com/org/repo-name.git\n\n# Advanced ingestion with full options\npython main.py kb:ingest https://github.com/org/repo.git \\\n  --branch develop \\\n  --extensions \"py,js,ts,java,go\" \\\n  --exclude-dirs \"node_modules,__pycache__,build\" \\\n  --batch-size 500 \\\n  --extract-git-info \\\n  --generate-summaries\n\n# Preview ingestion without inserting data\npython main.py kb:ingest https://github.com/org/repo.git --dry-run\n```\n\n#### Ingestion Process\n\n1. **Repository Cloning**: Clones the specified Git repository and branch\n2. **File Discovery**: Scans for files matching specified extensions\n3. **Code Parsing**: Extracts functions, classes, and methods using AST parsing\n4. **Metadata Extraction**: Collects file paths, languages, function names, and Git information\n5. **Batch Processing**: Inserts code chunks in configurable batch sizes for optimal performance\n6. **Progress Reporting**: Provides real-time feedback on ingestion progress\n\n#### Ingestion Options\n\n| Option | Description | Default | Example |\n|--------|-------------|---------|---------|\n| `--branch, -b` | Git branch to clone | `main` | `--branch develop` |\n| `--extensions` | Comma-separated file extensions | `py,js,ts,java,go,rs,cpp,c,h` | `--extensions \"py,js\"` |\n| `--exclude-dirs` | Directories to skip during ingestion | `.git,node_modules,__pycache__,.venv,build,dist` | `--exclude-dirs \"tests,docs\"` |\n| `--batch-size` | Number of records per batch insert | `500` | `--batch-size 1000` |\n| `--extract-git-info` | Include Git author and commit data | `false` | `--extract-git-info` |\n| `--generate-summaries` | Generate AI summaries (costs OpenAI credits) | `false` | `--generate-summaries` |\n| `--dry-run` | Preview without inserting data | `false` | `--dry-run` |\n| `--cleanup` | Remove temporary files after ingestion | `true` | `--no-cleanup` |\n\n#### Supported Languages\n\nThe ingestion engine supports AST-based parsing for:\n- **Python** (.py) - Functions, classes, methods\n- **JavaScript/TypeScript** (.js, .ts) - Functions, classes, arrow functions\n- **Java** (.java) - Methods, classes, interfaces\n- **Go** (.go) - Functions, methods, structs\n- **Rust** (.rs) - Functions, implementations, traits\n- **C/C++** (.c, .cpp, .h) - Functions, classes, structs\n\nFor unsupported languages, the system falls back to chunk-based extraction.\n\n#### Extracted Metadata\n\nEach ingested code chunk includes:\n\n| Field | Description | Source |\n|-------|-------------|--------|\n| `content` | The actual function/class code | AST parsing |\n| `filepath` | Relative path within repository | File system |\n| `language` | Programming language | File extension |\n| `function_name` | Name of function/class/method | AST parsing |\n| `repo` | GitHub repository URL | Git remote |\n| `last_modified` | Last commit timestamp | Git log |\n| `author` | Code author (if `--extract-git-info`) | Git blame |\n| `line_range` | Start-end line numbers (if `--extract-git-info`) | AST + file position |\n| `summary` | AI-generated summary (if `--generate-summaries`) | OpenAI API |\n\n#### Performance Considerations\n\n- **Batch Size**: Larger batches (500-1000) improve throughput but use more memory\n- **Git Info Extraction**: Adds processing time but provides richer metadata\n- **AI Summaries**: Significantly increases processing time and OpenAI costs\n- **Repository Size**: Large repositories (1000+ files) may require 10-30 minutes\n- **Network**: Repository cloning speed depends on internet connection\n\n#### Example Output\n\n```\nStarting repository ingestion: https://github.com/psf/requests.git\nThis may take a few minutes depending on repository size...\n\nExtracted 1,247 code chunks from repository\nStarting batch insertion into knowledge base...\nUsing batch size: 500 for stable insertion\nTotal records to insert: 1,247\n\nInserted batch 1/3: 500 records\nInserted batch 2/3: 500 records  \nInserted batch 3/3: 247 records\n\nSuccessfully ingested 1,247 code chunks\n\nLanguage breakdown:\n  python: 1,198 chunks\n  markdown: 31 chunks\n  yaml: 12 chunks\n  shell: 6 chunks\n```\n\n### AI Tables Management\n\n```bash\n# Initialize AI tables\npython main.py ai:init\n\n# Analyze code\npython main.py ai:analyze \"def authenticate_user(username, password): return username == 'admin'\" --all\n\n# List AI tables\npython main.py ai:list\n\n# Reset AI tables\npython main.py ai:reset\n```\n\nAI Analysis Types:\n- `--classify`: Code purpose classification\n- `--explain`: Natural language explanation\n- `--docstring`: Generate documentation\n- `--tests`: Suggest test cases\n- `--all`: Run all analysis types\n\n### Repository Sync Jobs\n\n```bash\n# Create sync job\npython main.py kb:sync https://github.com/org/repo-name.git\n\n# Custom schedule\npython main.py kb:sync https://github.com/org/repo.git --schedule \"EVERY 12 HOURS\"\n\n# List jobs\npython main.py kb:sync:list\n\n# Delete job\npython main.py kb:sync:delete sync_github_com_org_repo_git\n```\n\n### Workflow Demo\n\nExperience the complete AI-enhanced semantic search workflow:\n\n```bash\n# Complete workflow demonstration\npython demo.py workflow \"decorator function\" --limit 3\n\n# Create SQL view joining KB with AI tables\npython demo.py create-view\n\n# Query integrated workflow view\npython demo.py query-view --limit 5\n```\n\nDemo Features:\n- Complete pipeline demonstration (KB search to AI analysis)\n- Step-by-step workflow output\n- SQL view integration\n- Professional presentation format\n\n### Utility Commands\n\n```bash\n# Check status\npython main.py kb:status\n\n# View schema\npython main.py kb:schema\n\n# Create index\npython main.py kb:index\n\n# Reset knowledge base\npython main.py kb:reset --force\n```\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    Semantic Code Navigator                  │\n├─────────────────────────────────────────────────────────────┤\n│  CLI Interface (Click + Rich)                              │\n│  ├── kb:*        - Knowledge Base Operations               │\n│  ├── ai:*        - AI Table Management                     │\n│  └── demo:*      - Workflow Demonstrations                 │\n├─────────────────────────────────────────────────────────────┤\n│  MindsDB Client (Python SDK)                               │\n│  ├── Connection Management                                 │\n│  ├── Knowledge Base Operations                             │\n│  ├── AI Table Integration                                  │\n│  └── Batch Processing                                      │\n├─────────────────────────────────────────────────────────────┤\n│  MindsDB Knowledge Base                                     │\n│  ├── OpenAI Embeddings (text-embedding-3-large)           │\n│  ├── OpenAI Reranking (gpt-4o)                            │\n│  ├── Vector Storage \u0026 Indexing                             │\n│  └── Metadata Filtering                                    │\n├─────────────────────────────────────────────────────────────┤\n│  AI Tables (Generative AI Models)                          │\n│  ├── code_classifier - Purpose Classification              │\n│  ├── code_explainer - Natural Language Explanations       │\n│  ├── docstring_generator - Documentation Generation        │\n│  ├── test_case_outliner - Test Case Suggestions           │\n│  └── result_rationale - Search Match Explanations         │\n├─────────────────────────────────────────────────────────────┤\n│  Git Repository Ingestion Pipeline                         │\n│  ├── Repository Cloning \u0026 Discovery                        │\n│  ├── Function/Class Extraction                             │\n│  ├── Metadata Extraction                                   │\n│  └── Batch Processing                                      │\n└─────────────────────────────────────────────────────────────┘\n```\n\n## Knowledge Base Schema\n\n### Storage Structure\n- `chunk_content`: Code content with embedded metadata\n- `chunk_id`: Unique identifier\n- `metadata`: MindsDB internal metadata\n- `relevance`: Semantic search relevance score\n- `distance`: Vector similarity distance\n\n### Extracted Metadata\n- `filepath`: Relative path within repository\n- `language`: Programming language\n- `function_name`: Function/class/method name\n- `repo`: GitHub repository URL\n- `last_modified`: Git commit timestamp\n- `author`: Git commit author (optional)\n- `line_range`: Start-end line numbers (optional)\n\n### Supported Languages\n- Python, JavaScript, TypeScript, Java, Go, Rust, C/C++\n- Fallback chunking for unsupported languages\n\n## Example Queries\n\n```bash\n# Authentication code\npython main.py kb:query \"user authentication and login validation\"\n\n# HTTP handling\npython main.py kb:query \"http request\" --language python --limit 5\n\n# Error patterns\npython main.py kb:query \"exception handling\" --relevance-threshold 0.7\n\n# Test files\npython main.py kb:query \"test validation\" --filepath \"*/test*\"\n\n# Specific functions\npython main.py kb:query \"database connection\" --function \"*connect*\"\n\n# Recent changes\npython main.py kb:query \"authentication\" --author \"john@example.com\" --since \"2024-01-01\"\n```\n\n## Quick Start\n\n```bash\n# 1. Initialize\npython main.py kb:init\n\n# 2. Ingest repository\npython main.py kb:ingest https://github.com/psf/requests.git --extract-git-info\n\n# 3. Search\npython main.py kb:query \"http request handling\" --limit 5\n\n# 4. Check status\npython main.py kb:status\n\n# 5. Reset for new testing\npython main.py kb:reset --force\n```\n\n## Contributing\n\nThis project is part of MindsDB Knowledge Base stress testing. Contributions welcome.\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests if applicable\n5. Submit a pull request\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## Stress Testing\n\nThe project includes a comprehensive stress testing suite that evaluates the complete workflow across 10 GitHub repositories of varying sizes (30 to 120 files).\n\n### Stress Test Features\n\n- **Complete Workflow Testing**: Tests KB creation, data ingestion, indexing, semantic search, and AI analysis\n- **10 Repository Coverage**: From small Flask apps to large projects like Linux kernel and WebKit\n- **Serial Execution**: Tests run one after another to prevent memory issues\n- **Memory Management**: Automatic KB reset after each test to free memory\n- **Real-time Reporting**: Beautiful markdown reports with timestamps and metrics\n- **Performance Analysis**: Tracks ingestion speed, search response times, and success rates\n- **Failure Analysis**: Detailed error reporting and recommendations\n\n### Repository Test Matrix\n\n| Size Category | File Count | Examples | Batch Size |\n|---------------|------------|----------|------------|\n| Small | 50-150 | Flask, Express, Gin examples | 100-300 |\n| Medium | 200-450 | Django, React, Spring Boot | 200-400 |\n| Medium-Large | 500-700 | Angular, NestJS, FastAPI | 300-500 |\n| Large | 800-1000 | Kubernetes, TensorFlow.js | 400-600 |\n| Very Large | 1200-3000+ | VS Code, Chromium, Linux | 500-1000 |\n\n### Running Stress Tests\n\n```bash\n# Full stress test (10 repositories)\npython stress_test.py\n\n# View help\npython stress_test.py --help\n```\n\n### Test Workflow\n\nEach repository test follows this workflow:\n\n1. **KB Creation** - Initialize fresh knowledge base\n2. **AI Tables Setup** - Create AI analysis models\n3. **Data Ingestion** - Clone repo and extract code chunks\n4. **Index Creation** - Optimize for search performance\n5. **Semantic Search** - Test 5 different queries\n6. **AI Analysis** - Test code classification and explanation\n7. **Cleanup** - Reset KB to free memory for next test\n\n### Output Reports\n\nThe stress test generates detailed markdown reports including:\n\n- **Real-time Progress**: Timestamped updates during execution\n- **Performance Metrics**: Ingestion speed, search response times\n- **Success Rates**: Pass/fail statistics for each workflow step\n- **Memory Usage**: Peak memory consumption tracking\n- **Failure Analysis**: Detailed error logs and recommendations\n- **Comparative Analysis**: Performance across different repository sizes\n\n### Example Report Sections\n\n```markdown\n### Testing Repository: django-blog\n- **URL:** https://github.com/django/django\n- **Estimated Files:** 250\n- **Language:** Python\n- **Batch Size:** 200\n\n#### Step 1: Knowledge Base Creation\n**KB Creation:** Success in 2.34s\n\n#### Step 3: Data Ingestion\n**Data Ingestion:** Success in 45.67s\n  - Files Processed: 247\n  - Chunks Extracted: 1,234\n\n#### Step 5: Semantic Search Testing\n**Semantic Search:** Success\n  - Queries Tested: 5\n  - Average Response Time: 1.23s\n  - Total Results: 47\n```\n\n### Prerequisites for Stress Testing\n\n- MindsDB running locally (`docker-compose up`)\n- OpenAI API key configured in `.env`\n- Sufficient disk space (~5GB for temporary repositories)\n- Stable internet connection for repository cloning\n- 8GB+ RAM recommended for large repositories\n\n### Cost Considerations\n\n- **Embedding Costs**: ~$0.10-0.50 per repository (varies by size)\n- **AI Analysis Costs**: ~$0.05-0.20 per repository\n- **Total Estimated Cost**: $5-10 for full 10-repository test\n- **No Summary Generation**: Disabled to reduce OpenAI costs\n\nThe stress test is designed to thoroughly validate the system's reliability, performance, and scalability across diverse codebases while providing actionable insights for optimization.\n\n## AI Agents\n\nThe Semantic Code Navigator includes a powerful AI agent system that provides specialized code analysis and assistance. These agents have full access to your ingested codebase and can provide expert-level insights in specific domains.\n\n### Agent Features\n\n- **Template-Based Creation**: Pre-configured agent templates for different specializations\n- **Knowledge Base Integration**: Agents have full access to your ingested codebase\n- **Natural Language Interaction**: Query agents with natural language questions\n- **Specialized Expertise**: Each agent is optimized for specific domains (code review, architecture, security)\n- **Rich Output Formatting**: Beautiful formatted responses with structured analysis\n\n### Available Agent Templates\n\n| Template | Specialization | Model | Description |\n|----------|----------------|-------|-------------|\n| `code-reviewer` | Code Review | gpt-4o | Expert code reviewer focusing on security, performance, and best practices |\n| `architect` | System Architecture | gpt-4o | Software architect for system-level analysis and design patterns |\n| `security-auditor` | Security Analysis | gpt-4o | Security expert for vulnerability assessment and compliance |\n\nThe agent system transforms your ingested codebase into an interactive knowledge resource, providing expert-level analysis and guidance tailored to your specific code and requirements.\n\n## Acknowledgments\n\n- [MindsDB](https://mindsdb.com/) for the Knowledge Base platform\n- [MindsDB Python SDK](https://mindsdb.com/blog/introduction-to-python-sdk-interact-with-mindsdb-directly-from-python) for integration\n- OpenAI for embedding and reranking models\n\n---\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeptanshu-sankhwar%2Fsemantic-code-navigator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeptanshu-sankhwar%2Fsemantic-code-navigator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeptanshu-sankhwar%2Fsemantic-code-navigator/lists"}