{"id":28385982,"url":"https://github.com/kode-rex/insight-mesh","last_synced_at":"2026-05-08T01:36:20.062Z","repository":{"id":286031070,"uuid":"960133868","full_name":"Kode-Rex/insight-mesh","owner":"Kode-Rex","description":"The NextGen RAG Stack","archived":false,"fork":false,"pushed_at":"2025-06-13T17:05:59.000Z","size":1974,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-13T18:24:47.610Z","etag":null,"topics":["agentic-ai","agents","enterprise","permissions","rag","vibe-coding"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kode-Rex.png","metadata":{"files":{"readme":"README-migrations.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-03T23:10:02.000Z","updated_at":"2025-06-12T23:35:15.000Z","dependencies_parsed_at":"2025-06-06T02:26:04.755Z","dependency_job_id":"8688ed3e-06d2-421b-abf2-9ad134fc4c34","html_url":"https://github.com/Kode-Rex/insight-mesh","commit_stats":null,"previous_names":["t-rav/rag-in-a-box","t-rav/insight-mesh"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Kode-Rex/insight-mesh","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kode-Rex%2Finsight-mesh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kode-Rex%2Finsight-mesh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kode-Rex%2Finsight-mesh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kode-Rex%2Finsight-mesh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kode-Rex","download_url":"https://codeload.github.com/Kode-Rex/insight-mesh/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kode-Rex%2Finsight-mesh/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262067677,"owners_count":23253647,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-ai","agents","enterprise","permissions","rag","vibe-coding"],"created_at":"2025-05-30T12:37:54.159Z","updated_at":"2026-05-08T01:36:15.041Z","avatar_url":"https://github.com/Kode-Rex.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Database Migrations\n\nThis project uses **standard migration tools** for each database system with **automatic type detection**:\n\n## Migration Tools\n\n### PostgreSQL: Alembic\n- **Tool**: [Alembic](https://alembic.sqlalchemy.org/) (industry standard)\n- **Type**: `sql`\n- **Location**: `.weave/migrations/{database}/`\n- **Commands**: `weave db migrate {database}`\n\n### Neo4j: Neo4j Migrations\n- **Tool**: [neo4j-migrations](https://github.com/michael-simons/neo4j-migrations) (official)\n- **Type**: `graph`\n- **Location**: `.weave/migrations/neo4j/cypher/`\n- **Commands**: `weave db migrate neo4j`\n\n### Elasticsearch: HTTP-based Migrations\n- **Tool**: HTTP requests (compatible with [elasticsearch-evolution](https://github.com/senacor/elasticsearch-evolution))\n- **Type**: `search`\n- **Location**: `.weave/migrations/elasticsearch/scripts/`\n- **Commands**: `weave db migrate elasticsearch`\n\n## Quick Start\n\n### Smart Migration (Recommended)\nThe system automatically detects database types and uses the appropriate migration tool:\n\n```bash\n# Show all configured databases and their types\nweave db info\n\n# Migrate any database (auto-detects type and tool)\nweave db migrate slack          # Uses Alembic (SQL)\nweave db migrate neo4j          # Uses neo4j-migrations (Graph)\nweave db migrate elasticsearch  # Uses HTTP-based (Search)\n\n# Migrate all databases\nweave db migrate all\n```\n\n### Run All Migrations\n```bash\n# Migrate all database systems\nweave db migrate all\n\n# Preview what would be migrated\nweave db migrate all --dry-run\n```\n\n### Individual Database Migrations\n\n#### PostgreSQL (Alembic) - Type: `sql`\n```bash\n# Create new migration\nweave db create slack \"add user preferences\"\nweave db create insightmesh \"add context metadata\"\n\n# Apply migrations (smart command)\nweave db migrate slack\nweave db migrate insightmesh\n\n# Show current state\nweave db status slack\nweave db history slack\n```\n\n#### Neo4j (neo4j-migrations) - Type: `graph`\n```bash\n# Apply migrations (smart command)\nweave db migrate neo4j\n\n# Note: Neo4j migrations are handled automatically by 'weave db migrate neo4j'\n```\n\n#### Elasticsearch (HTTP-based) - Type: `search`\n```bash\n# Apply migrations (smart command)\nweave db migrate elasticsearch\n\n# Note: Elasticsearch migrations are handled automatically by 'weave db migrate elasticsearch'\n```\n\n## Database Configuration\n\nDatabases are configured in `.weave/config.json` with type information:\n\n```json\n{\n  \"databases\": {\n    \"slack\": {\n      \"type\": \"sql\",\n      \"migration_tool\": \"alembic\",\n      \"description\": \"Slack integration data\"\n    },\n    \"neo4j\": {\n      \"type\": \"graph\", \n      \"migration_tool\": \"neo4j-migrations\",\n      \"description\": \"Knowledge graph database\"\n    },\n    \"elasticsearch\": {\n      \"type\": \"search\",\n      \"migration_tool\": \"elasticsearch-evolution\", \n      \"description\": \"Search and analytics engine\"\n    }\n  }\n}\n```\n\n## Creating New Migrations\n\n### PostgreSQL Migrations\n```bash\n# Auto-generate from model changes\nweave db create slack \"add new table\" --auto\n\n# Create empty migration\nweave db create insightmesh \"custom changes\"\n```\n\n### Neo4j Migrations\nCreate a new file in `.weave/migrations/neo4j/cypher/`:\n```cypher\n// V002__add_user_relationships.cypher\n// Add user relationship constraints and indexes\n\nCREATE CONSTRAINT user_id IF NOT EXISTS\nFOR (u:User) REQUIRE u.id IS UNIQUE;\n\nCREATE INDEX user_email IF NOT EXISTS\nFOR (u:User) ON (u.email);\n```\n\n### Elasticsearch Migrations\nCreate a new file in `.weave/migrations/elasticsearch/scripts/`:\n```http\n# V002__add_user_index.http\n# Add user index for authentication\n\n### Create users index\nPUT /users\nContent-Type: application/json\n\n{\n  \"settings\": {\n    \"number_of_shards\": 1,\n    \"number_of_replicas\": 1\n  },\n  \"mappings\": {\n    \"properties\": {\n      \"email\": { \"type\": \"keyword\" },\n      \"name\": { \"type\": \"text\" },\n      \"created_at\": { \"type\": \"date\" }\n    }\n  }\n}\n```\n\n## Migration File Naming\n\n### Neo4j (Cypher)\n- Format: `V{version}__{description}.cypher`\n- Example: `V001__initial_schema.cypher`\n\n### Elasticsearch (HTTP)\n- Format: `V{version}__{description}.http`\n- Example: `V001__initial_indices.http`\n\n### PostgreSQL (Alembic)\n- Auto-generated: `{revision}_{description}.py`\n- Example: `abc123_add_user_table.py`\n\n## Command Reference\n\n### Smart Commands (Recommended)\n```bash\nweave db info                    # Show all databases and types\nweave db status                  # Show migration status for all databases\nweave db migrate \u003cdatabase\u003e      # Smart migrate (auto-detects tool)\nweave db migrate all             # Migrate all databases\n```\n\n### Type-Specific Commands\n```bash\n# All databases (unified commands)\nweave db create \u003cdatabase\u003e \u003cmessage\u003e     # Create migration\nweave db migrate \u003cdatabase\u003e              # Apply migrations\nweave db status \u003cdatabase\u003e               # Show current status\nweave db history \u003cdatabase\u003e              # Show migration history\nweave db rollback \u003cdatabase\u003e             # Rollback migrations\n```\n\n## Installation Requirements\n\n### Local Development\n```bash\n# Install Python dependencies\npip install -r requirements.txt\n\n# Install Neo4j Migrations CLI (Java required)\nwget https://github.com/michael-simons/neo4j-migrations/releases/download/1.16.0/neo4j-migrations-1.16.0.zip\nunzip neo4j-migrations-1.16.0.zip\nsudo mv neo4j-migrations-1.16.0 /opt/neo4j-migrations\nsudo ln -s /opt/neo4j-migrations/bin/neo4j-migrations /usr/local/bin/neo4j-migrations\n```\n\n### Docker\n```bash\n# Build migration service (includes all tools)\ndocker-compose build migrations\n\n# Run migrations in Docker\ndocker-compose --profile migrations run migrations weave db migrate all\n```\n\n## Environment Variables\n\n```bash\n# PostgreSQL\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_USER=postgres\nPOSTGRES_PASSWORD=password\n\n# Neo4j\nNEO4J_URI=bolt://localhost:7687\nNEO4J_USER=neo4j\nNEO4J_PASSWORD=password\nNEO4J_DATABASE=neo4j\n\n# Elasticsearch\nELASTICSEARCH_HOST=localhost\nELASTICSEARCH_PORT=9200\nELASTICSEARCH_SCHEME=http\n```\n\n## Best Practices\n\n### General\n1. **Use smart commands** - `weave db migrate \u003cdatabase\u003e` auto-detects the right tool\n2. **Check tool status** - `weave db tool status` shows available migration tools\n3. **Install missing tools** - `weave db tool install` installs required dependencies\n2. **Check database info** - `weave db info` shows all databases and their types\n3. **Always backup** before running migrations in production\n4. **Test migrations** in development first\n5. **Version control** all migration files\n\n### Neo4j Specific\n1. Use `IF NOT EXISTS` for constraints and indexes\n2. Use `MERGE` instead of `CREATE` for idempotent operations\n3. Keep migrations atomic and reversible when possible\n\n### Elasticsearch Specific\n1. Use index templates for consistent settings\n2. Consider index aliases for zero-downtime updates\n3. Test mapping changes carefully (they're often irreversible)\n\n### PostgreSQL Specific\n1. Use Alembic's autogenerate feature\n2. Review generated migrations before applying\n3. Add custom data migrations when needed\n\n## Troubleshooting\n\n### Check Database Configuration\n```bash\n# Show all databases with types and tools\nweave db info\n```\n\n### Neo4j Migrations\n```bash\n# Check if Neo4j is accessible\nneo4j-migrations info -c .weave/migrations/neo4j/neo4j.conf\n\n# Validate migration files\nneo4j-migrations validate -c .weave/migrations/neo4j/neo4j.conf\n```\n\n### Elasticsearch Migrations\n```bash\n# Check Elasticsearch connectivity\ncurl http://localhost:9200/_cluster/health\n\n# List current indices\ncurl http://localhost:9200/_cat/indices?v\n```\n\n### PostgreSQL Migrations\n```bash\n# Check current revision\nweave db current slack\n\n# Show migration history\nweave db history slack\n```\n\n## Migration Flow\n\n1. **Configure** databases in `.weave/config.json` with correct types\n2. **Create** migrations using appropriate tools/formats\n3. **Apply** migrations using smart commands: `weave db migrate \u003cdatabase\u003e`\n4. **Verify** using `weave db info` and type-specific status commands\n\nThe system automatically routes to the correct migration tool based on the database type configuration! ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkode-rex%2Finsight-mesh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkode-rex%2Finsight-mesh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkode-rex%2Finsight-mesh/lists"}