{"id":50650002,"url":"https://github.com/prakulhiremath/semantix","last_synced_at":"2026-06-07T18:00:32.630Z","repository":{"id":363143447,"uuid":"1262123307","full_name":"prakulhiremath/SEMANTIX","owner":"prakulhiremath","description":"A Rust-based PostgreSQL extension that makes relational query optimizers natively aware of LLM token costs, semantic entropy, and latency budgets.","archived":false,"fork":false,"pushed_at":"2026-06-07T16:38:39.000Z","size":114,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-07T17:23:56.438Z","etag":null,"topics":["database-engine","llmops","postgres-extension","query-optimizer","rag-pipeline","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prakulhiremath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-07T15:51:37.000Z","updated_at":"2026-06-07T16:40:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/prakulhiremath/SEMANTIX","commit_stats":null,"previous_names":["prakulhiremath/semantix"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/prakulhiremath/SEMANTIX","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakulhiremath%2FSEMANTIX","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakulhiremath%2FSEMANTIX/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakulhiremath%2FSEMANTIX/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakulhiremath%2FSEMANTIX/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prakulhiremath","download_url":"https://codeload.github.com/prakulhiremath/SEMANTIX/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakulhiremath%2FSEMANTIX/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34031952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-07T02:00:07.652Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database-engine","llmops","postgres-extension","query-optimizer","rag-pipeline","rust"],"created_at":"2026-06-07T18:00:16.299Z","updated_at":"2026-06-07T18:00:32.576Z","avatar_url":"https://github.com/prakulhiremath.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SEMANTIX: Learned Semantic Cost Models for LLM-Native Relational Engines\n\n[![DOI](https://zenodo.org/badge/1262123307.svg)](https://doi.org/10.5281/zenodo.20582657)\n[![Read on Medium](https://img.shields.io/badge/Read_on-Medium-000000?style=flat-square\u0026logo=medium\u0026logoColor=white)](https://medium.com/@prakulhiremath/the-database-finally-learned-to-talk-to-ai-7413f3f617d6)\n\n\u003e Treating AI/LLM inference as a core database engine primitive with information-theoretic cost modeling.\n\n## Overview\n\nSEMANTIX implements a paradigm shift in relational query optimization by treating LLM inference as a first-class database primitive, coupled with learned semantic cost estimation. Current decoupled architectures silo LLM-based retrieval from cost-aware query planning, resulting in token waste, semantic misalignment, and unbounded latency.\n\n### Key Contributions\n\n1. **Formal Cost Architecture** (Equation 1): Unified cost model embedding semantic entropy, relational context preservation, and execution schedule conditioning\n2. **Bidirectional Semantic Anchors** (Equation 4): Learned projections mapping NL intent to cost-parametric logical plans\n3. **Adaptive Token Scheduling** (Algorithm 1): Dynamic token allocation under latency constraints using Lagrangian relaxation\n\n### Performance Results\n\n- **3.2× reduction in inference token cost** (vs. Classical PostgreSQL)\n- **1.8× speedup in latency** (25.3ms vs 45.3ms)\n- **97.1% semantic accuracy** maintained\n- **65.6% energy reduction** compared to classical systems\n\n## System Architecture\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                  SEMANTIX Query Optimizer                   │\n├─────────────────────────────────────────────────────────────┤\n│                                                             │\n│  Phase 1: Semantic Parsing                                  │\n│  ├─ NL Query → Bidirectional Semantic Anchor                │\n│  └─ Output: LogicalPlan + Initial Cost Estimates            │\n│                                                             │\n│  Phase 2: Cost Refinement                                   │\n│  ├─ Learned Cost Model (GBDT)                               │\n│  └─ Output: Refined token cost estimates                    │\n│                                                             │\n│  Phase 3: Adaptive Token Scheduling                         │\n│  ├─ Constrained Optimization (Lagrangian)                   │\n│  └─ Output: Token allocation schedule                       │\n│                                                             │\n│  Phase 4: Execution + Feedback Loop                         │\n│  ├─ Execute with schedule                                   │\n│  └─ Update cost model with actual execution data            │\n│                                                             │\n└─────────────────────────────────────────────────────────────┘\n```\n\n## Installation\n\n### Prerequisites\n\n- Rust 1.70+ ([Install](https://rustup.rs/))\n- PostgreSQL 14+ ([Install](https://www.postgresql.org/download/))\n- Python 3.10+ (for data generation scripts)\n- 8GB RAM, 4-core CPU minimum\n- NVIDIA GPU (optional, for accelerated inference)\n\n### Quick Start\n\n#### 1. Clone Repository\n\n```bash\ngit clone https://github.com/novas-workshop-2026/learned-semantic-costs.git\ncd semantix\n```\n\n#### 2. Build Project\n\n```bash\n# Release build (optimized)\ncargo build --release\n\n# Or development build\ncargo build\n```\n\n#### 3. Initialize Database\n\n```bash\n# Create PostgreSQL database\ncreatedb semantix\n\n# Initialize schema\npsql -d semantix -f schema/tpch_schema.sql\n```\n\n#### 4. Generate Test Data\n\n```bash\n# Generate TPC-H with semantic annotations\ncargo run --release --bin data-generator\n\n# Load into PostgreSQL\npsql -d semantix -c \"COPY orders FROM 'tpch_orders_semantic.csv' CSV HEADER;\"\n```\n\n#### 5. Profile System\n\n```bash\n# Profile operator latency for learned cost models\ncargo run --release --bin cost-profiler\n```\n\n#### 6. Run Benchmark\n\n```bash\n# Execute comprehensive benchmark suite\ncargo run --release --bin benchmark\n```\n\n## Usage\n\n### Running the SEMANTIX Daemon\n\n```bash\n# Start optimizer service\ncargo run --release --bin semantix-daemon\n```\n\n### Programmatic API\n\n```rust\nuse semantix::SemanticQueryOptimizer;\n\n#[tokio::main]\nasync fn main() -\u003e anyhow::Result\u003c()\u003e {\n    // Initialize optimizer\n    let mut optimizer = SemanticQueryOptimizer::new(\n        \"postgresql://localhost/semantix\"\n    ).await?;\n\n    // Execute query with full semantic optimization\n    let result = optimizer.optimize_and_execute(\n        \"SELECT * FROM orders WHERE custkey = 1\"\n    ).await?;\n\n    // Check metrics\n    let metrics = optimizer.get_metrics();\n    println!(\"Tokens: {}, Latency: {}ms, Accuracy: {:.2}%\",\n        metrics.avg_token_cost,\n        metrics.avg_latency_ms,\n        metrics.avg_semantic_accuracy * 100.0\n    );\n\n    // Provide feedback for continuous learning\n    optimizer.feedback(\u0026result.context);\n\n    Ok(())\n}\n```\n\n### Command-Line Interface\n\n```bash\n# Profile specific query\ncargo run --release --bin benchmark -- --query \"SELECT * FROM orders LIMIT 100\"\n\n# Generate data with custom size\ncargo run --release --bin data-generator -- --scale-factor 10\n\n# Profile operator latencies\ncargo run --release --bin cost-profiler -- --operators \"Scan,Filter,Join\"\n```\n\n## Configuration\n\n### Configuration File\n\nCreate `semantix.toml`:\n\n```toml\n[anchor_config]\nencoder_model_path = \"models/bert-encoder-semantic.bin\"\ndecoder_model_path = \"models/bert-decoder-semantic.bin\"\nmax_sequence_length = 512\nembedding_dim = 768\nsemantic_drift_threshold = 0.15\n\n[cost_model_config]\nmodel_type = \"gbdt\"\nmodel_path = \"models/cost_model.xgb\"\nentropy_weight = 1.0\ndelay_weight = 0.3\nstaleness_weight = 0.5\nmin_token_budget = 100\nmax_token_budget = 10000\n\n[scheduler_config]\nmax_latency_ms = 50\nlatency_sigma = 0.1\nalpha = 0.01\nconvergence_threshold = 0.001\nmax_iterations = 1000\n\n[database]\nurl = \"postgresql://localhost/semantix\"\nlog_level = \"info\"\n```\n\n### Environment Variables\n\n```bash\nexport DATABASE_URL=\"postgresql://user:password@localhost/semantix\"\nexport LOG_LEVEL=\"debug\"\nexport SEMANTIX_CONFIG=\"path/to/semantix.toml\"\n```\n\n## Project Structure\n\n```\nsemantix/\n├── Cargo.toml                 # Rust dependencies\n├── src/\n│   ├── lib.rs               # Main library exports\n│   ├── semantic_anchors.rs  # NL → LogicalPlan translation\n│   ├── cost_model.rs        # Learned cost estimation\n│   ├── scheduler.rs         # Adaptive token scheduling (Algorithm 1)\n│   ├── database.rs          # PostgreSQL integration\n│   ├── executor.rs          # Query execution engine\n│   ├── metrics.rs           # Performance tracking\n│   ├── config.rs            # Configuration management\n│   ├── errors.rs            # Error types\n│   └── bin/\n│       ├── daemon.rs        # Main optimizer service\n│       ├── profiler.rs      # Latency profiler\n│       ├── data_gen.rs      # TPC-H data generation\n│       └── benchmark.rs     # Performance evaluation\n├── schema/\n│   └── tpch_schema.sql      # PostgreSQL schema\n├── tests/\n│   ├── integration_tests.rs # End-to-end tests\n│   └── unit_tests.rs        # Component tests\n├── docker/\n│   ├── Dockerfile           # Container image\n│   └── docker-compose.yml   # Multi-container setup\n└── README.md                # This file\n```\n\n## Mathematical Foundations\n\n### Equation 1: Semantic Token Cost\n\n```\nC_sem(π, σ) = Σ_i [H(i | Σ^ctx(i)) + γ·delay(o_j, σ) + β·staleness(o_j, σ)]\n```\n\nwhere:\n- `H(i | Σ^ctx(i))` = Conditional semantic entropy\n- `γ` = Delay weight parameter\n- `β` = Staleness weight parameter\n- `σ` = Execution schedule\n\n### Equation 4: Bidirectional Semantic Anchor\n\n```\nφ(NL query) = (LogicalPlan, {c_1, ..., c_k})\n```\n\nMaps natural language to plan with cost decorations.\n\n### Algorithm 1: Adaptive Token Scheduling\n\nConstrained optimization using Lagrangian relaxation:\n\n```\nminimize   Σ_j c_j^allocated\nsubject to Σ_j latency(o_j, c_j^allocated) ≤ L_max\n           c_j^min ≤ c_j^allocated ≤ c_j^max\n```\n\n## Evaluation Methodology\n\n### Metrics\n\n1. **Inference Token Cost** (K tokens) - Total tokens consumed\n2. **End-to-End Latency** (ms) - Query execution time\n3. **Semantic Accuracy** (%) - Tuple relevance accuracy\n4. **Compute Energy** (Wh) - GPU/CPU energy consumption\n\n### Baselines\n\n1. Classical PostgreSQL - Standard cost model\n2. RAG-Optimized - LLM retrieval post-planning\n3. Semantic Entropy - No schedule conditioning\n4. SEMANTIX - Full system (this work)\n\n### Benchmark Queries\n\nUses extended TPC-H queries with semantic metadata:\n- Q1-Q5: Standard TPC-H with cardinality analysis\n- Q6-Q10: Join-heavy queries with semantic predicates\n- Q11-Q15: Aggregation-focused queries\n- Q16-Q22: Complex multi-way joins\n\n## Docker Deployment\n\n### Build Image\n\n```bash\ndocker build -f docker/Dockerfile -t semantix:latest .\n```\n\n### Run Container\n\n```bash\ndocker run -it --rm \\\n  -e DATABASE_URL=\"postgresql://postgres:password@db:5432/semantix\" \\\n  -p 8080:8080 \\\n  semantix:latest\n```\n\n### Docker Compose\n\n```bash\ndocker-compose -f docker/docker-compose.yml up -d\n```\n\n## Performance Profiling\n\n### CPU Profiling\n\n```bash\ncargo install flamegraph\ncargo flamegraph --bin benchmark\n# Open flamegraph.svg in browser\n```\n\n### Memory Profiling\n\n```bash\nvalgrind --tool=massif ./target/release/benchmark\nms_print massif.out.\u003cpid\u003e\n```\n\n### Latency Profiling\n\n```bash\ncargo run --release --bin cost-profiler -- --detailed-report\n```\n\n## Testing\n\n### Run All Tests\n\n```bash\ncargo test\ncargo test --doc\ncargo test --all-features\n```\n\n### Integration Tests\n\n```bash\n# Start PostgreSQL first\ncargo test --test integration_tests -- --test-threads=1\n```\n\n### Benchmark Tests\n\n```bash\ncargo bench\n```\n\n## Contributing\n\nWe welcome contributions! Please:\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit changes (`git commit -m 'Add amazing feature'`)\n4. Push to branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\nSee CONTRIBUTING.md for detailed guidelines.\n\n## Citation\n\nIf you use SEMANTIX in your research, please cite:\n\n```bibtex\n@inproceedings{semantix2026,\n  title={Learned Semantic Cost Models for Adaptive Token-Efficient \n         Query Optimization in LLM-Native Relational Engines},\n  author={Prakul Sunil Hiremath},\n  year={2026}\n}\n```\n\n## License\n\nApache License 2.0 - See LICENSE file for details.\n\n---\n\n**Last Updated**: June 2026  \n**Status**: Production-Ready v0.1.0  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprakulhiremath%2Fsemantix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprakulhiremath%2Fsemantix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprakulhiremath%2Fsemantix/lists"}