{"id":25658734,"url":"https://github.com/justrach/bhumi","last_synced_at":"2025-08-08T17:14:31.118Z","repository":{"id":275084214,"uuid":"925021610","full_name":"justrach/bhumi","owner":"justrach","description":"⚡ Bhumi – The fastest AI inference client for Python, built with Rust for unmatched speed, efficiency, and scalability 🚀","archived":false,"fork":false,"pushed_at":"2025-06-14T06:54:16.000Z","size":93608,"stargazers_count":56,"open_issues_count":6,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-14T07:24:30.995Z","etag":null,"topics":["inference","llm","python","rust"],"latest_commit_sha":null,"homepage":"https://bhumi.trilok.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/justrach.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-31T04:24:55.000Z","updated_at":"2025-06-14T06:54:20.000Z","dependencies_parsed_at":"2025-02-16T09:18:00.513Z","dependency_job_id":"dd9d7aea-186a-49fb-8ee9-6de93218c2d4","html_url":"https://github.com/justrach/bhumi","commit_stats":null,"previous_names":["justrach/bhumi"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/justrach/bhumi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fbhumi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fbhumi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fbhumi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fbhumi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/justrach","download_url":"https://codeload.github.com/justrach/bhumi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/justrach%2Fbhumi/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269457912,"owners_count":24420317,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inference","llm","python","rust"],"created_at":"2025-02-24T00:28:17.568Z","updated_at":"2025-08-08T17:14:31.107Z","avatar_url":"https://github.com/justrach.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"/assets/bhumi_logo.png\" alt=\"Bhumi Logo\" width=\"1600\"/\u003e\n\u003c/p\u003e\n\n\n# 🌍 **BHUMI - The Fastest AI Inference Client** ⚡\n\n## **Introduction**\nBhumi is the fastest AI inference client, built with Rust for Python. It is designed to maximize performance, efficiency, and scalability, making it the best choice for LLM API interactions. \n\n### **Why Bhumi?**\n- 🚀 **Fastest AI inference client** – Outperforms alternatives with **2-3x higher throughput**\n- ⚡ **Built with Rust for Python** – Achieves high efficiency with low overhead\n- 🌐 **Supports multiple AI providers** – OpenAI, Anthropic, Google Gemini, Groq, SambaNova, and more\n- 🔄 **Streaming and async capabilities** – Real-time responses with Rust-powered concurrency\n- 🔁 **Automatic connection pooling and retries** – Ensures reliability and efficiency\n- 💡 **Minimal memory footprint** – Uses up to **60% less memory** than other clients\n- 🏗 **Production-ready** – Optimized for high-throughput applications\n\nBhumi (भूमि) is Sanskrit for **Earth**, symbolizing **stability, grounding, and speed**—just like our inference engine, which ensures rapid and stable performance. 🚀\n\n## Installation\n```bash\npip install bhumi\n```\n\n## Quick Start\n\n### OpenAI Example\n```python\nimport asyncio\nfrom bhumi.base_client import BaseLLMClient, LLMConfig\nimport os\n\napi_key = os.getenv(\"OPENAI_API_KEY\")\n\nasync def main():\n    config = LLMConfig(\n        api_key=api_key,\n        model=\"openai/gpt-4o\",\n        debug=True\n    )\n    \n    client = BaseLLMClient(config)\n    \n    response = await client.completion([\n        {\"role\": \"user\", \"content\": \"Tell me a joke\"}\n    ])\n    print(f\"Response: {response['text']}\")\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## ⚡ **Performance Optimizations**\n\nBhumi includes cutting-edge performance optimizations that make it **2-3x faster** than alternatives:\n\n### 🧠 **MAP-Elites Buffer Strategy**\n- **Ultra-fast archive loading** with Satya validation + orjson parsing (**3x faster** than standard JSON)\n- **Trained buffer configurations** optimized through evolutionary algorithms  \n- **Automatic buffer adjustment** based on response patterns and historical data\n- **Type-safe validation** with comprehensive error checking\n- **Secure loading** without unsafe `eval()` operations\n\n### 📊 **Performance Status Check**\nCheck if you have optimal performance with the built-in diagnostics:\n\n```python\nfrom bhumi.utils import print_performance_status\n\n# Check optimization status\nprint_performance_status()\n# 🚀 Bhumi Performance Status\n# ✅ Optimized MAP-Elites archive loaded  \n# ⚡ Optimization Details:\n#    • Entries: 15,644 total, 15,644 optimized\n#    • Coverage: 100.0% of search space\n#    • Loading: Satya validation + orjson parsing (3x faster)\n```\n\n### 🏆 **Archive Distribution**\nWhen you install Bhumi, you automatically get:\n- Pre-trained MAP-Elites archive for optimal buffer sizing\n- Fast orjson-based JSON parsing (2-3x faster than standard `json`)\n- Satya-powered type validation for bulletproof data loading\n- Performance metrics and diagnostics\n\n### Gemini Example\n```python\nimport asyncio\nfrom bhumi.base_client import BaseLLMClient, LLMConfig\nimport os\n\napi_key = os.getenv(\"GEMINI_API_KEY\")\n\nasync def main():\n    config = LLMConfig(\n        api_key=api_key,\n        model=\"gemini/gemini-2.0-flash\",\n        debug=True\n    )\n    \n    client = BaseLLMClient(config)\n    \n    response = await client.completion([\n        {\"role\": \"user\", \"content\": \"Tell me a joke\"}\n    ])\n    print(f\"Response: {response['text']}\")\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## Streaming Support\nAll providers support streaming responses:\n\n```python\nasync for chunk in await client.completion([\n    {\"role\": \"user\", \"content\": \"Write a story\"}\n], stream=True):\n    print(chunk, end=\"\", flush=True)\n```\n\n## 📊 **Benchmark Results**\nOur latest benchmarks show significant performance advantages across different metrics:\n![alt text](gemini_averaged_comparison_20250131_154711.png)\n\n### ⚡ Response Time\n- LiteLLM: 13.79s\n- Native: 5.55s\n- Bhumi: 4.26s\n- Google GenAI: 6.76s\n\n### 🚀 Throughput (Requests/Second)\n- LiteLLM: 3.48\n- Native: 8.65\n- Bhumi: 11.27\n- Google GenAI: 7.10\n\n### 💾 Peak Memory Usage (MB)\n- LiteLLM: 275.9MB\n- Native: 279.6MB\n- Bhumi: 284.3MB\n- Google GenAI: 284.8MB\n\nThese benchmarks demonstrate Bhumi's superior performance, particularly in throughput where it outperforms other solutions by up to 3.2x.\n\n## Configuration Options\nThe LLMConfig class supports various options:\n- `api_key`: API key for the provider\n- `model`: Model name in format \"provider/model_name\"\n- `base_url`: Optional custom base URL\n- `max_retries`: Number of retries (default: 3)\n- `timeout`: Request timeout in seconds (default: 30)\n- `max_tokens`: Maximum tokens in response\n- `debug`: Enable debug logging\n\n## 🎯 **Why Use Bhumi?**\n✔ **Open Source:** Apache 2.0 licensed, free for commercial use  \n✔ **Community Driven:** Welcomes contributions from individuals and companies  \n✔ **Blazing Fast:** **2-3x faster** than alternative solutions  \n✔ **Resource Efficient:** Uses **60% less memory** than comparable clients  \n✔ **Multi-Model Support:** Easily switch between providers  \n✔ **Parallel Requests:** Handles **multiple concurrent requests** effortlessly  \n✔ **Flexibility:** Debugging and customization options available  \n✔ **Production Ready:** Battle-tested in high-throughput environments\n\n## 🤝 **Contributing**\nWe welcome contributions from the community! Whether you're an individual developer or representing a company like Google, OpenAI, or Anthropic, feel free to:\n\n- Submit pull requests\n- Report issues\n- Suggest improvements\n- Share benchmarks\n- Integrate our optimizations into your libraries (with attribution)\n\n## 📜 **License**\nApache 2.0\n\n🌟 **Join our community and help make AI inference faster for everyone!** 🌟\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustrach%2Fbhumi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjustrach%2Fbhumi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustrach%2Fbhumi/lists"}