{"id":41302313,"url":"https://github.com/okikorg/okik","last_synced_at":"2026-01-23T04:52:39.411Z","repository":{"id":238661965,"uuid":"797157215","full_name":"okikorg/okik","owner":"okikorg","description":"Okik is serving framework to deploy LLMs and much more.","archived":false,"fork":false,"pushed_at":"2025-07-01T08:58:07.000Z","size":9275,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-26T07:43:42.193Z","etag":null,"topics":["deeplearning","llm","llm-inference","llm-serving","llmops","machine-learning","model-serving","python"],"latest_commit_sha":null,"homepage":"https://www.okik.ai/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/okikorg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-05-07T09:55:40.000Z","updated_at":"2024-08-22T13:51:54.000Z","dependencies_parsed_at":"2024-05-20T16:14:45.518Z","dependency_job_id":"815b9ec7-2fe5-4e17-92df-85489e1e2072","html_url":"https://github.com/okikorg/okik","commit_stats":null,"previous_names":["okikorg/okik"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/okikorg/okik","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okikorg%2Fokik","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okikorg%2Fokik/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okikorg%2Fokik/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okikorg%2Fokik/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/okikorg","download_url":"https://codeload.github.com/okikorg/okik/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/okikorg%2Fokik/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28680623,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T04:33:33.518Z","status":"ssl_error","status_checked_at":"2026-01-23T04:33:30.433Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deeplearning","llm","llm-inference","llm-serving","llmops","machine-learning","model-serving","python"],"created_at":"2026-01-23T04:52:39.287Z","updated_at":"2026-01-23T04:52:39.374Z","avatar_url":"https://github.com/okikorg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\n\n*Okik* is a powerful command-line interface (CLI) tool designed to simplify the process of running various services using different frameworks on any cloud platform. With *Okik*, users can effortlessly deploy and manage their services directly on any cloud infrastructure without the complexity of handling their own infrastructure. This tool bridges the gap between development and deployment, allowing teams to focus on creating innovative applications rather than getting bogged down in infrastructure management.\n\nKey features of *Okik* include:\n1. Managed Kubernetes clusters: Deploy services on Kubernetes clusters without the need to manage the underlying infrastructure. *Okik* handles all aspects of cluster management, including provisioning, scaling, and monitoring, allowing users to focus on building their applications.\n2. Multi-cloud support: Deploy services across various cloud providers seamlessly. This feature allows for greater flexibility and prevents vendor lock-in, enabling users to choose the best cloud solution for their specific needs.\n3. Framework agnostic: Compatible with a wide range of popular frameworks and technologies. Use Hugging Face, FastAPI, TensorFlow, PyTorch, and more to build your services, all managed by *Okik*.\n4. LLM and GenAI integration: Easily incorporate Large Language Models and Generative AI capabilities into your applications. *Okik* provides seamless integration with popular LLM frameworks and GenAI tools, enabling advanced natural language processing and content generation within your services.\n5. Automated predictive scaling: Easily scale your services up or down based on demand. *Okik* automatically adjusts resources to meet your application's needs, ensuring optimal performance and cost-efficiency, even for resource-intensive LLM and GenAI workloads.\n6. Built-in monitoring: Keep track of your services' performance and health with integrated monitoring tools. This feature provides real-time insights into your application's behavior, helping you identify and address issues promptly, including specialized metrics for LLM and GenAI components.\n7. Streamlined deployment: Quickly deploy services with a single command. This simplifies the deployment process, reducing the time and effort required to get your application up and running, including complex LLM and GenAI-powered services.\n8. Easy-to-use CLI: Simplify the deployment process with an intuitive command-line interface. The CLI is designed to be user-friendly, even for those who may not have extensive experience with cloud deployments or AI model management.\n\n*Okik* streamlines the development and deployment process, allowing developers to focus on building great applications rather than managing infrastructure. It abstracts away the complexities of cloud deployment, making it accessible to developers of all skill levels. Whether you're working on a small project or a large-scale enterprise application, *Okik* provides the flexibility and ease-of-use to meet your needs.\n\nBy leveraging *Okik*, teams can significantly reduce the time and resources spent on infrastructure management, leading to faster development cycles and more frequent releases. It also promotes best practices in deployment and scaling, helping to ensure that your applications are robust and performant.\n\nFor more information, including detailed documentation, tutorials, and community support, visit [okik.ai](www.okik.ai).\n\n## Installation\nUsing pip\n\n```bash\npip install okik\n```\n\nThis is the simplest method to install Okik. It will automatically handle all dependencies and install the latest stable version of Okik.\n\nOr\nTo install Okik from source, follow these steps:\n\n1. Clone the repository: `git clone https://github.com/okikorg/okik.git`\n2. Navigate to the project directory: `cd okik`\n3. Install Okik using pip: `pip install .`\n\nThis method is useful if you want to contribute to Okik's development or need the latest features that haven't been released in the stable version yet.\n\n## Quick Start\n\nTo run Okik, simply execute the following command in your terminal:\n`okik`\n```\n██████  ██   ██ ██ ██   ██\n██    ██ ██  ██  ██ ██  ██\n██    ██ █████   ██ █████\n██    ██ ██  ██  ██ ██  ██\n██████  ██   ██ ██ ██   ██\n\n\n\nSimplify. Deploy. Scale.\nType 'okik --help' for more commands.\n```\n\nThis command launches the Okik CLI, providing you with an overview of available commands and options.\n\n## Initialise the project\n```bash\nokik init\n```\n\nThis command sets up a new Okik project in your current directory. It creates necessary configuration files and project structure, preparing your environment for Okik-managed deployments.\n\n## Quick Example\nWrite this in your `main.py` file:\n\n```python\nfrom okik.endpoints import service, endpoint, app\nimport asyncio\nfrom typing import Any\nfrom sentence_transformers import SentenceTransformer\nimport sentence_transformers\nfrom torch.nn.functional import cosine_similarity as cosine\nimport torch\nimport random\n\n# your service configuration\n@service(\n    replicas=1,\n    resources={\"accelerator\": {\"type\": \"A40\", \"device\": \"cuda\", \"count\": 1, \"memory\": 4}},\n    backend=\"okik\" # \u003c- provisioning backend is okik\n)\nclass Embedder:\n    def __init__(self):\n        self.model = SentenceTransformer(\"paraphrase-MiniLM-L6-v2\", cache_folder=\".okik/cache\")\n\n    @endpoint()\n    def embed(self, sentence: str):\n        logits = self.model.encode(sentence)\n        return logits\n\n    @endpoint()\n    def similarity(self, sentence1: str, sentence2: str):\n        logits1 = self.model.encode(sentence1, convert_to_tensor=True)\n        logits2 = self.model.encode(sentence2, convert_to_tensor=True)\n        return cosine(logits1.unsqueeze(0), logits2.unsqueeze(0))\n\n    @endpoint()\n    def version(self):\n        return sentence_transformers.__version__\n\n    @endpoint(stream=True)\n    async def stream_data(self) -\u003e Any:\n        async def data_generator():\n            for i in range(10):\n                yield f\"data: {i}\\n\"\n                await asyncio.sleep(1)\n        return data_generator()\n\n# Mock LLM Service Example\n@service(replicas=1)\nclass MockLLM:\n    def __init__(self):\n        pass\n\n    @endpoint(stream=True) # \u003c- streaming response enabled for use cases like chatbot\n    async def stream_random_words(self, prompt: str = \"Hello\"):\n        async def word_generator():\n            words = [\"hello\", \"world\", \"fastapi\", \"stream\", \"test\", \"random\", \"words\", \"python\", \"async\", \"response\"]\n            for _ in range(10):\n                word = random.choice(words)\n                yield f\"{word}\\n\"\n                await asyncio.sleep(0.4)\n        return word_generator()\n\n```\n\nThis example demonstrates how to create services and endpoints using Okik. The `@service` decorator defines a service with specific configuration, while `@endpoint` decorators define individual API endpoints within the service. The example includes both synchronous and asynchronous endpoints, as well as streaming capabilities.\n\n## Verify the routes\n```bash\n# run the okik routes to check all available routes\nokik routes\n```\n```bash\n# output should be similar to this\nmain.py Application Routes\n├── \u003cHOST\u003e/health/\n│   └── /health | GET\n├── \u003cHOST\u003e/embedder/\n│   ├── /embedder/embed | POST\n│   ├── /embedder/similarity | POST\n│   ├── /embedder/stream_data | POST\n│   └── /embedder/version | POST\n└── \u003cHOST\u003e/mockllm/\n    └── /mockllm/stream_random_words | POST\n```\n\nThis command displays all the routes defined in your application, helping you verify that your endpoints are correctly set up.\n\n## Serving the app\n```bash\n# run the okik run to start the server in production mode\nokik server\n# or run in dev mode\nokik server --dev --reload\n#or\nokik server -d -r\n```\n\nThese commands start your Okik server. The `--dev` and `--reload` flags are useful during development as they enable auto-reloading when code changes are detected.\n\n## Test the app\n```bash\ncurl -X POST http://0.0.0.0:3000/embedder/version\n# or if you like to use httpie then\nhttp POST 0.0.0.0:3000/embedder/version\n\n# or test the stream endpoint\ncurl -X POST http://0.0.0.0:3000/mockllm/stream_random_words -d '{\"prompt\": \"Hello\"}'\n# or if you like to use httpie then\nhttp POST 0.0.0.0:3000/mockllm/stream_random_words prompt=\"hello\" --stream\n```\n\nThese commands demonstrate how to test your Okik endpoints using curl or httpie. They show both regular POST requests and how to handle streaming responses.\n\n## Build the app\n```bash\nokik build -a \"your_awesome_app\" -t latest\n```\n\nThis command builds your application, preparing it for deployment. The `-a` flag specifies the application name, and `-t` sets the tag for the build.\n\n## Deploy the app\n```bash\nokik deploy\n```\n\nThis command deploys your built application to the configured cloud provider.\n\n## Monitor the app\n```bash\n# similar to kubectl commands, infact you can use kubectl commands as well\nokik get deployments # for deployments\nokik get services # for services\n```\n\nThese commands allow you to monitor your deployed applications and services, providing information about their status and configuration.\n\n## Delete the app\n```bash\nokik delete deployment \"your_awesome_app\"\n```\n\nThis command removes a deployed application from your cloud environment.\n\n## Status\nOkik is currently in development so expect sharp edges and bugs. Feel free to contribute to the project by submitting a pull request. Your feedback and contributions are valuable in improving and stabilizing Okik for the wider development community.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokikorg%2Fokik","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fokikorg%2Fokik","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fokikorg%2Fokik/lists"}