{"id":31809931,"url":"https://github.com/outerbounds/litellm-ob","last_synced_at":"2025-10-11T05:55:55.714Z","repository":{"id":310754199,"uuid":"1037666622","full_name":"outerbounds/litellm-ob","owner":"outerbounds","description":null,"archived":false,"fork":false,"pushed_at":"2025-08-20T01:43:05.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-20T03:33:27.576Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/outerbounds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-13T23:52:27.000Z","updated_at":"2025-08-13T23:56:56.000Z","dependencies_parsed_at":"2025-08-22T00:33:03.213Z","dependency_job_id":null,"html_url":"https://github.com/outerbounds/litellm-ob","commit_stats":null,"previous_names":["outerbounds/litellm-ob"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/outerbounds/litellm-ob","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Flitellm-ob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Flitellm-ob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Flitellm-ob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Flitellm-ob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/outerbounds","download_url":"https://codeload.github.com/outerbounds/litellm-ob/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Flitellm-ob/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279006356,"owners_count":26084088,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-11T05:55:54.397Z","updated_at":"2025-10-11T05:55:55.702Z","avatar_url":"https://github.com/outerbounds.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Running LiteLLM on Outerbounds\n\n[LiteLLM](https://docs.litellm.ai/docs/) has two components that can be used with Outerbounds.\n1. The [Python SDK](https://docs.litellm.ai/docs/#basic-usage) \n2. The [Proxy Server](https://docs.litellm.ai/docs/simple_proxy)\n\n## Use case 1: Batch inference\n\nThe eventual goal is to run this workflow, which is a minimal example from which you can build batch inference pipelines.\nFollow the two-step setup plan within this section before running this command.\n\n```bash\ncd batch\npython flow.py --environment=fast-bakery run --with kubernetes\n```\n\n### Step 1: Set up Outerbounds integrations for each LLM provider you want to leverage\n\nOne of the main functions of LiteLLM is to unify interfaces across LLM providers. \nProviders typically require API keys. For example, to use LiteLLM's OpenAI integration, \nyou need to set the `OPENAI_API_KEY` environment variable. Please read [here](https://docs.litellm.ai/docs/set_keys) for more details.\n\nAs a one time setup step on Outerbounds, you should visit the `Integrations` tab in Outerbounds UI, \nwhere you can register provider API keys as secrets by clicking `Add New Integration`.\nThis makes the secret keys and values accessible in a secure manner in your Outerbounds environments such as workstations, inference servers, and workflow tasks.\n\n### Step 2: Run a workflow using LiteLLM Python client\n\nIn `flow.py`, you will find a sample workflow that uses these components to call the LiteLLM client from a Metaflow task:\n1. Leverage the secret created in step 1 to authenticate to LLM providers\n2. Install LiteLLM using Metaflow's `@pypi` decorator\n\nRunning the flow will use LiteLLM from the runtime of the Metaflow task. \n\n## Use case 2: Real-time inference (LiteLLM proxy server)\n\n```bash\ncd realtime\nouterbounds app deploy --config-file app_config.yaml\n```\n\n### Step 1: Set up Outerbounds integration (or reuse from use case 1)\n\nSee Step 1 from the batch inference section, the same Outerbounds Integrations functionality is reused in the `realtime/app_config.yaml` specification.\nIf you already configured your integrations, you can simply reuse them. \n\n### Step 2: Deploy and endpoint for the LiteLLM proxy server as an Outerbounds Deployment\n\nAfter running the `outerbounds app deploy ...` command, you'll see output such as:\n\n```bash\n2025-08-13 15:45:04.520 🚀 Deploying litellm-proxy to the Outerbounds platform...\n2025-08-13 15:45:04.521 📦 Packaging directory : /path/to/working-dir/litellm-ob/realtime\n2025-08-13 15:45:04.522 🐳 Using the docker image : ghcr.io/berriai/litellm:main-latest\n2025-08-13 15:45:05.320 💾 Code package saved to : s3://obp-**-metaflow/metaflow/mf.obp-apps/**/**\n2025-08-13 15:45:05.451 🚀 Deploying endpoint to the platform....                                   \n2025-08-13 15:45:09.504 ⏳ 1 new worker(s) pending. Total pending (1)to serve traffic ⠏\n2025-08-13 15:45:31.294 🚀 1 worker(s) started running. Total running (1)erve traffic ⠏\n2025-08-13 15:45:31.294 ✅ First worker came online\n2025-08-13 15:45:31.294 🎉 All workers are now running\n2025-08-13 15:46:16.922 💊 Endpoint deployment status: completed ady to serve traffic ⠼\n2025-08-13 15:46:16.922 💊 Running last minute readiness check for **...\n2025-08-13 15:46:21.496 💊 Endpoint ** is ready to serve traffic on the URL: https://api-**.**.outerbounds.xyz\n2025-08-13 15:46:21.510 💊 Endpoint litellm-proxy (**) deployed! Endpoint available on the URL: https://api-**.**.outerbounds.xyz\n```\n\nThe key thing to extract is the URL, which contains an HTTP endpoint that you can send\nrequests to from any machine with your Outerbounds user credentials, \nor machine user credentials in CI environments,\nto authorize access when making requests to the LiteLLM endpoint.\nSelect this value and store in the environment where you want to make client-side requests from,\nusing the environment variable `LITELLM_PROXY_URL`. \n\n### Step 3: Access the endpoint\n\nNow you have a full-fledged LiteLLM server. \nYou can query it using CURL or from a Python script, anywhere where your Outerbounds user or a machine user is authenticated.\nAfter setting `LITELLM_PROXY_URL` from such an environment, you can run a client-side test with\n```bash\ncd realtime\npython client_sample.py\n```\n\n## Extensions \n\nMost LiteLLM integrations should extend naturally from these two interaction mode templates. \nIf you need custom support, or have general question, please reach out in your dedicate Outerbounds Slack channel.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Flitellm-ob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fouterbounds%2Flitellm-ob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Flitellm-ob/lists"}