{"id":19815931,"url":"https://github.com/replicate/cog-vllm","last_synced_at":"2025-05-01T10:31:51.320Z","repository":{"id":242741465,"uuid":"668894465","full_name":"replicate/cog-vllm","owner":"replicate","description":"Run LLMs on Replicate with vLLM","archived":false,"fork":false,"pushed_at":"2024-10-11T18:15:38.000Z","size":190,"stargazers_count":18,"open_issues_count":1,"forks_count":4,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-05-01T04:46:38.417Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/replicate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-20T21:09:07.000Z","updated_at":"2025-04-13T05:14:30.000Z","dependencies_parsed_at":"2024-06-20T23:42:26.494Z","dependency_job_id":"1149ac37-613b-415a-ab26-7b6b8190577e","html_url":"https://github.com/replicate/cog-vllm","commit_stats":null,"previous_names":["replicate/cog-vllm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-vllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-vllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-vllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-vllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/replicate","download_url":"https://codeload.github.com/replicate/cog-vllm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251860011,"owners_count":21655658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T10:07:46.863Z","updated_at":"2025-05-01T10:31:50.945Z","avatar_url":"https://github.com/replicate.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cog-vLLM: Run vLLM on Replicate\n\n[Cog](https://github.com/replicate/cog) \nis an open-source tool that lets you package machine learning models\nin a standard, production-ready container. \nvLLM is a fast and easy-to-use library for LLM inference and serving.\n\nYou can deploy your packaged model to your own infrastructure, \nor to [Replicate].\n\n## Highlights\n\n* 🚀 **Run vLLM in the cloud with an API**.\n  Deploy any [vLLM-supported language model] at scale on Replicate.\n\n* 🏭 **Support multiple concurrent requests**.\n  Continuous batching works out of the box.\n\n* 🐢 **Open Source, all the way down**.\n  Look inside, take it apart, make it do exactly what you need.\n\n## Quickstart\n\nGo to [replicate.com/replicate/vllm](https://replicate.com/replicate/vllm)\nand create a new vLLM model from a [supported Hugging Face repo][vLLM-supported language model],\nsuch as [google/gemma-2b](https://huggingface.co/google/gemma-2b)\n\n\u003e [!IMPORTANT]  \n\u003e Gated models require a [Hugging Face API token](https://huggingface.co/settings/tokens),\n\u003e which you can set in the `hf_token` field of the model creation form.\n\n\u003cimg width=\"1055\" alt=\"Create a new vLLM model on Replicate\" src=\"https://github.com/replicate/cog-vllm/assets/7659/a8f31837-0ed3-40f7-974c-d0a16ae48350\"\u003e\n\nReplicate downloads the model files, packages them into a `.tar` archive,\nand pushes a new version of your model that's ready to use.\n\n\u003cimg width=\"1322\" alt=\"Trained vLLM model on Replicate\" src=\"https://github.com/replicate/cog-vllm/assets/7659/ebb84e12-9173-4fb0-8749-7293a105cf13\"\u003e\n\nFrom here, you can either use your model as-is,\nor customize it and push up your changes.\n\n## Local Development\n\nIf you're on a machine or VM with a GPU,\nyou can try out changes before pushing them to Replicate.\n\nStart by [installing or upgrading Cog](https://cog.run/#install).\nYou'll need Cog [v0.10.0-alpha11](https://github.com/replicate/cog/releases/tag/v0.10.0-alpha11):\n\n```console\n$ sudo curl -o /usr/local/bin/cog -L \"https://github.com/replicate/cog/releases/download/v0.10.0-alpha11/cog_$(uname -s)_$(uname -m)\"\n$ sudo chmod +x /usr/local/bin/cog\n```\n\nThen clone this repository:\n\n```console\n$ git clone https://github.com/replicate/cog-vllm\n$ cd cog-vllm\n```\n\nGo to the [Replicate dashboard](https://replicate.com/trainings) and \nnavigate to the training for your vLLM model.\nFrom that page, copy the weights URL from the \u003ckbd\u003eDownload weights\u003c/kbd\u003e button.\n\n\u003cimg width=\"642\" alt=\"Copy weights URL from Replicate training\" src=\"https://github.com/replicate/cog-vllm/assets/7659/97c403a9-ec49-418a-a7e2-b37cb0e0bb8c\"\u003e\n\nSet the `COG_WEIGHTS` environment variable with that copied value: \n\n```console\n$ export COG_WEIGHTS=\"...\"\n```\n\nNow, make your first prediction against the model locally:\n\n```console\n$ cog predict -e \"COG_WEIGHTS=$COG_WEIGHTS\" \\ \n              -i prompt=\"Hello!\"\n```\n\nThe first time you run this command,\nCog downloads the model weights and save them to the `models` subdirectory.\n\nTo make multiple predictions,\nstart up the HTTP server and send it `POST /predictions` requests.\n\n```console\n# Start the HTTP server\n$ cog run -p 5000 -e \"COG_WEIGHTS=$COG_WEIGHTS\" python -m cog.server.http\n\n# In a different terminal session, send requests to the server\n$ curl http://localhost:5000/predictions -X POST \\\n    -H 'Content-Type: application/json' \\\n    -d '{\"input\": {\"prompt\": \"Hello!\"}}'\n```\n\nWhen you're finished working,\nyou can push your changes to Replicate.\n\nGrab your token from [replicate.com/account](https://replicate.com/account) \nand set it as an environment variable:\n\n```shell\nexport REPLICATE_API_TOKEN=\u003cyour token\u003e\n```\n\n```console\n$ echo $REPLICATE_API_TOKEN | cog login --token-stdin\n$ cog push r8.im/\u003cyour-username\u003e/\u003cyour-model-name\u003e\n--\u003e ...\n--\u003e Pushing image 'r8.im/...'\n```\n\nAfter you push your model, you can try running it on Replicate.\n\nInstall the [Replicate Python SDK][replicate-python]:\n\n```console\n$ pip install replicate\n```\n\nCreate a prediction and stream its output:\n\n```python\nimport replicate\n\nmodel = replicate.models.get(\"\u003cyour-username\u003e/\u003cyour-model-name\u003e\")\nprediction = replicate.predictions.create(\n    version=model.latest_version,\n    input={ \"prompt\": \"Hello\" },\n    stream=True\n)\n\nfor event in prediction.stream():\n    print(str(event), end=\"\")\n```\n\n[Replicate]: https://replicate.com\n[vLLM-supported language model]: https://docs.vllm.ai/en/latest/models/supported_models.html\n[replicate-python]: https://github.com/replicate/replicate-python\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplicate%2Fcog-vllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freplicate%2Fcog-vllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplicate%2Fcog-vllm/lists"}