{"id":16507593,"url":"https://github.com/chmp/lm-proxy","last_synced_at":"2026-05-27T06:30:20.097Z","repository":{"id":246759734,"uuid":"822703755","full_name":"chmp/lm-proxy","owner":"chmp","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-01T17:48:46.000Z","size":32,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-13T01:35:01.041Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chmp.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"License.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-01T16:39:41.000Z","updated_at":"2024-07-01T17:56:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"44404b00-1af9-4f7f-b2f3-c79a329c238a","html_url":"https://github.com/chmp/lm-proxy","commit_stats":null,"previous_names":["chmp/lm-proxy"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chmp%2Flm-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chmp%2Flm-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chmp%2Flm-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chmp%2Flm-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chmp","download_url":"https://codeload.github.com/chmp/lm-proxy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240149875,"owners_count":19755755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T15:29:13.772Z","updated_at":"2026-05-27T06:30:19.921Z","avatar_url":"https://github.com/chmp.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `lm-proxy` - (large) language model proxy\n\nA proxy for (large) language models that forwards to external servers. It manages external servers\nand spins them up and down on demand.\n\nConfig:\n\n```toml\n[proxy]\nport = 8080\n\n# without running requests, keep models alive for 60s\nkeep_alive = 60\n\n# with a running request, keep models alive for 300s\nrequest_keep_alive = 300\n\n[models.phi3]\nargs = [\n    \"llama-server\",\n    \"--model\",\n    \"phi-3-mini-4k-instruct-q4.gguf\",\n    \"--port\",\n    \"{{ port }}\",\n]\n\n[models.gemma2]\nargs = [\n    \"llama-server\",\n    \"--model\",\n    \"gemma-2-9b-it-q5_k_m.gguf\",\n    \"--port\",\n    \"{{ port }}\",\n]\n```\n\nStart the server:\n\n```bash\nlm-proxy serve config.toml\n```\n\nUse the server:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url = 'http://localhost:8080/v1',\n    api_key='unused',\n)\n\n# use the phi3 model\nresponse = client.chat.completions.create(\n  model=\"phi3\",\n  messages=[{\"role\": \"user\", \"content\": \"What is 2 + 3?\"}]\n)\nprint(response.choices[0].message.content)\n\n# use the gemma2 model\nresponse = client.chat.completions.create(\n  model=\"gemma2\",\n  messages=[{\"role\": \"user\", \"content\": \"How can I add 2 and 3 in Python?\"}]\n)\nprint(response.choices[0].message.content)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchmp%2Flm-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchmp%2Flm-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchmp%2Flm-proxy/lists"}