{"id":13642371,"url":"https://github.com/aniketmaurya/llm-inference","last_synced_at":"2025-10-01T02:31:35.422Z","repository":{"id":153256510,"uuid":"625900842","full_name":"aniketmaurya/llm-inference","owner":"aniketmaurya","description":"Large Language Model (LLM) Inference API and Chatbot","archived":true,"fork":false,"pushed_at":"2024-04-02T15:11:22.000Z","size":3496,"stargazers_count":123,"open_issues_count":0,"forks_count":14,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-09-23T11:33:24.088Z","etag":null,"topics":["chatbot","langchain","llama","llm","llm-inference","mistral"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/llm-inference/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aniketmaurya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-10T11:06:26.000Z","updated_at":"2024-08-08T06:05:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"f79f2f0c-3a41-46ae-a67e-abdaa373962f","html_url":"https://github.com/aniketmaurya/llm-inference","commit_stats":{"total_commits":51,"total_committers":4,"mean_commits":12.75,"dds":0.2941176470588235,"last_synced_commit":"5bb323c4cce70dcbe81cf794aaa0a66b87fe3083"},"previous_names":["aniketmaurya/llm-inference-api","aniketmaurya/llm-inference","aniketmaurya/llama-inference-api"],"tags_count":10,"template":false,"template_full_name":"aniketmaurya/python-project-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aniketmaurya%2Fllm-inference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aniketmaurya%2Fllm-inference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aniketmaurya%2Fllm-inference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aniketmaurya%2Fllm-inference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aniketmaurya","download_url":"https://codeload.github.com/aniketmaurya/llm-inference/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219875269,"owners_count":16554661,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","langchain","llama","llm","llm-inference","mistral"],"created_at":"2024-08-02T01:01:30.569Z","updated_at":"2025-10-01T02:31:30.074Z","avatar_url":"https://github.com/aniketmaurya.png","language":"Python","funding_links":[],"categories":["Python","Chatbots"],"sub_categories":[],"readme":"# Large Language Model (LLM) Inference API and Chatbot 🦙\n\n![project banner](https://github.com/aniketmaurya/llm-inference/raw/main/assets/llm-inference-min.png)\n\nInference API for LLMs like LLaMA and Falcon powered by Lit-GPT from [Lightning AI](https://lightning.ai)\n\n```\npip install llm-inference\n```\n\n### Install from main branch\n```bash\npip install git+https://github.com/aniketmaurya/llm-inference.git@main\n\n# You need to manually install [Lit-GPT](https://github.com/Lightning-AI/lit-gpt) and setup the model weights to use this project.\npip install lit_gpt@git+https://github.com/aniketmaurya/install-lit-gpt.git@install\n```\n\n## For Inference\n\n```python\nfrom llm_inference import LLMInference, prepare_weights\n\npath = prepare_weights(\"EleutherAI/pythia-70m\")\nmodel = LLMInference(checkpoint_dir=path)\n\nprint(model(\"New York is located in\"))\n```\n\n\n## How to use the Chatbot\n\n![chatbot image](./assets/llm-inference-llama2_chatbot.png)\n\n\n```python\nfrom llm_chain import LitGPTConversationChain, LitGPTLLM\nfrom llm_inference import prepare_weights\n\npath = str(prepare_weights(\"meta-llama/Llama-2-7b-chat-hf\"))\nllm = LitGPTLLM(checkpoint_dir=path, quantize=\"bnb.nf4\")  # 7GB GPU memory\nbot = LitGPTConversationChain.from_llm(llm=llm, prompt=llama2_prompt_template)\n\nprint(bot.send(\"hi, what is the capital of France?\"))\n```\n\n## Launch Chatbot App\n\n\u003cvideo width=\"320\" height=\"240\" controls\u003e\n  \u003csource src=\"/assets/chatbot-demo.mp4\" type=\"video/mp4\"\u003e\n\u003c/video\u003e\n\n**1. Download weights**\n```py\nfrom llm_inference import prepare_weights\npath = prepare_weights(\"meta-llama/Llama-2-7b-chat-hf\")\n```\n\n**2. Launch Gradio App**\n\n```\npython examples/chatbot/gradio_demo.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faniketmaurya%2Fllm-inference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faniketmaurya%2Fllm-inference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faniketmaurya%2Fllm-inference/lists"}