{"id":19509046,"url":"https://github.com/ooridata/ogbujipt","last_synced_at":"2025-04-04T11:07:48.943Z","repository":{"id":172609134,"uuid":"649448000","full_name":"OoriData/OgbujiPT","owner":"OoriData","description":"Client-side toolkit for using large language models, including where self-hosted","archived":false,"fork":false,"pushed_at":"2024-11-20T21:57:52.000Z","size":1310,"stargazers_count":107,"open_issues_count":10,"forks_count":6,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-28T10:06:50.056Z","etag":null,"topics":["ai","gpt","langchain","large-language-models","llm","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OoriData.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-04T21:39:24.000Z","updated_at":"2025-03-22T00:03:13.000Z","dependencies_parsed_at":"2023-10-10T22:15:11.053Z","dependency_job_id":"87b0d3dc-697b-49cf-854f-1cff0aa3c622","html_url":"https://github.com/OoriData/OgbujiPT","commit_stats":{"total_commits":492,"total_committers":8,"mean_commits":61.5,"dds":0.5142276422764227,"last_synced_commit":"1670817cd31fa3325f8859e53413d8d511f1171d"},"previous_names":["uogbuji/ogbujipt","ooridata/ogbujipt"],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OoriData%2FOgbujiPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OoriData%2FOgbujiPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OoriData%2FOgbujiPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OoriData%2FOgbujiPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OoriData","download_url":"https://codeload.github.com/OoriData/OgbujiPT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247166166,"owners_count":20894654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","gpt","langchain","large-language-models","llm","python"],"created_at":"2024-11-10T23:10:46.875Z","updated_at":"2025-04-04T11:07:48.911Z","avatar_url":"https://github.com/OoriData.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![ogbujipt github header](https://github.com/OoriData/OgbujiPT/assets/43561307/1a88b411-1ce2-43df-83f0-c9c39d6679bc)\n\n\nToolkit for using self-hosted large language models (LLMs), but also with support for full-service such as OpenAI's GPT models.\n\nIncludes demos with RAG (\"chat your documents\") and AGI/AutoGPT/privateGPT-style capabilities, via streamlit, Discord, command line, etc.\n\nThere are some helper functions for common LLM tasks, such as those provided by projects such as langchain, but not meant to be as extensive. The OgbujiPT approach emphasizes simplicity and transparency.\n\nTested back ends are [llama.cpp](https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md) (custom HTTP API), [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)  (OpenAI HTTP API), [text-generation-webui](https://github.com/oobabooga/text-generation-webui) (AKA Oobabooga or Ooba) and in-memory hosted LLaMA-class (and more) models via [ctransformers](https://github.com/marella/ctransformers). In our own practice we apply these with Nvidia and Apple M1/M2 GPU enabled.\n\nWe also test with OpenAI's full service GPT (3, 3.5, and 4) APIs, and apply these in our practice.\n\n\u003ctable\u003e\u003ctr\u003e\n  \u003ctd\u003e\u003ca href=\"https://oori.dev/\"\u003e\u003cimg src=\"https://www.oori.dev/assets/branding/oori_Logo_FullColor.png\" width=\"64\" /\u003e\u003c/a\u003e\u003c/td\u003e\n  \u003ctd\u003eOgbujiPT is primarily developed by the crew at \u003ca href=\"https://oori.dev/\"\u003eOori Data\u003c/a\u003e. We offer data pipelines and software engineering services around AI/LLM applications.\u003c/td\u003e\n\u003c/tr\u003e\u003c/table\u003e\n\n[![PyPI - Version](https://img.shields.io/pypi/v/ogbujipt.svg)](https://pypi.org/project/ogbujipt)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/ogbujipt.svg)](https://pypi.org/project/ogbujipt)\n\n## Quick links\n\n- [Getting started](#getting-started)\n- [License](#license)\n\n-----\n\n## Getting started\n\n```console\npip install ogbujipt\n```\n\n### Just show me some code, dammit!\n\n```py\nfrom ogbujipt.llm_wrapper import openai_chat_api, prompt_to_chat\n\nllm_api = openai_chat_api(base_url='http://localhost:8000')  # Update for your LLM API host\nprompt = 'Write a short birthday greeting for my star employee'\n\n# You can set model params as needed\nresp = llm_api.call(prompt_to_chat(prompt), temperature=0.1, max_tokens=256)\n# Extract just the response text, but the entire structure is available\nprint(resp.first_choice_text)\n```\n\nThe [Nous-Hermes 13B](https://huggingface.co/TheBloke/Nous-Hermes-13B-GGML) LLM offered the following response:\n\n\u003e Dear [Employee's Name],\n\u003e I hope this message finds you well on your special day! I wanted to take a moment to wish you a very happy birthday and express how much your contributions have meant to our team. Your dedication, hard work, and exceptional talent have been an inspiration to us all.\n\u003e On this occasion, I want you to know that you are appreciated and valued beyond measure. May your day be filled with joy and laughter.\n\n### Asynchronous by design\n\nAbove example shows the synchronous API, provided for dumb convenience, but for most use cases you'll want to use the asynchronous API. This example also adds a system message.\n\n```py\nimport asyncio\nfrom ogbujipt.llm_wrapper import openai_chat_api, prompt_to_chat\n\nllm_api = openai_chat_api(base_url='http://localhost:8000')  # Update for your LLM API host\nprompt = 'Write a short birthday greeting for my star employee'\n\nmessages = prompt_to_chat(prompt, system='You are a helpful AI agent…')\nresp = await asyncio.run(llm_api(messages, temperature=0.1, max_tokens=256))\n# Extract just the response text, but the entire structure is available\nprint(resp.first_choice_text)\n```\n\n### llama.cpp HTTP API for flexible LLM control\n\nHere's an example using a model hosted directly by [llama.cpp's server](https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md).\n\n```py\nimport asyncio\nfrom ogbujipt.llm_wrapper import prompt_to_chat, llama_cpp_http_chat\n\nllm_api = llama_cpp_http_chat('http://localhost:8000')\nresp = asyncio.run(llm_api(prompt_to_chat('Knock knock!'), min_p=0.05))\nprint(resp.first_choice_text)\n```\n\n### ctransformers for local in-process loaded LLMs\n\nHere's an example using a model loaded in-process using ctransformers.\n\n```py\nfrom ctransformers import AutoModelForCausalLM\n\nfrom ogbujipt.llm_wrapper import ctransformer as ctrans_wrapper\n\nmodel = AutoModelForCausalLM.from_pretrained('TheBloke_LlongOrca-13B-16K-GGUF',\n        model_file='llongorca-13b-16k.Q5_K_M.gguf', model_type=\"llama\", gpu_layers=50)\nllm = ctrans_wrapper(model=model)\n\nprint(llm(prompt='Write a short birthday greeting for my star employee', max_new_tokens=100))\n```\n\n### For more examples…\n\nSee the [demo directory](https://github.com/uogbuji/OgbujiPT/tree/main/demo). Demos include:\n\n* Basics:\n  * Use of basic LLM text completion to correct a data format (XML)\n  * Multiple simultaneous LLM queries via multiprocessing\n* Chatbots/agents:\n  * Simple Discord bot\n* Advanced LLM API features:\n  * OpenAI-style function calling\n* Retrieval Augmented Generation (RAG):\n  * Ask LLM questions based on web site contents, on the command line\n  * Ask LLM questions based on uploaded PDF, via Streamlit interactive UI\n  * Use PostgreSQL/PGVector for extracting context which can be fed to LLMs\n\n## A bit more explanation\n\nMany self-hosted AI large language models are now astonishingly good, even running on consumer-grade hardware, which provides an alternative for those of us who would rather not be sending all our data out over the network to the likes of ChatGPT \u0026 Bard. OgbujiPT provides a toolkit for using and experimenting with LLMs as loaded into memory via or via OpenAI API-compatible network servers such as:\n\n* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)\n* [text-generation-webui](https://github.com/oobabooga/text-generation-webui) (AKA Oobabooga or Ooba)\n\nOgbujiPT can invoke these to complete prompted tasks on self-hosted LLMs. It can also be used for building front ends to ChatGPT and Bard, if these are suitable for you.\n\n* [Quick setup for llama-cpp-python](https://github.com/uogbuji/OgbujiPT/wiki/Quick-setup-for-llama-cpp-python-backend)\n* [Quick setup for Ooba](https://github.com/uogbuji/OgbujiPT/wiki/Quick-setup-for-text-generation-webui-(Ooba)-backend)\n\nRight now OgbujiPT requires a bit of Python development on the user's part, but more general capabilities are coming.\n\n## Bias to sound software engineering\n\nI've seen many projects taking stabs at something like this one, but they really just seem to be stabs, usually by folks interested in LLM who admit they don't have strong coding backgrounds. This not only leads to a lumpy patchwork of forks and variations, as people try to figure out the narrow, gnarly paths that cater to their own needs, but also hampers maintainability just at a time when everything seems to be changing drastically every few days.\n\nI have a strong Python and software engineering background, and I'm looking to apply that in this project, to hopefully create something more easily speclailized for other needs, built-upon, maintained and contributed to.\n\nThis project is packaged using [hatch](https://hatch.pypa.io/), a modern Python packaging tool. I plan to write tests as I go along, and to incorporate continuous integration. Admit I may be slow to find the cycles for all that, but at least the intent and architecture is there from the beginning.\n\n## Prompting patterns\n\nDifferent LLMs have different conventions you want to use in order to get high\nquality responses. If you've looked into [self-hosted LLMs](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) you might have heard\nof the likes of alpaca, vicuña or even airoboros. OgbujiPT includes some shallow\ntools in order to help construct prompts according to the particular conventions\nthat would be best for your choice of LLM. This makes it easier to quickly launch\nexperiments, adapt to and adopt other models.\n\n# Contributions\n\nIf you want to run the test suite, a quick recipe is as follows:\n\n```shell\npip install ruff pytest pytest-mock pytest-asyncio respx pgvector asyncpg pytest-asyncio\npytest test\n```\n\nIf you want to make contributions to the project, please [read these notes](https://github.com/OoriData/OgbujiPT/wiki/Notes-for-contributors).\n\n# Resources\n\n* [Against mixing environment setup with code](https://huggingface.co/blog/ucheog/separate-env-setup-from-code)\n\n# License\n\nApache 2. For tha culture!\n\n# Credits\n\nSome initial ideas \u0026 code were borrowed from these projects, but with heavy refactoring:\n\n* [ChobPT/oobaboogas-webui-langchain_agent](https://github.com/ChobPT/oobaboogas-webui-langchain_agent)\n* [wafflecomposite/langchain-ask-pdf-local](https://github.com/wafflecomposite/langchain-ask-pdf-local)\n\n# Related projects\n\n* [mlx-tuning-fork\n](https://github.com/chimezie/mlx-tuning-fork)—\"very basic framework for parameterized Large Language Model (Q)LoRa fine-tuning with MLX. It uses mlx, mlx_lm, and OgbujiPT, and is based primarily on the excellent mlx-example libraries but adds very minimal architecture for systematic running of easily parameterized fine tunes, hyperparameter sweeping, declarative prompt construction, an equivalent of HF's train on completions, and other capabilities.\"\n* [living-bookmarks](https://github.com/uogbuji/living-bookmarks)—\"Uses [OgbujiPT] to Help a user manage their bookmarks in context of various chat, etc.\"\n\n# FAQ\n\n- [What's unique about this toolkit?](#whats-unique-about-this-toolkit)\n- [Does this support GPU for locally-hosted models](#does-this-support-gpu-for-locally-hosted-models)\n- [What's with the crazy name?](#whats-with-the-crazy-name)\n\n## What's unique about this toolkit?\n\nI mentioned the bias to software engineering, but what does this mean?\n\n* Emphasis on modularity, but seeking as much consistency as possible\n* Support for multitasking\n* Finding ways to apply automated testing\n\n## Does this support GPU for locally-hosted models\n\nYes, but you have to make sure you set up your back end LLM server (llama.cpp or text-generation-webui) with GPU, and properly configure the model you load into it.\n\nMany install guides I've found for Mac, Linux and Windows touch on enabling GPU, but the ecosystem is still in its early days, and helpful resouces can feel scattered.\n\n* [Quick setup for llama-cpp-python](https://github.com/uogbuji/OgbujiPT/wiki/Quick-setup-for-llama-cpp-python-backend)\n* [Quick setup for Ooba](https://github.com/uogbuji/OgbujiPT/wiki/Quick-setup-for-text-generation-webui-(Ooba)-backend)\n\n## What's with the crazy name?\n\nEnh?! Yo mama! 😝 My surname is Ogbuji, so it's a bit of a pun.\nThis is the notorious OGPT, ya feel me?\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fooridata%2Fogbujipt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fooridata%2Fogbujipt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fooridata%2Fogbujipt/lists"}