{"id":13579597,"url":"https://github.com/jncraton/languagemodels","last_synced_at":"2025-05-14T05:10:37.950Z","repository":{"id":162748290,"uuid":"637258008","full_name":"jncraton/languagemodels","owner":"jncraton","description":"Explore large language models in 512MB of RAM","archived":false,"fork":false,"pushed_at":"2025-02-28T19:06:18.000Z","size":409,"stargazers_count":1189,"open_issues_count":2,"forks_count":81,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-11T00:04:00.940Z","etag":null,"topics":["llm","nlp","python"],"latest_commit_sha":null,"homepage":"https://jncraton.github.io/languagemodels/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jncraton.png","metadata":{"files":{"readme":"readme.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"license.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-07T01:53:20.000Z","updated_at":"2025-04-04T22:41:04.000Z","dependencies_parsed_at":"2023-09-27T07:14:32.704Z","dependency_job_id":"7612641b-2079-4062-ab7f-f30362e0ff0e","html_url":"https://github.com/jncraton/languagemodels","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jncraton%2Flanguagemodels","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jncraton%2Flanguagemodels/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jncraton%2Flanguagemodels/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jncraton%2Flanguagemodels/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jncraton","download_url":"https://codeload.github.com/jncraton/languagemodels/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254076849,"owners_count":22010611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","nlp","python"],"created_at":"2024-08-01T15:01:40.948Z","updated_at":"2025-05-14T05:10:37.886Z","avatar_url":"https://github.com/jncraton.png","language":"HTML","funding_links":[],"categories":["HTML","Building"],"sub_categories":["LLM Models"],"readme":"Language Models\n===============\n\n[![PyPI version](https://badge.fury.io/py/languagemodels.svg)](https://badge.fury.io/py/languagemodels)\n[![docs](https://img.shields.io/badge/docs-online-brightgreen)](https://languagemodels.netlify.app/)\n[![x64 Build](https://github.com/jncraton/languagemodels/actions/workflows/build.yml/badge.svg)](https://github.com/jncraton/languagemodels/actions/workflows/build.yml)\n[![ARM64 Build](https://github.com/jncraton/languagemodels/actions/workflows/pi.yml/badge.svg)](https://github.com/jncraton/languagemodels/actions/workflows/pi.yml)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jncraton/languagemodels/blob/master/examples/translate.ipynb)\n\nPython building blocks to explore large language models in as little as 512MB of RAM\n\n![Translation hello world example](media/hello.gif)\n\nThis package makes using large language models from Python as simple as possible. All inference is performed locally to keep your data private by default.\n\nInstallation and Getting Started\n--------------------------------\n\nThis package can be installed using the following command:\n\n```sh\npip install languagemodels\n```\n\nOnce installed, you should be able to interact with the package in Python as follows:\n\n```python\n\u003e\u003e\u003e import languagemodels as lm\n\u003e\u003e\u003e lm.do(\"What color is the sky?\")\n'The color of the sky is blue.'\n```\n\nThis will require downloading a significant amount of data (~250MB) on the first run. Models will be cached for later use and subsequent calls should be quick.\n\nExample Usage\n-------------\n\nHere are some usage examples as Python REPL sessions. This should work in the REPL, notebooks, or in traditional scripts and applications.\n\n### Instruction Following\n\n```python\n\u003e\u003e\u003e import languagemodels as lm\n\n\u003e\u003e\u003e lm.do(\"Translate to English: Hola, mundo!\")\n'Hello, world!'\n\n\u003e\u003e\u003e lm.do(\"What is the capital of France?\")\n'Paris.'\n```\n\nOutputs can be restricted to a list of choices if desired:\n\n```python\n\u003e\u003e\u003e lm.do(\"Is Mars larger than Saturn?\", choices=[\"Yes\", \"No\"])\n'No'\n```\n\n### Adjusting Model Performance\n\nThe base model should run quickly on any system with 512MB of memory, but this memory limit can be increased to select more powerful models that will consume more resources. Here's an example:\n\n```python\n\u003e\u003e\u003e import languagemodels as lm\n\u003e\u003e\u003e lm.do(\"If I have 7 apples then eat 5, how many apples do I have?\")\n'You have 8 apples.'\n\u003e\u003e\u003e lm.config[\"max_ram\"] = \"4gb\"\n4.0\n\u003e\u003e\u003e lm.do(\"If I have 7 apples then eat 5, how many apples do I have?\")\n'I have 2 apples left.'\n```\n\n### GPU Acceleration\n\nIf you have an NVIDIA GPU with CUDA available, you can opt in to using the GPU for inference:\n\n```python\n\u003e\u003e\u003e import languagemodels as lm\n\u003e\u003e\u003e lm.config[\"device\"] = \"auto\"\n```\n\n### Text Completions\n\n```python\n\u003e\u003e\u003e import languagemodels as lm\n\n\u003e\u003e\u003e lm.complete(\"She hid in her room until\")\n'she was sure she was safe'\n```\n\n### External Retrieval\n\nHelper functions are provided to retrieve text from external sources that can be used to augment prompt context.\n\n```python\n\u003e\u003e\u003e import languagemodels as lm\n\n\u003e\u003e\u003e lm.get_wiki('Chemistry')\n'Chemistry is the scientific study...\n\n\u003e\u003e\u003e lm.get_weather(41.8, -87.6)\n'Partly cloudy with a chance of rain...\n\n\u003e\u003e\u003e lm.get_date()\n'Friday, May 12, 2023 at 09:27AM'\n```\n\nHere's an example showing how this can be used (compare to previous chat example):\n\n```python\n\u003e\u003e\u003e lm.do(f\"It is {lm.get_date()}. What time is it?\")\n'The time is 12:53PM.'\n```\n\n### Semantic Search\n\nSemantic search is provided to retrieve documents that may provide helpful context from a document store.\n\n```python\n\u003e\u003e\u003e import languagemodels as lm\n\u003e\u003e\u003e lm.store_doc(lm.get_wiki(\"Python\"), \"Python\")\n\u003e\u003e\u003e lm.store_doc(lm.get_wiki(\"C language\"), \"C\")\n\u003e\u003e\u003e lm.store_doc(lm.get_wiki(\"Javascript\"), \"Javascript\")\n\u003e\u003e\u003e lm.get_doc_context(\"What does it mean for batteries to be included in a language?\")\n'From Python document: It is often described as a \"batteries included\" language due to its comprehensive standard library.Guido van Rossum began working on Python in the late 1980s as a successor to the ABC programming language and first released it in 1991 as Python 0.9.\n\nFrom C document: It was designed to be compiled to provide low-level access to memory and language constructs that map efficiently to machine instructions, all with minimal runtime support.'\n```\n\n[Full documentation](https://languagemodels.netlify.app/)\n\n### Speed\n\nThis package currently outperforms Hugging Face `transformers` for CPU inference thanks to int8 quantization and the [CTranslate2](https://github.com/OpenNMT/CTranslate2) backend. The following table compares CPU inference performance on identical models using the best available quantization on a 20 question test set.\n\n| Backend                   | Inference Time | Memory Used |\n|---------------------------|----------------|-------------|\n| Hugging Face transformers | 22s            | 1.77GB      |\n| This package              | 11s            | 0.34GB      |\n\nNote that quantization does technically harm output quality slightly, but it should be negligible at this level.\n\n### Models\n\nSensible default models are provided. The package should improve over time as stronger models become available. The basic models used are 1000x smaller than the largest models in use today. They are useful as learning tools, but perform far below the current state of the art.\n\nHere are the current default models used by the package for a supplied `max_ram` value:\n\n| max_ram | Model Name            | Parameters (B)\n| ------- | --------------------- | --------------\n| 0.5     | LaMini-Flan-T5-248M   | 0.248\n| 1.0     | LaMini-Flan-T5-783M   | 0.783\n| 2.0     | LaMini-Flan-T5-783M   | 0.783\n| 4.0     | flan-alpaca-gpt4-xl   | 3.0\n| 8.0     | openchat-3.5-0106     | 7.0\n\nFor code completions, the [CodeT5+](https://arxiv.org/abs/2305.07922) series of models are used.\n\nCommercial Use\n--------------\n\nThis package itself is licensed for commercial use, but the models used may not be compatible with commercial use. In order to use this package commercially, you can filter models by license type using the `require_model_license` function.\n\n```python\n\u003e\u003e\u003e import languagemodels as lm\n\u003e\u003e\u003e lm.config['instruct_model']\n'LaMini-Flan-T5-248M-ct2-int8'\n\u003e\u003e\u003e lm.require_model_license(\"apache|bsd|mit\")\n\u003e\u003e\u003e lm.config['instruct_model']\n'flan-t5-base-ct2-int8'\n```\n\nIt is recommended to confirm that the models used meet the licensing requirements for your software.\n\nProjects Ideas\n--------------\n\nOne of the goals for this package is to be a straightforward tool for learners and educators exploring how large language models intersect with modern software development. It can be used to do the heavy lifting for a number of learning projects:\n\n- CLI Chatbot (see [examples/chat.py](examples/chat.py))\n- Streamlit chatbot (see [examples/streamlitchat.py](examples/streamlitchat.py))\n- Chatbot with information retrieval\n- Chatbot with access to real-time information\n- Tool use\n- Text classification\n- Extractive question answering\n- Semantic search over documents\n- Document question answering\n\nSeveral example programs and notebooks are included in the `examples` directory.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjncraton%2Flanguagemodels","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjncraton%2Flanguagemodels","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjncraton%2Flanguagemodels/lists"}