{"id":18060510,"url":"https://github.com/florents-tselai/pgllm","last_synced_at":"2025-04-11T12:13:23.236Z","repository":{"id":212924371,"uuid":"732622054","full_name":"Florents-Tselai/pgllm","owner":"Florents-Tselai","description":"Use LLMs in Postgres","archived":false,"fork":false,"pushed_at":"2024-10-31T09:03:52.000Z","size":641,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-03T02:13:14.085Z","etag":null,"topics":["ai","embeddings","llm","postgresql","sql"],"latest_commit_sha":null,"homepage":"https://pgllm.tselai.com","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Florents-Tselai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["Florents-Tselai"]}},"created_at":"2023-12-17T09:54:21.000Z","updated_at":"2024-11-14T03:32:20.000Z","dependencies_parsed_at":"2024-08-18T11:56:42.936Z","dependency_job_id":null,"html_url":"https://github.com/Florents-Tselai/pgllm","commit_stats":null,"previous_names":["florents-tselai/pgllm"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Florents-Tselai%2Fpgllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Florents-Tselai%2Fpgllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Florents-Tselai%2Fpgllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Florents-Tselai%2Fpgllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Florents-Tselai","download_url":"https://codeload.github.com/Florents-Tselai/pgllm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248398804,"owners_count":21097294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","embeddings","llm","postgresql","sql"],"created_at":"2024-10-31T04:08:44.435Z","updated_at":"2025-04-11T12:13:23.209Z","avatar_url":"https://github.com/Florents-Tselai.png","language":"C","funding_links":["https://github.com/sponsors/Florents-Tselai","https://github.com/sponsors/Florents-Tselai/","https://img.shields.io/static/v1?label=Sponsor\u0026message=%E2%9D%A4\u0026logo=GitHub\u0026link=https://github.com/sponsors/Florents-Tselai/"],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n   \u003cimg width=\"50%\" height=\"40%\" src=\"https://raw.githubusercontent.com/Florents-Tselai/pgllm/main/docs/static/logo.webp\" alt=\"Logo\"\u003e\n  \u003c/p\u003e\n  \u003ch1 align=\"center\"\u003epgllm: Use LLMs in Postgres\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"#api\"\u003e\u003cstrong\u003e API \u003c/strong\u003e\u003c/a\u003e |\n    \u003ca href=\"#usage\"\u003e\u003cstrong\u003e Usage \u003c/strong\u003e\u003c/a\u003e |\n    \u003ca href=\"#llamafile\"\u003e\u003cstrong\u003e llamafile \u003c/strong\u003e\u003c/a\u003e |\n    \u003ca href=\"#embeddings\"\u003e\u003cstrong\u003e Embeddings \u003c/strong\u003e\u003c/a\u003e |\n    \u003ca href=\"#installation\"\u003e\u003cstrong\u003e Installation \u003c/strong\u003e\u003c/a\u003e\n    \n   \u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/Florents-Tselai/pgllm\"\u003e\u003cimg src=\"https://img.shields.io/badge/GitHub-repo-green\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/Florents-Tselai/pgllm/actions/workflows/build.yml?branch=mainline\"\u003e\u003cimg src=\"https://github.com/Florents-Tselai/pgllm/actions/workflows/build.yml/badge.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"http://pgllm.tselai.com/\"\u003e\u003cimg src=\"https://readthedocs.org/projects/pgllm/badge/?version=stable\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.linkedin.com/in/florentstselai/\"\u003e\u003cimg src=\"https://img.shields.io/badge/LinkedIn-0077B5?logo=linkedin\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/sponsors/Florents-Tselai/\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Sponsor\u0026message=%E2%9D%A4\u0026logo=GitHub\u0026link=https://github.com/sponsors/Florents-Tselai/\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n**pgllm** brings LLMs to Postgres.\n\nIt does so primarily by embedding CPython and wrapping the beautiful [llm](https://github.com/simonw/llm) Python library.\n\n## Features\n\n```tsql\nCREATE EXTENSION pgllm;\n```\n\nText generation with both local and remote models.\n\n```tsql\nSELECT llm_generate('hello world', 'markov', '{\"length\": 20, \"delay\": 0.2}');\nSELECT llm_generate('hello world', 'mistral', '{\"mistral\": \"abc0123\"}');\n```\nllamafile support\n\n```tsql\nSELECT llm_generate('A story about a frog', 'llamafile')\n```\n\nEmbedding models and pgvector support\n\n```tsql\nSELECT llm_embed('hello world', 'jina-embeddings-v2-small-en')::vector;\n```\n\nYou can use any [LLM plugin](https://llm.datasette.io/en/stable/plugins/index.html)\n\n## API\n\n* `llm_generate(input text, model text[, params jsonb]) → text`\n* `llm_embed(input text/bytea, model text[, params jsonb]) → float8[]`\n\n## Usage\n\n### Generation\n\nLet's start by installing a simple generational model\n\n```shell\npython3 -m llm install llm-markov\n```\n\n**IMPORTANT**: \nYou have to be sure that the `python3` you're using is the same one that you pointed to during the Installation;\nbetter be explicit.\n\n```sql\nselect llm_generate('hello world', 'markov');\n          llm_generate          \n--------------------------------\n world hello world world hello ....\n(1 row)\n```\n\n### Model Parameters\n\nCan be passed as a `jsonb` argument. \n\n```sql\nselect llm_generate('hello world', 'markov', '{\"length\": 20, \"delay\": 0.2}');\n                                                       llm_generate                                                       \n--------------------------------------------------------------------------------------------------------------------------\n world world hello world hello world hello world world hello world world world world world world world world world hello \n(1 row)\n```\n\n### Embeddings\n\nInstall a dummy embedding model\n\n```shell\npython3 -m llm install llm-embed-hazo\n```\n\n```sql\nselect llm_embed('hello world', 'hazo');\n\n             llm_embed             \n-----------------------------------\n{5,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0}\n(1 row)\n```\n\n#### pgvector\n\nIf you have **pgvector** already installed \nyou can cast the resulting `float8[]` to a `vector` type instead,\nand use pgvector as usual.\n\nFor example to get the L2 distance:\n\n```sql\nselect llm_embed('hello world', 'hazo')::vector \u003c-\u003e llm_embed('world hold on', 'hazo')::vector;\n     ?column?     \n------------------\n 2.23606797749979\n(1 row)\n```\n\n## Local Models\n\n### llamafile\n\npgllm supports llamafile by using curl to query its web API.\nThis does not use the `llm-llamafile` plugin!\n\nInstall with `WITH_LLAMAFILE=1` flag\n\n\u003cdetails\u003e\n\u003csummary\u003eStart llamafile server\u003c/summary\u003e\n\n1. Download [llava-v1.5-7b-q4.llamafile](https://huggingface.co/Mozilla/llava-v1.5-7b-llamafile/resolve/main/llava-v1.5-7b-q4.llamafile?download=true) (4.29 GB).\n\n2. Open your computer's terminal.\n\n3. If you're using macOS, Linux, or BSD, you'll need to grant permission\nfor your computer to execute this new file. (You only need to do this\nonce.)\n\n```sh\nchmod +x llava-v1.5-7b-q4.llamafile\n```\n\n4. If you're on Windows, rename the file by adding \".exe\" on the end.\n\n5. Run the llamafile. e.g.:\n\n```sh\n./llava-v1.5-7b-q4.llamafile\n```\n\n6. Your browser should open automatically and display a chat interface.\n(If it doesn't, just open your browser and point it at http://localhost:8080)\n\n7. When you're done chatting, return to your terminal and hit\n`Control-C` to shut down llamafile.\n\n\u003c/details\u003e\n\n```sql\nSELECT llm_generate('3 neat characteristics of a pelican', 'llamafile')::jsonb\n                                                                                                                                                                                                                                                                                                   llm_generate                                                                                                                                                                                                                                                                                                   \n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n {\"id\": \"chatcmpl-qLMCLi9ghpvobVxrwr83DOVVdvaMICef\", \"model\": \"LLaMA_CPP\", \"usage\": {\"total_tokens\": 132, \"prompt_tokens\": 58, \"completion_tokens\": 74}, \"object\": \"chat.completion\", \"choices\": [{\"index\": 0, \"message\": {\"role\": \"assistant\", \"content\": \"1. Pelicans have a large, broad beak that is adapted for catching fish.\\n2. They have a pouch under their beak, which they use to hold their catch.\\n3. Pelicans are known for their distinctive wading and fishing behavior, where they stand on one leg while waiting for fish to swim by.\u003c/s\u003e\"}, \"finish_reason\": \"stop\"}], \"created\": 1725269144}\n(1 row)\n```\n\n## Remote APIs\n\nLLM plugins for [remote APIs](https://llm.datasette.io/en/stable/plugins/directory.html#remote-apis) \nshould work easily.\n\nStart by installing the model plugin you want, for example:\n\n```shell\npython3 -m llm install llm-mistral\n```\n\nAnd then use you can pass the API_KEY as a model parameter.\n\n```shell\nselect llm_generate('hello world', 'mistral', '{\"mistral\": \"abc0123\"}');\n```\n\n**WARNING**:\nYou can easily exhaust any credits you may have by a simple `select` query.\nHence, use with caution!\n\n## Embeddings\n\n### JinaAI\n\n- [`jina-embeddings-v2-small-en`](https://huggingface.co/jinaai/jina-embeddings-v2-small-en): 33 million parameters.\n- [`jina-embeddings-v2-base-en`](https://huggingface.co/jinaai/jina-embeddings-v2-base-en): 137 million parameters.\n\n```shell\npython3 -m llm install llm-embed-jina\n```\n\n```sql\nselect llm_embed('hello world', 'jina-embeddings-v2-small-en');\n```\n\n### Onnx\n\n```\nonnx-bge-micro\nonnx-gte-tiny\nonnx-minilm-l6\nonnx-minilm-l12\nonnx-bge-small\nonnx-bge-base\nonnx-bge-large\n```\n\n```shell\npython3 -m llm install llm-embed-onnx\n```\n\n```sql\nselect llm_embed('hello world', 'onnx-bge-micro');\n```\n\n\n## Installation\n\nThe crucial thing in the installation process is to be sure which `python3` Postgres uses.\n\n```shell\ngit clone https://github.com/Florents-Tselai/pgllm.git\ncd pgllm\n\n# make sure that Python 3.XX minor versions match \nmake all PYTHON=/path/to/bin/python3.11 PYTHON_CONFIG=/path/to/python3.11-config\nmake install\nmake installcheck\n```\n\nThe host `postgrse` process must have access to the Python library too.\nYou can set LD_LIBRARY_PATH before starting postgres.\n\nFor example you may have to to something like:\n\n```shell\nexport LD_LIBRARY_PATH=\"$pythonLocation\"/lib:/usr/local/lib:/usr/lib:$HOME/local/lib:$LD_LIBRARY_PATH\n$PGBIN/initdb $PGDATA\n$PGBIN/pg_ctl --pgdata $PGDATA start\n```\n\nSee [build.yml](./.github/workflows/build.yml)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflorents-tselai%2Fpgllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflorents-tselai%2Fpgllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflorents-tselai%2Fpgllm/lists"}