{"id":15042511,"url":"https://github.com/antononcube/raku-llm-retrievalaugmentedgeneration","last_synced_at":"2025-04-10T00:32:31.168Z","repository":{"id":255898730,"uuid":"853832185","full_name":"antononcube/Raku-LLM-RetrievalAugmentedGeneration","owner":"antononcube","description":"Raku package for doing LLM Retrieval Augmented Generation (RAG).","archived":false,"fork":false,"pushed_at":"2024-10-04T01:16:32.000Z","size":258,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T02:13:36.718Z","etag":null,"topics":["distance-function","llm","llms","rag","raku","rakulang","retrieval-augmented-generation"],"latest_commit_sha":null,"homepage":"https://raku.land/zef:antononcube/LLM::RetrievalAugmentedGeneration","language":"Raku","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"artistic-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/antononcube.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-07T16:55:44.000Z","updated_at":"2024-10-04T01:16:35.000Z","dependencies_parsed_at":"2024-09-11T19:16:36.652Z","dependency_job_id":"fd8b1840-dab6-48a8-8bb2-37f68c2680c4","html_url":"https://github.com/antononcube/Raku-LLM-RetrievalAugmentedGeneration","commit_stats":null,"previous_names":["antononcube/raku-llm-retrievalaugmentedgeneration"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-LLM-RetrievalAugmentedGeneration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-LLM-RetrievalAugmentedGeneration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-LLM-RetrievalAugmentedGeneration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-LLM-RetrievalAugmentedGeneration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/antononcube","download_url":"https://codeload.github.com/antononcube/Raku-LLM-RetrievalAugmentedGeneration/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248135726,"owners_count":21053747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distance-function","llm","llms","rag","raku","rakulang","retrieval-augmented-generation"],"created_at":"2024-09-24T20:47:25.161Z","updated_at":"2025-04-10T00:32:31.140Z","avatar_url":"https://github.com/antononcube.png","language":"Raku","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM::RetrievalAugmentedGeneration\n\n[![Actions Status](https://github.com/antononcube/Raku-LLM-RetrievalAugmentedGeneration/actions/workflows/linux.yml/badge.svg)](https://github.com/antononcube/Raku-LLM-RetrievalAugmentedGeneration/actions)\n[![Actions Status](https://github.com/antononcube/Raku-LLM-RetrievalAugmentedGeneration/actions/workflows/macos.yml/badge.svg)](https://github.com/antononcube/Raku-LLM-RetrievalAugmentedGeneration/actions)\n\u003c!--- [![Actions Status](https://github.com/antononcube/Raku-LLM-RetrievalAugmentedGeneration/actions/workflows/windows.yml/badge.svg)](https://github.com/antononcube/Raku-LLM-RetrievalAugmentedGeneration/actions) --\u003e\n\n[![](https://raku.land/zef:antononcube/LLM::RetrievalAugmentedGeneration/badges/version)](https://raku.land/zef:antononcube/LLM::RetrievalAugmentedGeneration)\n[![License: Artistic-2.0](https://img.shields.io/badge/License-Artistic%202.0-0298c3.svg)](https://opensource.org/licenses/Artistic-2.0)\n\n\nRaku package for doing LLM Retrieval Augment Generation (RAG).\n\n-----\n\n## Motivation and general procedure\n\nAssume we have a large (or largish) collection of (Markdown) documents and we want \nto interact with it as if a certain LLM model has been specially trained with that collection.\n\nHere is one way to achieve this:\n\n1. The \"data wrangling problem\" is the conversion of the a collection of documents into Markdown files, and then partitioning those files into text chunks.\n   - There are several packages and functions that can do the conversion.\n   - It is not trivial to partition texts into reasonable text chunks.\n     - Certain text paragraphs might too big for certain LLMs to make embeddings for.\n2. Each of the text chunks is \"vectorized\" via LLM embedding.\n3. Then the vectors are put in a vector database or \"just\" into a \"nearest neighbors\" finding function object.\n   - A large nearest neighbors finding object can be made with [\"Math::Nearest\"](https://raku.land/zef:antononcube/Math::Nearest), [AAp6]. \n   - Alternatively, a recommender system can be used like [\"ML::StreamsBlendingRecommender\"](https://github.com/antononcube/Raku-ML-StreamsBlendingRecommender), [AAp8]. \n4. When a user query is given:\n   - The LLM embedding vector is being found.\n   - The closest text chunk vectors are found.\n5. The corresponding closest text chunks are given to the LLM to formulate a response to user's query.\n\n------\n\n## Workflow\n\nHere is the Retrieval Augmented Generation (RAG) workflow we consider:\n\n- The document collection is ingested.\n- The documents are split into chunks of relevant sizes.\n  - LLM embedding models have token limit that have to be respected.\n  - It might be beneficial or desirable to split into \"meaningful\" chunks.\n    - I.e. complete sentences or paragraphs.\n- Large Language Model (LLM) embedding vectors are obtained for all chunks.\n- A vector database is created with these embedding vectors and stored locally. Multiple local databases can be created.\n- A relevant local database is imported for use.\n- An input query is provided to a retrieval system.\n- The retrieval system retrieves relevant documents based on the query.\n- The top K documents are selected for further processing.\n- The model is fine-tuned using the selected documents.\n- The fine-tuned model generates an answer based on the query.\n- The output answer is presented to the user.\n\n### Component diagram\n\nHere is a Mermaid-JS component diagram that shows the components of performing the Retrieval Augmented Generation (RAG) workflow:\n\n```mermaid\nflowchart TD\n    subgraph LocalVDB[Local Folder]\n        A(Vector Database 1)\n        B(Vector Database 2)\n        C(Vector Database N)\n    end\n    ID[Ingest document collection]\n    SD[Split Documents]\n    EV[Get LLM Embedding Vectors]\n    CD[Create Vector Database]\n    ID --\u003e SD --\u003e EV --\u003e CD\n\n    EV \u003c-.-\u003e LLMs\n    \n    CD -.- CArray[[CArray\u003cbr\u003erepresentation]]\n\n    CD -.-\u003e |export| LocalVDB\n\n    subgraph Creation\n        ID\n        SD\n        EV\n        CD\n    end\n\n    LocalVDB -.- JSON[[JSON\u003cbr\u003erepresentation]]\n\n    LocalVDB -.-\u003e |import|D[Ingest Vector Database]\n \n    D -.- CArray\n    F -.- |nearest neighbors\u003cbr\u003edistance function|CArray\n    D --\u003e E\n    E[/User Query/] --\u003e F[Retrieval]\n    F --\u003e G[Document Selection]\n    G --\u003e|Top K documents| H(Model Fine-tuning)\n    H --\u003e I[[Generation]]\n    I \u003c-.-\u003e LLMs\n    I --\u003eJ[/Output Answer/]\n    G --\u003e|Top K passages| K(Model Fine-tuning)\n    K --\u003e I\n\n    subgraph RAG[Retrieval Augmented Generation]\n        D \n        E\n        F\n        G\n        H\n        I\n        J\n        K\n    end\n    \n    subgraph LLMs\n        direction LR\n        OpenAI{{OpenAI}}\n        Gemini{{Gemini}}\n        MistralAI{{MistralAI}}\n        LLaMA{{LLaMA}}\n    end\n```\n\nIn this diagram:\n\n- Document collections are ingested, processed, and corresponding vector databases are made.\n  - LLM embedding models are used for obtain the vectors.\n- There are multiple local vector databases that are stored and maintained locally.\n- A vector database from the local collection is selected and ingested.\n- An input query provided by the user initiates the RAG workflow.\n- The workflow then proceeds with:\n   - retrieval\n   - document selection\n   - model fine-tuning\n   - answer generation\n   - presenting the final output\n\n-----\n\n## Implementation notes\n\n### Fast nearest neighbors\n\n- Since Vector DataBases (VDBs) are slow and \"expensive\" to compute, their stored in local directory.\n  - By default `XDG_DATA_HOME` is used; for example, `~/.local/share/raku/LLM/SemanticSearchIndex`.\n- LLM embeddings produce large, dense vectors, hence nearest neighbors finding algorithms like \n  [K-d Tree](https://en.wikipedia.org/wiki/K-d_tree#Degradation_in_performance_with_high-dimensional_data) do not apply.\n  (Although, those algorithms perform well in low-dimensions.)\n  - For example we can have 500 vectors each with dimension 1536.\n- Hence, fast C-implementations of the common distance functions were made; see [AAp7].\n\n### Smaller export files, faster imports\n\n- Exporting VDBs files in JSON format produces large files.\n  - For example: \n    - Latest LLaMA models make vectors with dimension 4096\n    - So the transcript of \"normal\" ≈ 3.5 hours long podcast would produce ≈ 55 MB JSON file size\n    - It takes ≈ 13 seconds to JSON-import that file.\n- Hence, a format for smaller file size and faster import should be investigated.\n- I investigated the use of [CBOR](https://cbor.io) via [\"CBOR::Simple\"](https://raku.land/zef:japhb/CBOR::Simple).\n- In order to facilitate the use of CBOR:\n  - The VDB class `.export` method takes `:format` argument.\n  - The `.import` method uses the file extension to determine with which format to import with.\n  - The package \"Math::DistanceFunctions::Native:ver\u003c0.1.1\u003e\" works with `num64` (`double`) and `num32` (`float`) C-arrays.\n  - There is (working) precision attribute `$num-type` in the VDB class that can be `num32` or `num64`.\n- Using CBOR instead of JSON to export/import VDB objects:\n  - Produces ≈ 2 times smaller files using `num64`; ≈ 3 times with `num32`\n  - Exporting is 30% faster with CBOR \n  - Importing VDB CBOR files is ≈ 3.5 times faster\n  - Importing `num32` CBOR exported files is problematic\n  - Importing using CBOR is \"too slow\" to make VDBs summaries (done with regexes over JSON text blobs) \n\n\n-----\n\n## TODO\n\n- [ ] TODO Implementation\n  - [X] DONE \"Short file spec\"\n  - [ ] TODO Weak and strong equivalence of VDBs\n- [ ] TODO Unit testing\n  - [X] DONE Ingest VDB\n  - [X] DONE Joining VDBs\n  - [X] DONE Using `vector-database-objects`\n  - [X] DONE Round trip export and import with CBOR and JSON formats\n  - [ ] TODO Expected \"correct\" nearest neighbors tests\n- [ ] TODO Documentation\n  - [X] DONE VDB creation notebook\n  - [X] DONE VDB ingestion and RAG notebook\n  - [X] DONE Raku-RAG demo notebook and video\n  - [ ] TODO Review (and complete if needed)\n  - [ ] TODO Overview blog post\n- [ ] TODO Applications and workflows \n  - [X] DONE Simple RAG demos\n  - [X] DONE Making nearest neighbor graphs\n  - [ ] TODO Using a recommender system\n    - I.e. [\"ML::StreamsBlendingRecommender\"](https://github.com/antononcube/Raku-ML-StreamsBlendingRecommender)\n  - [ ] TODO Complete RAG workflow over a real life, largish Raku package\n  - [ ] TODO RAG over \"App::Rak\"\n  - [ ] TODO RAG over docs.raku.org\n\n-----\n\n## References\n\n### Packages\n\n[AAp1] Anton Antonov,\n[WWW::OpenAI Raku package](https://github.com/antononcube/Raku-WWW-OpenAI),\n(2023),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp2] Anton Antonov,\n[WWW::PaLM Raku package](https://github.com/antononcube/Raku-WWW-PaLM),\n(2023),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp3] Anton Antonov,\n[LLM::Functions Raku package](https://github.com/antononcube/Raku-LLM-Functions),\n(2023-2024),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp4] Anton Antonov,\n[LLM::Prompts Raku package](https://github.com/antononcube/Raku-LLM-Prompts),\n(2023-2024),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp5] Anton Antonov,\n[ML::FindTextualAnswer Raku package](https://github.com/antononcube/Raku-ML-FindTextualAnswer),\n(2023-2024),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp6] Anton Antonov,\n[Math::Nearest Raku package](https://github.com/antononcube/Raku-Math-Nearest),\n(2024),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp7] Anton Antonov,\n[Math::DistanceFunctions::Native Raku package](https://github.com/antononcube/Raku-Math-DistanceFunctions-Native),\n(2024),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp8] Anton Antonov,\n[ML::StreamsBlendingRecommender Raku package](https://github.com/antononcube/Raku-ML-StreamsBlendingRecommender),\n(2021-2023),\n[GitHub/antononcube](https://github.com/antononcube).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantononcube%2Fraku-llm-retrievalaugmentedgeneration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fantononcube%2Fraku-llm-retrievalaugmentedgeneration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantononcube%2Fraku-llm-retrievalaugmentedgeneration/lists"}