{"id":13584968,"url":"https://github.com/mamei16/LLM_Web_search","last_synced_at":"2025-04-07T06:32:07.723Z","repository":{"id":212749479,"uuid":"723478797","full_name":"mamei16/LLM_Web_search","owner":"mamei16","description":"An extension for  oobabooga/text-generation-webui that enables the LLM to search the web using DuckDuckGo","archived":false,"fork":false,"pushed_at":"2024-10-29T17:47:08.000Z","size":191,"stargazers_count":166,"open_issues_count":9,"forks_count":26,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-29T19:00:35.422Z","etag":null,"topics":["chatgpt","duckduckgo","faiss","gpt","langchain","llm","llms","oobabooga","rag","text-generation-webui"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mamei16.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-25T19:51:25.000Z","updated_at":"2024-10-28T17:23:38.000Z","dependencies_parsed_at":"2023-12-20T16:38:33.710Z","dependency_job_id":"6c4e8bb4-edb6-452c-96af-c63e78145a94","html_url":"https://github.com/mamei16/LLM_Web_search","commit_stats":null,"previous_names":["mamei16/llm_web_search"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mamei16%2FLLM_Web_search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mamei16%2FLLM_Web_search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mamei16%2FLLM_Web_search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mamei16%2FLLM_Web_search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mamei16","download_url":"https://codeload.github.com/mamei16/LLM_Web_search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247607207,"owners_count":20965938,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","duckduckgo","faiss","gpt","langchain","llm","llms","oobabooga","rag","text-generation-webui"],"created_at":"2024-08-01T15:04:38.060Z","updated_at":"2025-04-07T06:32:07.713Z","avatar_url":"https://github.com/mamei16.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Give your local LLM the ability to search the web!\n![unit tests](https://github.com/mamei16/LLM_Web_search/actions/workflows/unit_tests.yml/badge.svg?branch=main)\n\nThis project gives local LLMs the ability to search the web by outputting a specific\ncommand. Once the command has been found in the model output using a regular expression,\n[duckduckgo-search](https://pypi.org/project/duckduckgo-search/)\nis used to search the web and return a number of result pages. Finally, an\nensemble of a dense embedding model and \n[Okapi BM25](https://en.wikipedia.org/wiki/Okapi_BM25) (Or alternatively, [SPLADE](https://github.com/naver/splade))\nis used to extract the relevant parts (if any) of each web page in the search results\nand the results are appended to the model's output.\n![llm_websearch](https://github.com/mamei16/LLM_Web_search/assets/25900898/f9d2d83c-e3cf-4f69-91c2-e9c3fe0b7d89)\n\n\n* **[Table of Contents](#table-of-contents)**\n  * [Installation](#installation)\n  * [Usage](#usage)\n    + [Using a custom regular expression](#using-a-custom-regular-expression)\n    + [Reading web pages](#reading-web-pages)\n  * [Search backends](#search-backends)\n    + [DuckDuckGo](#duckduckgo)\n    + [SearXNG](#searxng)\n      + [Search parameters](#search-parameters)\n  * [Search types](#search-types)\n\t  + [Simple search](#simple-search)\n\t  + [Full search](#full-search)\n  * [Keyword retrievers](#keyword-retrievers)\n    + [Okapi BM25](#okapi-bm25)\n    + [SPLADE](#splade)\n  * [Chunking Methods](#chunking-methods)\n    + [Character-based Chunking](#character-based-chunking)\n    + [Semantic Chunking](#semantic-chunking)\n  * [Recommended models](#recommended-models)\n\n## Installation\n1. Go to the \"Session\" tab of the web UI and use \"Install or update an extension\" \nto download the latest code for this extension.\n2. To install the extension's depencies you have two options:  \n   1. **The easy way:** Run the appropriate `update_wizard` script inside the text-generation-webui folder\n   and choose `Install/update extensions requirements`. This installs everything using `pip`,\n   which means using the unofficial `faiss-cpu` package. Therefore, it is not guaranteed to\n   work with your system (see [the official disclaimer](https://github.com/facebookresearch/faiss/wiki/Installing-Faiss#why-dont-you-support-installing-via-xxx-)).\n   2. **The safe way:** Manually update the conda environment in which you installed the dependencies of \n   [oobabooga's text-generation-webui](https://github.com/oobabooga/text-generation-webui).\n   Open the subfolder `text-generation-webui/extensions/LLM_Web_search` in a terminal or conda shell.\n   If you used the one-click install method, run the command \n   `conda env update -p \u003cpath_to_your_environment\u003e --file environment.yml`,\n   where you need to replace `\u003cpath_to_your_environment\u003e` with the path to the \n   `/installer_files/env` subfolder within the text-generation-webui folder.\n   Otherwise, if you made your own environment, \n   use `conda env update -n \u003cname_of_your_environment\u003e --file environment.yml`  \n     (NB: Solving the environment can take a while)\n3. Launch the Web UI by running the appropriate `start` script and enable the extension under the session tab.  \n Alternatively,\nyou can also start the server directly using the following command (assuming you have activated your conda/virtual environment):  \n```python server.py --extension LLM_Web_search```\n\nIf the installation was successful and the extension was loaded, a new tab with the \ntitle \"LLM Web Search\" should be visible in the web UI.\n\nSee https://github.com/oobabooga/text-generation-webui/wiki/07-%E2%80%90-Extensions for more\ninformation about extensions.\n\n## Usage\n\nSearch queries are extracted from the model's output using a regular expression. This is made easier by prompting the model\nto use a fixed search command (see `system_prompts/` for example prompts).\n\nAn example workflow of using this extension could be:\n1. Load a model\n2. Head over to the \"LLM Web search\" tab\n3. Load a custom system message/prompt\n4. Ensure that the query part of the command mentioned in the system message \ncan be matched using the current \"Search command regex string\" \n(see \"Using a custom regular expression\" below)\n5. Pick a generation parameter preset that works well for you. You can read more about generation parameters [here](https://github.com/oobabooga/text-generation-webui/wiki/03-%E2%80%90-Parameters-Tab#generation)\n6. Choose \"chat-instruct\" or \"instruct\" mode and start chatting\n\n### Using a custom regular expression\nThe default regular expression is:  \n```regexp\nSearch_web\\(\"(.*)\"\\)\n```\nWhere `Search_web` is the search command and everything between the quotation marks\ninside the parentheses will be used as the search query. Every custom regular expression must use a\n[capture group](https://www.regular-expressions.info/brackets.html) to extract the search\nquery. I recommend https://www.debuggex.com/ to try out custom regular expressions. If a regex\nfulfills the requirement above, the search query should be matched by \"Group 1\" in Debuggex.\n\nHere is an example of a more flexible, but more complex, regex that works for several\ndifferent models:\n```regexp\n[Ss]earch_web\\((?:[\"'])(.*)(?:[\"'])\\)\n```\n### Reading web pages\nBasic support exists for extracting the full text content from a webpage. The default regex to use this\nfunctionality is:\n```regexp\nDownload_webpage\\(\"(.*)\"\\)\n```\n**Note**: The full content of a web page is likely to exceed the maximum context length of your average local LLM.\n## Search backends\n\n### DuckDuckGo\nThis is the default web search backend.\n\n### SearXNG\n\nTo use a local or remote SearXNG instance instead of DuckDuckGo, simply paste the URL into the \n\"SearXNG URL\" text field of the \"LLM Web Search\" settings tab (be sure to include `http://` or `https://`). The instance must support\nreturning results in JSON format.\n\n#### Search parameters\nTo modify the categories, engines, languages etc. that should be used for a\nspecific query, it must follow the\n[SearXNG search syntax](https://docs.searxng.org/user/search-syntax.html). Currently, \nautomatic redirect and Special Queries are not supported.\n\n\n## Search types\n### Simple search\nQuickly finds answers using just the highlighted snippets from websites returned by the search engine. If you simply want results *fast*, choose this search type.  \nNote: Some advanced options in the UI will be hidden when simple search is enabled, as they have no effect in this case.  \nNote2: The snippets returned by SearXNG are often much more useful than those returned by DuckDuckGo, so consider using SearXNG as the search backend if you use simple search.\n### Full search\nScans entire websites in the results for a more comprehensive search. Ideally, this search type should be able to find \"needle in the haystack\" information hidden somewhere in the website text. Hence, choose this option if you want to trade a more resource intensive search process for generally more relevant search results.   \nFor the best possible search results, also enable semantic chunking and use SPLADE as the keyword retriever.\n## Keyword retrievers\n### Okapi BM25\nThis extension comes out of the box with \n[Okapi BM25](https://en.wikipedia.org/wiki/Okapi_BM25) enabled, which is widely used and very popuplar\nfor keyword based document retrieval. It runs on the CPU and,\nfor the purpose of this extension, it is fast.  \n### SPLADE\nIf you don't run the extension in \"CPU only\" mode and have some VRAM to spare,\nyou can also select [SPLADE](https://github.com/naver/splade) in the \"Advanced settings\" section\nas an alternative. It has been [shown](https://arxiv.org/pdf/2207.03834.pdf) to outperform BM25 in multiple benchmarks \nand uses a technique called \"query expansion\" to add additional contextually relevant words\nto the original query. However, it is slower than BM25. You can read more about it [here](https://www.pinecone.io/learn/splade/).  \n\nTo improve performance, documents are embedded in batches and in parallel. Increasing the\n\"SPLADE batch size\" parameter setting improves performance up to a certain point,\nbut VRAM usage ramps up quickly with increasing batch size. A batch size of 8 appears \nto be a good trade-off, but the default value is 2 to avoid running out of memory on smaller\nGPUs.\n\n## Chunking Methods\n\n### Character-based Chunking\n\nNaively partitions a website's text into fixed sized chunks without any regard for the text content. This is the default, since it is fast and requires no GPU.\n\n### Semantic Chunking\n\nTries to partition a website's text into chunks based on semantics. If two consecutive sentences have very different embeddings (based on the cosine distance between their embeddings), a new chunk will be started. How different two consecutive sentences have to be for them to end up in different chunks can be tuned using the ` sentence split threshold` parameter in the UI.  \nFor natural language, this method generally produces much better results than character-based chunking. However, it is noticable slower, even when using the GPU.\n\n## Recommended models\nIf you (like me) have ≤ 12 GB VRAM, I recommend using one of:\n- [Llama-3.1-8B-instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)\n- [Gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it)\n- [Mistral-Nemo-Instruct-2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407)\n- [Gemma-3-it](https://huggingface.co/google/gemma-3-12b-it)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmamei16%2FLLM_Web_search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmamei16%2FLLM_Web_search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmamei16%2FLLM_Web_search/lists"}